Description
Engineering Data Service is a shared technology service to provide advance data processing, management and analytic capability in a cost and operationally efficient manner. EDS built to support on-premise (HADOOP) & on-cloud (GCP) Platforms with latest big data technologies and store data from strategic manner with proper governance, security and control in place. This is a Scheduled/EDI based File/Data ingestion framework (ETL) used to ingest & Process structured/Semi-structured/ unstructured files from different sources and load into on-premise(data lake), on-cloud(GCS, BigQuery table) along with different validations & reconciliation.
Roles and Responsibilities
• Take ownership of OKR and perform design, development and deployment planning and implementation. • Defined new OKR's to optimise performance, new ingestion strategies (e.g. EDI), Security & regulatory compliance, SLO, SLA and monitoring. • Coordination & collaboration with management & business customers to gather business requirements, define global strategies, procedures & decisions. • Leadership & Management of 8 people. training, mentoring, code review, helping hands in development on complex, critical stories. Collaboration with other team leads to deliver OKR in time. • Defined procedures on code quality & assurance, enforce team to follow same. Defined automation in deployment process with following compliance banking policies. • I also focus on personal profession development to learn new things, industry trends, best practices and apply them where ever needed.
Technologies: Python, PySpark, Hadoop (HDFS, Sqoop, Hive, MapReduce/Yarn), GCP(GCS, Dataflow, Dataproc, BigQuery, Composer), Control-M, OpenShift, Java, Spring Boot, Kafka
Industry: Finance