Software Engineer • July-2020
Description: • Collaborating with cross-functional teams (Ireland Retail-IT Rx Business) to understand their data requirements and design appropriate data models, architectures, pipelines, and deploy the respective data product within aggressive timelines, following the security practices (EIS approved) . • Standardized ETL process to create Netezza Products using Spark & Airflow, reduced data footprint by 99% • Migrated and maintaining 300TB data from Netezza and 35k+ tables from DB2 iSeries data to HDFS. • Implemented data mesh architecture on Azure using Scala & Pyspark in Databricks and Data Factory to create Data Products, additonally developed features to add resilience, auto-recovery and alerting mechanisms via ServiceNow APIs, Logic Apps and Data Factory. • Automated Unit Testing framework for any enhancement to the data product delivery process, by adding apt artifacts and configs in CI/CD pipleine in Azure Devops, Jenkins, writing testcases in Python.