Data Engineer • September-2021
Description: • Responsible for Design, develop and maintain scalable data pipelines for large scale data using Cloud Data Fusion ETL tool. • Extract or Read data from various sources such as Google Big Query,Cloud Storage in multiple forms - tables, flat files etc. • Analyze data requirements and design efficient end-to-end ETL pipelines on Cloud Data Fusion, taking into consideration data volume, velocity, and variety. • Transform data while writing complex transformation logics using SQL, Python, or other scripting languages, to clean, standardize, and aggregate data. • Involved in designing end to end data pipelines orchestration and automate ETL processes using Google Cloud Composer or Apache Airflow. • Ensure data is being processed and stored in compliance with privacy and security regulations into Google Cloud Storage, Google BigQuery, or other data storage solutions. • Debug and troubleshoot data processing issues and perform root cause analysis to prevent future issues. • Collaborated with cross-functional teams in an SAFe agile methodology to ensure timely delivery of products. • Utilized version control systems such as Github for managing and collaborating on code projects. • Write technical documentation about ETL processes on confluence and support team members in the deployment and maintenance of data pipelines. • Experience with tracking and managing project tasks and bugs using JIRA. • Experience at all phase of the Software Development Life Cycle(SDLC) including Requirement Analysis, Design and Development,Testing,UAT,Production deployment & Support.