AugmntX

AugmntX is the world’s largest network of pre-vetted talent.

Contact Info

305, IRIS Corporate Centre,
Hiranandani Meadows, 400610.
hello@augmntx.com
+919820045154

Follow Us

Home / Developers / AXF20126
profile image
Agrawal P AXF20126 Active
Data Engineer in Pune
Data engineer with over 10 years of experince in Data migration, ETL pipelines and batch processing. Worked on designing data migration processes from RDMS system to Hadoop system and then on GCP cloud. Implemented python scripting for data cleansing and batch processing. Experience with Apach NiFi and other AWS services like AWS Glue, Lambda, S3.
Industries : Telecommunication, Healthcare
Share
Hire Agrawal P
Download PDF
Availability
Full-time
Total experience
10 years
Technical skills
SQL
10 Years
ETL
6 Years
Hadoop
5 Years
Big Data Technologies
5 Years
GCP
3 Years
Python
2 Years
Airflow
2 Years
AWS Glue
1 Years
AWS Lambda
1 Years
Scoop, AWS, Nifi
Projects
Data Migration
March-2014 to December-2019
Description
Design and implementation of data migration process from RDMS system to Hadoop data architecture
Roles and Responsibilities
Migrating 460 tables from oracle, MySQL and Netezza databases using Sqoop to Hadoop Developed utility to pull multiple tables from different environments by taking config parameters Creating layered architecture to bring data from incoming, staging, work and then to gold layer. Bring history then performing data cleansing activities like removing duplicates, removing extra characters, with strategy of loading full-load and incremental
Technologies: SQL, Hadoop, Scoop
Industry: Telecommunication
ETL Processing
January-2020 to December-2021
Description
Implementation of ETL processes and Data Migration
Roles and Responsibilities
Providing solutions on customer datasets like Fraud Detection in Retail domain Improving data quality by performing DQ and DNTL checks Finding top 20 and bottom 20 search products by customers as per each state, department, and access_channel then generating report Python scripts for custom DAGs to pull data from S3 to GCS and load to Big Query Data cleaning using Python script Data migration from RDBMS to Hadoop and GCP Transforming existing SQL queries to Pig/Hive CI/CD pipelines using Jenkins
Technologies: Python, GCP, Hadoop
Industry: Healthcare
ETL Pipeline and Data Migration
January-2022 to
Description
Implementation of ETL pipeline and Data Migration from one system to another
Roles and Responsibilities
Leading a team of 5 developers for various projects Data migrations between different systems like Hadoop to GCP Big Query using DAGs File transfers using Apache Nifi Built a data processing pipeline on AWS cloud using Redshift, S3, Lambda, AWS Glue, Event rules Find the product search trends of the client and recommend them with specific offers Stakeholder management like Business Analysts and Product Leads for requirements and creating tech specs
Technologies: AWS, GCP, Nifi
Industry: Healthcare
Work history
Module Lead • March-2014
Description: • Migrating 460 tables from oracle, MySQL and Netezza databases using Sqoop to Hadoop • Developed utility to pull multiple tables from different environments by taking config parameters • Creating layered architecture to bring data from incoming, staging, work and then to gold layer. • Bring history then performing data cleansing activities like removing duplicates, removing extra characters, with strategy of loading full-load and incremental
Soft skills
Communication
Problem-solving
Teamwork
Education
Bachelors degree in Computer Engineering
Amravati University
2008 to 2012
Language
English - Advanced