Data Engineer/ Analytical Engineer - New York, United States - Tata Consultancy Services

Description

• Programmed user-defined functions using Python packages like Pandas, NumPy, JSON, and Itertools for the optimized ingestion flow with a data orchestration tool called Prefect.
• Performed Python Unit testing of developed code snippets using pytest and mock.
• Maintained data Pipelines up to 99.8% while ingesting streaming and transactional data across various data sources using Python and Prefect.
• Ingested data from disparate data sources using a combination of SQL, and Google Analytics API using Python to create data views to be used in BI tools like Tableau and PowerBI.
• Created multiple code snippets of DBT for Data Transformation, creating transformed tables and views in Big Query Production datasets.
• Performing joins, group by, and other aggregate operations as per Business rules on GCP Big query.
• Data Modelling of the raw ingested data using SQL, Python, and DBT tools while increasing the quality and consistency of the business applications.
• Migration from atomic layer to DBT, while automating the ETL process across billions of rows of data.
• Built an ETL that ingests historical and transactional data from a Web API with about 12,000 daily actives users.
• Analyzed data of developers in the SecAPI where the issues are about a million records in a day and ingested this data in Big Query through the Google Storage Buckets while using Python CICD.
• Worked with event-based Google Cloud functions which trigger the ingestion pipelines when there is new data in the storage buckets.
• Used Kanban framework to implement agile and DevOps software development which requires real time communication and full transparency of work.