Big Data developer - Ohio City, United States - Infoprompt Inc

Description

Responsibilities Participate in Team activities Design discussions Stand up meetings and planning Review with team.

Perform data analysis data profiling data quality and data ingestion in various layers using big data/Hadoop/Hive/Impala queries PySpark programs and UNIX shell scripts.

Follow the organization coding standard document Create mappings sessions and workflows as per the mapping specification document. Perform Gap and impact analysis of ETL and IOP jobs for the new requirement and enhancements. Create jobs in Hadoop using SQOOP PYSPARK and Stream Sets to meet the business user needs. Create mockup data perform Unit testing and capture the result sets against the jobs developed in lower environment. Updating the production support Run book Control M schedule document as per the production release. Create and update design documents provide detail description about workflows after every production release.

Continuously monitor the production data loads fix the issues update the tracker document with the issues Identify the performance issues.

Performance tuning long running ETL/ELT jobs by creating partitions enabling full load and other standard approaches. Perform Quality assurance check Reconciliation post data loads and communicate to