Lead Data Engineer - Newark, United States - HireTalent - Diversity Staffing & Recruiting Firm

Description

NOTE: NO C2C

Job Description:

• Implement and support end-to-end data lake/warehousing/mart/business intelligence/ analytics/ services solutions (ingest, storage, integration, processing, services, access) in AWS.
• Data lake, data intake/request/onboarding services and service documentation, data lake ingestion services for batch/real time data ingest and service documentation, data processing services (ETL/ELT) for batch/real time (Glue/Kinesis/EMR) and service documentation, data storage services for data lake (S3)/ data warehouses (RDS/Redshift)/ data marts and service documentation, data services layer including Athena, Redshift, RDS, microservices and APIs pipeline orchestration services including lambda, step functions, MWAA (optional)
• Data security services (IAM/KMS/SM/encryption/anonymization/RBAC) and service documentation
• Data access provisioning services (Accounts, IAM Roles RBAC), processes, documentation and education
• Data provisioning services for data consumption patterns including microservices, APIs and extracts
• Metadata capture and catalog services for data lake(S3/Athena), data warehouses (RDS/Redshift), Microservices/APIs
• Metadata capture and catalog services for pipeline/log data for monitoring /support
• Implement CI/CD pipelines
• Prepare documentation for data projects utilizing AWS based enterprise data platform
• Implement high velocity streaming solutions using Amazon Kinesis, SQS, and SMS
• Migrate data from traditional relational database systems to AWS relational databases such as Amazon RDS, Aurora, and Redshift
• Migrate data from traditional file systems and NAS shares to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
• Migrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
• Provide cost/spend monitoring reporting for AWS based data platform data initiatives
• Provide governance/audit reporting for access of AWS based data platform
• Lead the implementation of a data lake strategy to enable LOBs and Corporate Functions with a robust, holistic view of data - driven decision making
• Serve as delivery lead for EDP data initiatives product owner
• Partner with immediate engineering team, product owner, IT, partners on EDP agenda
• Provide technology thought leadership, consulting, and coaching/mentoring
• Establish development, qa, stage and production migration/support processes
• Establish best practices for development and support teams
• Deliver end-end data initiatives from ingest-consume via microservices/ apis, jdbc/ odbc, file extracts etc.
• Work with scrum master to develop and own backlog, stories, epics, sprints

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience
• Experience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprises
• Programming experience with Java, Python/ Scala, Shell scripting
• Solid experience of AWS services such as CloudFormation, S3, Glue, EMR/ Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.
• Solid experience implementing solutions on AWS based data lakes
• Experience implementing metadata solutions leveraging AWS non-relational data solutions such as ElastiCache and DynamoDB
• AWS Solutions Architect or AWS Big Data Certification preferred
• Experience in AWS data lake/data warehouse/business analytics
• Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Datasync, CloudHealth, CloudWatch, Lambda, Athena, and Redshift
• Experience in system analysis, design, development, and implementation of data ingestion pipeline in AWS
• Experience with DevOps and Continuous Integration/ Delivery (CI/ CD) concepts and tools
• Experience with business intelligence tools such as Tableau, Power BI or equivalent
• Knowledge of ETL/ ELT
• Experience in production support from Level 1 to Level 3
• Awareness of Data Management & Governance tools