ETL / Spark Developer - Mclean, United States - PETADATA

PETADATA Mclean, United States

3 weeks ago

Description

POSITION: ETL / Spark Developer

LOCATION: McLean, VA. (Hybrid)

Experience: 10+ years

Work type: Fulltime (W2 / C2C)

PETADATA is currently looking to hire for the ETL / Spark Developer role for one of their clients.

Roles & Responsibilities:

Should have experience with both streaming and batch workflows will be essential in ensuring the efficient flow and processing of data to support our clients.

Collaborate with cross-functional teams to understand data requirements and design robust data architecture solutions.

Design, develop, and implement scalable data processing solutions using Apache Spark

Ability to organize and to keep the projects well-arranged and structured.

Ensure data quality, integrity, and consistency throughout the ETL pipeline.

Integrate data from different systems and sources to provide a unified view for analytical purposes.

Collaborate with data analysts to implement solutions that meet their data integration needs.

Design and implement streaming workflows using PySpark Streaming or other relevant technologies.

Develop batch processing workflows for large-scale data processing and analysis.

Analyze the business requirement to determine the volume of data extracted from different sources, data models, to ensure the quality of the data involved.

Should be able to figure out the best storage medium required for the data warehouse needed.

Identify the data storage needs to determine the amount of data to deal with the company's requirements.

Must ensure the data quality that everything is in place at the transformation stage to eliminate errors and fix unstructured and unorganized data extracted.

Responsible for ensuring that the data is loaded into the warehouse system and meets the business needs and standards.

Should be responsible for data flow validation, creating and building a secured database warehouse that meets a given company's needs and standards.

Must be responsible for determining the storage needs of a business and the volume of data involved.

Required skills:

Should have 10+ years of experience in implementing ETL processes to extract, transform, and load data from various sources to ensure data quality, integrity, and consistency throughout the ETL pipeline.

Must be expertise in Python, PySpark, ETL processes, CI/CD (Jenkins or GitHub).

Experienced in Python and PySpark to develop efficient data processing and analysis scripts.

Optimize code for performance and scalability, keeping up-to-date with the latest industry best practices.

Must load data and be proficient in valuable technical skills such as SQL, JAVA, XML, and DOM, among others.

Extensive knowledge and experience with Spark and its technologies.

Hands-on Experience with Apache Spark framework, including the Spark SQL module for querying databases.

Good knowledge on data analysis, design, and programming skills such as JavaScript, SQL and XML, and DOM.

Familiar with various coding languages used in web development, including HTML, CSS, JavaScript, Python, Java, Scala, or R proficiency.

Should have experience in writing clean code that's free of bugs and reproducible by other developers.

Experience in managing SQL databases and organizing big data.

Hands-on experience with ETL such as MS SQL, SSIS (Server Integration Services), Python / Perl, Oracle, SQL Server/ MySQL.

Solid understanding of Data warehousing schemes, Dimensional modeling, and implementing data storage solutions to support efficient data retrieval and analysis.

Must be an expert in debugging ETL processes, optimizing data flows, and ensuring that the data pipeline is robust and error-free.

Good knowledge of Verbal and communication skills.

Educational Qualification:

Bachelor's/ Master's degree in Computer Science, Engineering, or a related field.

We offer a professional work environment and are given every opportunity to grow in the Information technology world.

Note:

Candidates required to attend Phone/Video Call / In person interviews and after Selection of candidate (He/She) should go through all background checks on Education and Experience.

Please email your resume to:

After carefully reviewing your experience and skills one of our HR team members will contact you on the next steps.

Spark/Big Data Developer

5 days ago

Dtcc McLean, United States

Job Description · Short Description for Internal Candidates · The data engineer role is a technical person who is involved with architecting, building, testing, and maintaining the data platform. Data engineers will implement infrastructure for data processing, analysis, report ...
Lead Java Developer

2 days ago

Hexaware Technologies McLean, United States

Position : Java lead Developer · Location : McLean, VA · Required skills and experience: · 10+ years of experience in Solution, Design and Development of applications using J2EE, Spring Boot, Microservices, RESTful services and Angular · Very strong experience in Microservices de ...
Senior Data Engineer

1 week ago

Technology Ventures McLean, United States

Must Haves: Minimum of 5 years of hands on experience with Python. Hand on experience with Snowflake. Proficiency with database management, specifically SQL. Must have hands on experience with AWS - specifically must have experience with Lambda, S3, EC2, CloudWatch, SSM, and EMR. ...
Salesforce Technical Manager

2 days ago

HCLTech McLean, United States

HCLTech is looking for a highly talented and self- motivated candidate to join it in advancing the technological world through innovation and creativity. · Job Title: Salesforce Technical Project Manager · Position Type: Full-time · Location: Tysons Corner, Virginia (Day 1 Onsite ...
Snowflake Admin

1 week ago

Dexian McLean, United States

Title: Snowflake Admin · Duration: 7 Months + Possible extensions · Location: Mclean VA · Responsibilities: · Develop data filtering, transformational and loading requirements · Define and execute ETLs using Apache Sparks on Hadoop among other Data technologies · Determine approp ...
Senior Java Fullstack Developer

2 days ago

Hexaware Technologies McLean, United States

Job Title: Senior Java Full Stack Engineer - Professional · McLean, VA (hybrid – 3 days onsite, 2 days remote) · Employment Type - Contract/Full time Both · Job Description · Mandatory: · 7+ years of experience in in solution, design and development of applications using Java 8 ...
Big Data Engineer

4 weeks ago

Mitchell Martin Inc. McLean, United States

*****W2 ONLY, NO C2C***** · Job Title: Big Data Engineer · Client Location: McLean, VA · Type: Contract · Set-up: Hybrid (3 days a week in the office) · JOB DESCRIPTION · As a Software Engineer, your responsibilities will include developing and maintaining a data processing appli ...
AWS Data Engineer

2 weeks ago

Pyramid Consulting, Inc McLean, United States

Immediate need for a talented AWS Data Engineer. This is a 06+ Months Contract opportunity with long-term potential and is located in McLean, VA (Hybrid). Please review the job description below and contact me ASAP if you are interested. · Job ID: · Pay Range: $50 - $60/hour. Em ...
Data Engineer

3 weeks ago

QinetiQ Mc Lean, United States

Company Overview · We are a world-class team of professionals who deliver next generation technology and products in robotic and autonomous platforms, ground, soldier, and maritime systems in 50+ locations world-wide. Much of our work contributes to innovative research in the fie ...
Data Engineer

3 weeks ago

QinetiQ US Mc Lean, United States

Company Overview · We are a world-class team of professionals who deliver next generation technology and products in robotic and autonomous platforms, ground, soldier, and maritime systems in 50+ locations world-wide. Much of our work contributes to innovative research in the fie ...
Lead Data Engineer

4 weeks ago

Ascendion McLean, United States

About Ascendion · Ascendion is a full-service digital engineering solutions company. We make and manage software platforms and products that power growth and deliver captivating experiences to consumers and employees. Our engineering, cloud, data, experience design, and talent so ...
Senior Spark

4 days ago

Mavens Guild McLean, United States

What we would like to see: · In a senior developer role, you will design and build data flow and data integration processes to enhance loss prevention technologies for a leading financing firm. Drawing from your vast hands-on experience in Python (PySpark), Spark, REST, Java, and ...
Python Developer

4 days ago

Saxon Global McLean, United States

Required Qualifications · 5+ Years Experience ? in the following: Python, Java, Scala, Spark, AirFlow,? Javascript/TypeScript · 3+ Years Experience%?20in AWS Cloud Technologies (ECS, EC2/Fargate, S3, Lambd? a, EMR, SQS, DynamoDB etc.) · 3+ ? Years Experience in DevOps (Git, Do ...
Systems Engineer

1 week ago

ABBTECH Professional Resources McLean, United States

**Systems Engineer** · **Location- Mclean, Va** · **Clearance- TS/SCI w/ FSP** · **Salary- 220k-240k** · _The above salary range represents the range expected for the position; however, final salary offers are based on a number of factors such as the positions responsibilitie ...
Application Developer III

1 week ago

vodastra McLean, United States

Company Overview · Job Title: Application Developer III - (Surge Team BI - Data Engineering Team) · Job Summary · Duration: 6+ Months (High Possibility of Extension) · Responsibilities and Duties Required Skills: · Apache Spark (or) Apache Flink (or) Apache Hadoop (or) Apach ...
Java Developer III

1 week ago

vodastra McLean, United States

Company Overview · Urgently need Developer for Multiple Java role in McLean, VA · Job Summary · Familiar with standard concepts, practices, and procedures within a particular field. Relies on experience and judgment to plan and accomplish goals. Performs a variety of tasks. A ...
Systems Engineer

1 week ago

The DarkStar Group LLC McLean, United States

· The DarkStar Group is seeking a Systems Engineer with a TS/SCI + Poly clearance (applicable to this customer) to join one of our top projects in McLean, VA. Below is an overview of the project, as well as information on our company, our benefits, and our $25,000 referral progr ...
Data Engineer

4 days ago

Eclaro McLean, United States

Job Number: · Join the Eclaro Team Eclaro is looking for a Data Engineer for our client in McLean, VA. · Eclaro's client is an industry-leading managed service provider that works with multinational hospitality clients to enable an integrated approach to analyze and manage exte ...
Senior Data Engineer

3 days ago

GCI McLean, United States

GCI, embodies excellence, integrity and professionalism. The employees supporting our customers deliver unique, high-value mission solutions while effectively leverage the technological expertise of our valued workforce to meet critical mission requirements in the areas of Data A ...
Java FSD with AWS

1 week ago

Diverse Lynx McLean, United States

Role: Java FSD with AWS · Location : McLean, VA (Hybrid - 3 days onsite, 2 days remote) · JobType: Contract · Mandatory: · 7+ years of experience in in solution, design and development of applications using Java 8+/J2EE, Spring, Spring-Boot, Micro Services, RESTful Services an ...