- Should have experience with both streaming and batch workflows will be essential in ensuring the efficient flow and processing of data to support our clients.
- Collaborate with cross-functional teams to understand data requirements and design robust data architecture solutions.
- Design, develop, and implement scalable data processing solutions using Apache Spark
- Ability to organize and to keep the projects well-arranged and structured.
- Ensure data quality, integrity, and consistency throughout the ETL pipeline.
- Integrate data from different systems and sources to provide a unified view for analytical purposes.
- Collaborate with data analysts to implement solutions that meet their data integration needs.
- Design and implement streaming workflows using PySpark Streaming or other relevant technologies.
- Develop batch processing workflows for large-scale data processing and analysis.
- Analyze the business requirement to determine the volume of data extracted from different sources, data models, to ensure the quality of the data involved.
- Should be able to figure out the best storage medium required for the data warehouse needed.
- Identify the data storage needs to determine the amount of data to deal with the company's requirements.
- Must ensure the data quality that everything is in place at the transformation stage to eliminate errors and fix unstructured and unorganized data extracted.
- Responsible for ensuring that the data is loaded into the warehouse system and meets the business needs and standards.
- Should be responsible for data flow validation, creating and building a secured database warehouse that meets a given company's needs and standards.
- Must be responsible for determining the storage needs of a business and the volume of data involved.
- Should have 10+ years of experience in implementing ETL processes to extract, transform, and load data from various sources to ensure data quality, integrity, and consistency throughout the ETL pipeline.
- Must be expertise in Python, PySpark, ETL processes, CI/CD (Jenkins or GitHub).
- Experienced in Python and PySpark to develop efficient data processing and analysis scripts.
- Optimize code for performance and scalability, keeping up-to-date with the latest industry best practices.
- Must load data and be proficient in valuable technical skills such as SQL, JAVA, XML, and DOM, among others.
- Extensive knowledge and experience with Spark and its technologies.
- Hands-on Experience with Apache Spark framework, including the Spark SQL module for querying databases.
- Good knowledge on data analysis, design, and programming skills such as JavaScript, SQL and XML, and DOM.
- Familiar with various coding languages used in web development, including HTML, CSS, JavaScript, Python, Java, Scala, or R proficiency.
- Should have experience in writing clean code that's free of bugs and reproducible by other developers.
- Experience in managing SQL databases and organizing big data.
- Hands-on experience with ETL such as MS SQL, SSIS (Server Integration Services), Python / Perl, Oracle, SQL Server/ MySQL.
- Solid understanding of Data warehousing schemes, Dimensional modeling, and implementing data storage solutions to support efficient data retrieval and analysis.
- Must be an expert in debugging ETL processes, optimizing data flows, and ensuring that the data pipeline is robust and error-free.
- Good knowledge of Verbal and communication skills.
-
Spark/Big Data Developer
1 week ago
Dtcc McLean, United StatesJob Description · Short Description for Internal Candidates · The data engineer role is a technical person who is involved with architecting, building, testing, and maintaining the data platform. Data engineers will implement infrastructure for data processing, analysis, report ...
-
Backend Java Developer
1 day ago
Pyramid Consulting, Inc McLean, United StatesImmediate need for a talented Backend Java Developer. This is a 09+ months contract opportunity with long-term potential and is located in McLean, VA (Hybrid). Please review the job description below and contact me ASAP if you are interested. · Job ID: · Pay Range: $60 - $65/hou ...
-
Visionary Innovative Technology Solutions LLC McLean, United StatesPosition: ETL IICS/ Informatica Developer with PySpark Experience · Location: Mclean, VA (Onsite) · Duration: 6+ Month · Job Description: · We are looking for a seasoned ETL Developer with IICS experience. · Experience with Python, PySpark. · Knowledge and experience with Java / ...
-
Senior Data Engineer
3 weeks ago
Technology Ventures McLean, United StatesMust Haves: Minimum of 5 years of hands on experience with Python. Hand on experience with Snowflake. Proficiency with database management, specifically SQL. Must have hands on experience with AWS - specifically must have experience with Lambda, S3, EC2, CloudWatch, SSM, and EMR. ...
-
Backend Developer
21 hours ago
Radiansys Inc. McLean, United StatesWe have open position of Backend Developer – Mclean ,VA (Hybrid) – Only W2 · Please share your resume at if you are available for new position. · Backend Developer · Mclean ,VA (Hybrid) · Must have - Java , Scala , Python · Good to have AWS – Glue (Knowledge is fine), Lambda, S3 ...
-
Lead Java Developer
1 week ago
Hexaware Technologies McLean, United StatesPosition : Java lead Developer · Location : McLean, VA · Required skills and experience: · 10+ years of experience in Solution, Design and Development of applications using J2EE, Spring Boot, Microservices, RESTful services and Angular · Very strong experience in Microservices de ...
-
Salesforce Technical Manager
1 week ago
HCLTech McLean, United StatesHCLTech is looking for a highly talented and self- motivated candidate to join it in advancing the technological world through innovation and creativity. · Job Title: Salesforce Technical Project Manager · Position Type: Full-time · Location: Tysons Corner, Virginia (Day 1 Onsite ...
-
Senior Java Fullstack Developer
1 week ago
Hexaware Technologies McLean, United StatesJob Title: Senior Java Full Stack Engineer - Professional · McLean, VA (hybrid – 3 days onsite, 2 days remote) · Employment Type - Contract/Full time Both · Job Description · Mandatory: · 7+ years of experience in in solution, design and development of applications using Java 8 ...
-
Snowflake Admin
3 weeks ago
Dexian McLean, United StatesTitle: Snowflake Admin · Duration: 7 Months + Possible extensions · Location: Mclean VA · Responsibilities: · Develop data filtering, transformational and loading requirements · Define and execute ETLs using Apache Sparks on Hadoop among other Data technologies · Determine approp ...
-
Lead Java Developer
4 days ago
HCLTech McLean, United StatesHCLTech is looking for a highly talented and self- motivated Lead Java Developer to join it in advancing the technological world through innovation and creativity. · Job Title: Lead Java Developer · Job ID: BR · Position Type: Full-time · Location: McLean, VA, USA (Hybrid) · Tech ...
-
Stem IT McLean, United StatesR&D group of a Fortune 1000 Data Conglomerate is looking to hire a Senior Software Engineer to assist with a multitude of DARPA based projects for its Mclean based operation(2 days onsite a week). This person will working on all new green development projects utilizing an array o ...
-
Data Analyst
6 days ago
Pyramid Consulting, Inc McLean, United StatesImmediate need for a talented Data Analyst (Junior-Mid-level). This is a 06+Months contract opportunity with long-term potential and is located in Mclean, VA (Hybrid). Please review the job description below and contact me ASAP if you are interested. · Job ID: · Pay Range: $40/h ...
-
AWS Data Engineer
4 weeks ago
Pyramid Consulting, Inc McLean, United StatesImmediate need for a talented AWS Data Engineer. This is a 06+ Months Contract opportunity with long-term potential and is located in McLean, VA (Hybrid). Please review the job description below and contact me ASAP if you are interested. · Job ID: · Pay Range: $50 - $60/hour. Em ...
-
Java Tech Lead
6 days ago
Hexaware Technologies McLean, United StatesWhat Working at Hexaware offers: · Hexaware is a dynamic and innovative IT organization committed to delivering cutting-edge solutions to our clients worldwide. We pride ourselves on fostering a collaborative and inclusive work environment where every team member is valued and em ...
-
Senior Spark
1 week ago
Mavens Guild McLean, United StatesWhat we would like to see: · In a senior developer role, you will design and build data flow and data integration processes to enhance loss prevention technologies for a leading financing firm. Drawing from your vast hands-on experience in Python (PySpark), Spark, REST, Java, and ...
-
Python Developer
1 week ago
Saxon Global McLean, United StatesRequired Qualifications · 5+ Years Experience ? in the following: Python, Java, Scala, Spark, AirFlow,? Javascript/TypeScript · 3+ Years Experience%?20in AWS Cloud Technologies (ECS, EC2/Fargate, S3, Lambd? a, EMR, SQS, DynamoDB etc.) · 3+ ? Years Experience in DevOps (Git, Do ...
-
System Engineer
3 days ago
Sensible Solutions and Technologies Inc McLean, United StatesJob Description · The Sponsor requires a team to support their program that automates processing of large forensic images, extract and enrich metadata, and display resulting information in meaningful ways for analysts to conduct their assessments. · Job Description · The Spons ...
-
Systems Engineer
2 weeks ago
ABBTECH Professional Resources McLean, United States**Systems Engineer** · **Location- Mclean, Va** · **Clearance- TS/SCI w/ FSP** · **Salary- 220k-240k** · _The above salary range represents the range expected for the position; however, final salary offers are based on a number of factors such as the positions responsibilitie ...
-
Application Developer III
2 weeks ago
vodastra McLean, United StatesCompany Overview · Job Title: Application Developer III - (Surge Team BI - Data Engineering Team) · Job Summary · Duration: 6+ Months (High Possibility of Extension) · Responsibilities and Duties Required Skills: · Apache Spark (or) Apache Flink (or) Apache Hadoop (or) Apach ...
-
Java Developer III
2 weeks ago
vodastra McLean, United StatesCompany Overview · Urgently need Developer for Multiple Java role in McLean, VA · Job Summary · Familiar with standard concepts, practices, and procedures within a particular field. Relies on experience and judgment to plan and accomplish goals. Performs a variety of tasks. A ...
ETL / Spark Developer - Mclean, United States - PETADATA
Description
POSITION: ETL / Spark Developer
LOCATION: McLean, VA. (Hybrid)
Experience: 10+ years
Work type: Fulltime (W2 / C2C)
PETADATA is currently looking to hire for the ETL / Spark Developer role for one of their clients.
Roles & Responsibilities:
Required skills:
Educational Qualification:
Bachelor's/ Master's degree in Computer Science, Engineering, or a related field.
We offer a professional work environment and are given every opportunity to grow in the Information technology world.
Note:
Candidates required to attend Phone/Video Call / In person interviews and after Selection of candidate (He/She) should go through all background checks on Education and Experience.
Please email your resume to:
After carefully reviewing your experience and skills one of our HR team members will contact you on the next steps.
