Data Pipeline Engineer - Addison, United States - CarOffer

Description

CarOffer, a CarGurus Company, is seeking a motivated and detail-oriented

Data Pipeline Engineer

to join our data engineering team.

In this role, you will be responsible for designing, implementing, and maintaining data pipelines to efficiently ingest and stream data into our Snowflake data warehouse from diverse sources, including web analytics, third-party APIs, and FTP servers.

As a key member of our team, you will play a crucial role in ensuring the reliability, scalability, and performance of our data infrastructure, enabling timely and accurate data insights for business decision-making.

This position offers an exciting opportunity to work with cutting-edge technologies and industry-standard practices in data engineering while contributing to the success of our data-driven initiatives.

What You'll Do:
Collaborate with stakeholders to understand data requirements and source systems, identifying data sources and defining data ingestion strategies.

Design and implement robust data pipelines to ingest, process, and stream data into our Snowflake data warehouse from various sources, ensuring data quality and reliability.

Develop connectors and integrations with web analytics platforms, third-party APIs, and FTP servers to automate data extraction and ingestion processes.

Implement industry-standard practices for data pipelining, including error handling, data validation, and monitoring, to ensure data integrity and availability.
Optimize data pipeline performance and resource utilization, identifying bottlenecks and opportunities for improvement in data processing and streaming.

Collaborate with data engineers, software engineers, and infrastructure teams to integrate data pipelines into our data ecosystem and ensure seamless data flow.

Document data pipeline architecture, configurations, and dependencies, ensuring clear and comprehensive documentation for future reference and troubleshooting.
Monitor data pipeline health and performance metrics, proactively identifying and addressing issues to minimize downtime and ensure data availability.

Stay current with emerging technologies and best practices in data engineering and streaming technologies, continuously evaluating and adopting new tools and techniques.

Participate in team meetings, code reviews, and knowledge-sharing sessions, contributing to a collaborative and supportive team culture.

Who You Are:
Bachelors degree in Computer Science, Engineering, or a related field; Masters degree preferred.

1-3 years of experience in data engineering, software development, or a related field, with a focus on building and maintaining data pipelines.

Hands-on experience with data pipelining tools and technologies, such as Apache Kafka, Apache NiFi, or AWS Glue, preferred.
Proficiency in programming languages such as Python, Java, or Scala, with experience in scripting and automation.

Familiarity with cloud platforms and services, such as AWS, Azure, or Google Cloud Platform, and experience with cloud-based data services (e.g., S3, Redshift, BigQuery).

Strong understanding of data formats and protocols, including JSON, XML, REST APIs, and FTP/SFTP.
Excellent problem-solving and analytical skills, with the ability to troubleshoot and resolve complex data pipeline issues.

Strong communication and collaboration skills, with the ability to work effectively in a team environment and interact with stakeholders at all levels.

Self-motivated and proactive, with a passion for learning and staying abreast of new technologies and industry trends.
Ability to prioritize tasks, manage time efficiently, and meet project deadlines in a fast-paced, dynamic environment.

#J-18808-Ljbffr