- Architect and develop data pipelines to optimize performance, quality, and scalability
- Build, maintain & operate scalable, performant, and containerized infrastructure required for optimal extraction, transformation, and loading of data from various data sources
- Design, develop, and own robust, scalable data processing and data integration pipelines using Python, dbt, Kafka, Airflow, PySpark, SparkSQL, and REST API endpoints to ingest data from various external data sources to Data Lake
- Develop testing frameworks and monitoring to improve data quality, observability, pipeline reliability, and performance
- Orchestrate sophisticated data flow patterns across a variety of disparate tooling
- Support analytics engineers, data analysts, and business partners in building tools and data marts that enable self-service analytics
- Partner with the rest of the Data Platform team to set best practices and ensure the execution of them
- Partner with the analytics engineers to ensure the performance and reliability of our data sources
- Partner with machine learning engineers to deploy predictive models
- Partner with the legal and security teams to build frameworks and implement data compliance and security policies
- Partner with DevOps to build IaC and CI/CD pipelines
- Support code versioning and code deployments for data Pipelines
- 8+ years of professional experience designing, creating and maintaining scalable data pipelines using Python, API calls, SQL, and scripting languages
- Demonstrated experience writing clean, efficient & well-documented Python code and are willing to become effective in other languages as needed
- Demonstrated experience writing complex, highly optimized SQL queries across large data sets
- Experience with Azure Cloud technologies
- Experience with Databricks platform
- Experience with data warehouses like BigQuery, Databricks, Snowflake, and Postgres
- Experience building event streaming pipelines using Kafka/Confluent Kafka
- Experience with modern data stack like Airflow/Astronomer, Databricks, dbt, Fivetran, Confluent, Tableau/Looker
- Experience with containers and container orchestration tools such as Docker or Kubernetes
- Experience with CI/CD (Jenkins, GitHub Actions, Circle CI)
- Thorough understanding of SDLC and Agile frameworks
-
Data Engineer
6 days ago
Compunnel Inc. Charlotte, United StatesThis is a W2 role. · Responsibilities: · The person on this role will play a crucial role in building scalable and cost-effective data pipelines, data lakes, and analytics systems. · Data Ingestion: Implement data ingestion processes to collect data from various sources, includin ...
-
Data Engineer
1 week ago
Alpha Theory Charlotte, United StatesAbout Alpha Theory: · Alpha Theory is a fintech software company that prides itself on an inclusive and transparent culture that attracts curious teammates who are excited about tackling big ideas. We hire smart, motivated team members that crave feedback to be better. We believe ...
-
Data Engineer
3 days ago
Belk, Inc. & Belk eCommerce LLC Charlotte, United States OTHEREDW Data Engineer is responsible for designing, maintaining, and optimizing data infrastructure for data collection, management, transformation, and access. Creates pipelines that convert raw data into usable formats for data scientists and other data consumers to utilize. Builds ...
-
DATA ENGINEER
6 days ago
Diverse Lynx Charlotte, United StatesWe are looking for a highly capable Data Engineer who has experience in building end-to-end systems · • Experience on implementation of Data Lakes using Cloud (AWS or GCP), Hands-on with Big Data and related technologies including Spark, Kafka, ELK. · • Define and setup develop ...
-
Data Engineer
6 days ago
CRG Charlotte, United StatesData Engineer · Location: Remote EST hours · Duration: 12-month contract w/ potential to hire. · Pay: $65+/hr · JOB DESCRIPTION · Our global Fortune 500 client, with U.S. headquarters in Charlotte, NC, is a world class food service provider with a strong presence across the ...
-
Data Engineer
6 days ago
Syntricate Technologies Charlotte, United StatesHi, · Hope you are doing well · Number of position : 3 · Only Full Time no C2C and No Contract · Full-time · I, Shakib(i3 infotek) would like to share a job opportunity as Data Engineer based in Charlotte, NC (Onsite) location for a Full-time position. · *** In case, if you are ...
-
Data Engineer
12 hours ago
LHH Charlotte, United States**Must be local to Charlotte area for a weekly hybrid commute** · LHH is seeking a Data Engineer for a rapidly growing company based out the Charlotte, NC area. The ideal candidate will have 3+ years of experience with data engineering and a proven track record of implementing ro ...
-
Data Engineer
6 days ago
Truist Financial Charlotte, United StatesThe position is described below. If you want to apply, click the Apply button at the top or bottom of this page. You'll be required to create an account or sign in to an existing one. · Need Help? · If you have a disability and need assistance with the application, you can requ ...
-
Data Engineer
5 days ago
PamTen Charlotte, United StatesExperience in Data Warehouse related projects in product or service-based organization · • Expert knowledge of Linux operating systems and environment and Scripting (Shell and Python preferred) · • A deep expertise in your field of Software Engineering · • Expert at troubleshoo ...
-
Data Engineer
1 day ago
Sumitomo Mitsui Banking Corp Charlotte, United StatesSMBC Group is a top-tier global financial group. Headquartered in Tokyo and with a 400-year history, SMBC Group offers a diverse range of financial services, including banking, leasing, securities, credit cards, and consumer finance. The Group has more than 130 offices and 80,000 ...
-
Data Engineer
1 week ago
Odyssey Logistics & Technology Charlotte, United StatesData Engineer · 1916 Ayrsley Town Blvd suite 400, Charlotte, NC 28273, USA · Req #2397 · Wednesday, February 14, 2024 · See yourself at Odyssey. · Join the team as our new Data Engineer · Who We Are & Why We're Hiring · Odyssey Logistics & Technology Corporation (Odyssey) is ...
-
Data Engineer
1 hour ago
SMBC Charlotte, United StatesSMBC Group is a top-tier global financial group. Headquartered in Tokyo and with a 400-year history, SMBC Group offers a diverse range of financial services, including banking, leasing, securities, credit cards, and consumer finance. The Group has more than 130 offices and 80,000 ...
-
Data Engineer
1 week ago
Sia Partners Charlotte, United StatesSia Partners is a next-generation management consulting firm. We offer a unique blend of AI and design capabilities, augmenting traditional consulting to deliver superior value to our clients. Counting 3,000 consultants in 19 countries, we expect to achieve USD 420 million in tur ...
-
Data Engineer
2 days ago
Netskope Charlotte, United StatesAbout Netskope · Today, there's more data and users outside the enterprise than inside, causing the network perimeter as we know it to dissolve. We realized a new perimeter was needed, one that is built in the cloud and follows and protects data wherever it goes, so we started Ne ...
-
Data Engineer
4 days ago
Experis Charlotte, United StatesDallas or Charlotte - Hybrid 2-3 days in office · Pay: · 24 month contract · Lead Data Engineer · Must Have: · typical ETL/Abintio but specifically Metadata hub-> an abinitio etl tool: allowing users to leverage metadata-driven development and automation capabilities. Metada ...
-
Data Engineer
1 week ago
Oriental Trading Company Charlotte, United StatesChello is built on a foundation of pioneering data-intelligence and our team of innovators, driven to tackle the financial challenges of the industries we serve. At its core, it's the first of its kind digital banking platform that delivers daily business insights, cash flow pred ...
-
Data Engineer
1 week ago
The Hartford Charlotte, United StatesData Engineer - GE08AE · We're determined to make a difference and are proud to be an insurance company that goes well beyond coverages and policies. Working here means having every opportunity to achieve your goals - and to help others accomplish theirs, too. Join our team as w ...
-
Data Engineer
1 week ago
Clark Davis Associates Charlotte, United States*Must be US Citizen or GC holder · Scroll down to find the complete details of the job offer, including experience required and associated duties and tasks. · *No C2C · You Will: · Architect and develop data pipelines to optimize performance, quality, and scalability · Build ...
-
Data Engineer Data Analytics
1 week ago
Jobot Charlotte, NC, United StatesThis Jobot Job is hosted by: Jake Durham · We are driven by a singular mission: to empower individuals and businesses to achieve their financial goals through cutting-edge technology. As a leading Fin-tech innovator, we understand the evolving landscape of finance and the import ...
-
Data Engineer
3 weeks ago
Infinicsinc Charlotte, United States Full timeBachelors' degree in Computer Science Engineering Information Technology or related area 7+ years' experience on ETL Development Tools Spark / Java (preferred) Python Ab Initio Informatica Advanced experience & demonstrated proficiency in Spark. Experience with Apache Beam Cloud ...
Data Engineer - Charlotte, United States - Clark Davis Associates
Description
*Must be US Citizen or GC holder
*No C2C
You Will:
You Have: