- Apply advanced knowledge of Data Engineering principles, methodologies and techniques to design and implement data loading and aggregation frameworks across broad areas of the organization.
- Gather and process raw, structured, semi-structured and unstructured data using batch and real-time data processing frameworks.
- Implement and optimize data solutions in enterprise data warehouses and big data repositories, focusing primarily on movement to the cloud.
- Drive new and enhanced capabilities to Enterprise Data Platform partners to meet the needs of product / engineering / business.
- Experience building enterprise systems especially using Databricks, Snowflake and platforms like Azure, AWS, GCP etc
- Leverage strong Python, Spark, SQL programming skills to construct robust pipelines for efficient data processing and analysis.
- Implement CI/CD pipelines for automating build, test, and deployment processes to accelerate the delivery of data solutions.
- Implement data modeling techniques to design and optimize data schemas, ensuring data integrity and performance.
- Drive continuous improvement initiatives to enhance performance, reliability, and scalability of our data infrastructure.
- Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and translate them into technical solutions.
- Implement best practices for data governance, security, and compliance to ensure the integrity and confidentiality of our data assets.
- Bachelor's or master's degree in computer science, Engineering, or a related field.
- Proven experience (8+) in a data engineering role, with expertise in designing and building data pipelines, ETL processes, and data warehouses.
- Strong proficiency in SQL, Python and Spark programming languages.
- Strong experience with cloud platforms such as AWS, Azure, or GCP is a must.
- Hands-on experience with big data technologies such as Hadoop, Spark, Kafka, and distributed computing frameworks.
- Knowledge of data lake and data warehouse solutions, including Databricks, Snowflake, Amazon Redshift, Google BigQuery, Azure Data Factory, Airflow etc.
- Experience in implementing CI/CD pipelines for automating build, test, and deployment processes.
- Solid understanding of data modeling concepts, data warehousing architectures, and data management best practices.
- Excellent communication and leadership skills, with the ability to effectively collaborate with cross-functional teams and drive consensus on technical decisions.
- Relevant certifications (e.g., Azure, databricks, snowflake) would be a plus.
Lead Data Engineer - Charlotte, United States - Irvine Technology
Description
Position: Lead Data Engineer (Python, SQL, Databricks or Snowflake or Fabric, and Spark is nice to have)
Location: Charlotte, NC or Tempe, AZ (Hybrid Model - 3 Remote / 2 Onsite)
Duration: 6+ month contract, will go longer
Hourly Rate: 80/hr to 90/hr
Industry Experience: Retail (nice to have, not required)
Do you have the skills to fill this role Read the complete details below, and make your application today.
What You Will Do:
What Gets You The Job:
Please send your resume to Dave Lim, Senior Technical Recruiter for immediate consideration.
Irvine Technology Corporation (ITC) is a leading provider of technology and staffing solutions for IT, Security, Engineering, and Interactive Design disciplines servicing startups to enterprise clients, nationally. We pride ourselves in the ability to introduce you to our intimate network of business and technology leaders – bringing you opportunity coupled with personal growth, and professional development Join us. Let us catapult your career
Irvine Technology Corporation provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability or genetics. In addition to federal law requirements, Irvine Technology Corporation complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities.