Azure Databricks - Auburn Hills, United States - Diverse Lynx

Diverse Lynx Auburn Hills, United States

3 weeks ago

Description

Azure Databricks
Auburn Hills, MI

Develop deep understanding of the data sources, implement data standards, and maintain data quality and master data management.
Expert in building Databricks notebooks in extracting the data from various source systems like DB2, Teradata and perform data cleansing, data wrangling, data ETL processing and loading to AZURE SQL DB.
Expert in building Ephemeral Notebooks in Databricks like wrapper, driver and config for processing the data, back feeding the data to DB2 using multiprocessing thread pool.
Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data.
Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data.
Performed ETL operations in Azure Databricks by connecting to different relational database source systems using jdbc connectors.
Developed Python scripts to do file validations in Databricks and automated the process using ADF.
Analyzed the SQL scripts and designed it by using Pyspark SQL for faster performance.
Worked on reading and writing multiple data formats like JSON, Parquet, and delta from various sources using Pyspark.
Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Azure SQL DB.
Expert in optimizing the Pyspark jobs to run on different Cluster for faster data processing.
Developed spark applications in python (Pyspark) on distributed environment to load huge number of CSV files with different schema in to Pyspark Dataframes and process them to reload in to Azure SQL DB tables.
Analyzed data where it lives by Mounting Azure Data Lake and Blob to Databricks.
Used Logic App to take decisional actions based on the workflow and developed custom alerts using Azure Data Factory, SQLDB and Logic App.
Developed Databricks ETL pipelines using notebooks, Spark Dataframes, SPARK SQL and python scripting.
Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Good Knowledge and exposure to the Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks.
Involved in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
Expert in understanding current production state of application and determine the impact of new implementation on existing business processes.
Involved in Migration of data from On-prem server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
Good Hands on experience in setting up Azure infrastructure like storage accounts, integration runtime, service principal id, and app registrations to enable scalable and optimized utilization of business user analytical requirements in Azure.
Expert in ingesting streaming data with Databricks Delta tables and Delta Lake to enable ACID transaction logging.
Expert in building Delta Lake On top Of Data Lake and performing transformations in Delta Lake.
Expert in implementation of distributed stream processing platform with low latency and seamless integration, with data and analytics services inside and outside Azure to build your complete big data pipeline.
Expert in performance tuning of delta lake (optimize, rollback, cloning, time travel) implementation.
Developed complex SQL queries using stored procedures, common table expressions (CTEs), temporary table to support Power BI reports.
Development level experience in Microsoft Azure providing data movement and scheduling functionality to cloud-based technologies such as Azure Blob Storage and Azure SQL Database.
Independently manage development of ETL processes - development to delivery.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

Azure Databricks

2 weeks ago

Diverse Lynx Auburn Hills, United States

Job Description :: Develop deep understanding of the data sources, implement data standards, and maintain data quality and master data management. · Expert in building Databricks notebooks in extracting the data from various source systems like DB2, Teradata and perform data c ...
AI Data Engineer

1 day ago

Hearst Troy, United States

Job Description · MOTOR Information Systems, an operating group of Hearst, is actively seeking a team of enthusiastic, creative, and driven technologists. People with a proven track record, who are willing to experiment with new ideas, invest time in them, fail-fast, and move on ...
Solution Architect

1 week ago

HTC Global Services, Inc. Troy, United States

HTC Global Services, Inc. has an opening for the position of Solution Architect (Ref# SA510AS) to work in Troy, MI and various unanticipated locations throughout the U.S. · JOB DUTIES: Define, design, and deliver architecture-based solutions to application development projects t ...
Data Engineer

1 week ago

TEKsystems Troy, United States

: Knowledge in Data Engineering with DataBricks, Hadoop and spark SQL to ensure code is optimized as per request if needed. c. Review BI product to ensure that the requirements are met d. Validate data e. Quick Daily Stand up and see any Open issues or blockers team is facing c. ...
Data Engineer

2 days ago

Unified Tech Group Inc Bloomfield Hills, United States

Job Description · Job DescriptionSalary: · Job Title: Data Engineer · Job Duties: Perform full life cycle of data project architecture implementation. Design, code and debug database and business intelligence applications in various platforms including Oracle and MS SQL Server. ...
Lead Cloud Engineer

2 weeks ago

FordDirect Pontiac, United States

At FordDirect, our emphasis on diversity is much more than words. We encourage you to be yourself. We open our minds and doors to all walks of life, respecting and celebrating the uniqueness within each of us. We value the voice and contributions of all team members. · You could ...
Data Quality Engineer, Data Warehousing

3 weeks ago

Heitmeyer Consulting Farmington Hills, United States Contract

Job Summary · : · Heitmeyer is seeking a Data Quality Engineer to support its global banking client on a long-term regulatory and cloud initiative. This role is required to be on-site 2 days a week in the following locations: Mount Laurel, NJ Greenville, SC, Falmouth, ME, Farmin ...
Future Opportunity- Senior Azure Data Engineer

3 weeks ago

Avanade Detroit, United States

RECRUITING FOR THE FUTUREThis posting is for a future job opportunity. To support the needs of our clients, Avanade is actively recruiting and interviewing for the role outlined below, as we continue to build on another year of growth in our business. Now, with more than 6, emplo ...
Sr. Data Scientist

3 weeks ago

Carhartt us Dearborn, United States

Summary · The Sr. Data Scientist will drive Carhartt's analytics direction to help us gain a deeper understanding of every stage of our business from demand to supply and the optimization of everything in between. This position will partner with business leaders and project own ...
Lead Cloud Data Engineer

3 weeks ago

Blue Chip Talent Bloomfield Hills, United States

Job Summary: · We're seeking a skilled Lead Cloud Engineer specialized in Azure to spearhead the development and deployment of robust, high-performance data engineering solutions utilizing Azure Databricks and Azure Data Factory. · Primary Responsibilities: · Lead the conceptu ...
Sr. Data

2 weeks ago

Saxon Global Farmington, United States

This is a 18 month contract with Daimler Financial Service. Must be onsite 2-3 days in FARMINGTON HILLS, MI. Linkedin. F2F Interview required. Only locals. Any visa. · Data & Analytics Data Engineer - Senior · The Data & Analytics Data Engineer is responsible to support both D ...
Data Engineer

3 weeks ago

A-Line Staffing Solutions Troy, United States

Position: Data Engineer · Direct Hire Opportunity · This is NOT a C2C Position · Must be located outside of: TX, TN, or GA · Required Experience (must Have) · Minimum 5+ years data engineering experience working with designing data models in both column store and relational ...
Cloud Data

1 week ago

PwC Detroit, United States

Specialty/Competency: Data, Analytics & AI · Industry/Sector: Not Applicable · Time Type: Full time · Travel Requirements: Up to 80% · A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive bus ...
Data Engineer

3 weeks ago

MSX INTERNATIONAL Southfield, United States

Title: Data Engineer · Location: Hybrid/Southfield, MI, US · Job Type: Full-time with Benefits · Who We Are · For more than 25 years, MSX International has been a dedicated partner to leading automotive brands around the world. We support them in transforming their businesses an ...
Senior Manager, Data Platform Engineering

4 days ago

Scout Motors Novi, United States

Scout Motors · Here at Scout Motors, we're carrying forward the heritage of one of the most iconic American vehicles in history. A vehicle dating back to 1960. One that forged the path for future generations of rugged SUVs and will do so once again. · But Scout is more than jus ...
Senior Java/Python Developer

1 week ago

General Motors Warren, United States

Job Description · Hybrid: This role is categorized as hybrid. This means the successful candidate is expected to report onsite at the Austin IT Innovation Center North - Austin IT Innovation Center North or the GM Global Technical Center - Cole Engineering Center Tower three time ...
Lead Cloud Data Engineer

2 weeks ago

ITC Infotech Detroit, United States

ITC Infotech is looking for a · Lead Cloud Data Engineer · to join the · US team. · Location: Hybrid – Detroit, MI (Hybrid) · Rate - $85000 per annum plus benefits. · Your X-Factor · Work ethic · - You are a consummate professional. · Aptitude · - You have an innate capacity ...
Lead Cloud Data Engineer

3 weeks ago

ITC Infotech Detroit, United States

ITC Infotech is looking for a Lead Cloud Data Engineer to join the US team. · Make your application after reading the following skill and qualification requirements for this position. · Location: Hybrid – Detroit, MI (Hybrid) · Rate - $85000 per annum plus benefits. · Your X- ...
Resident Solutions Architect, Healthcare

3 weeks ago

Databricks Detroit, United States

CSQ424R222 · While candidates in the listed location(s) are encouraged for this role, candidates in the US will be considered located anywhere in Central time zone. · As a Resident Solutions Architect, you will have the opportunity to shape the future big data landscape for lea ...
Senior Data Engineer

3 weeks ago

DCTek Inc. Detroit, United States

Requirement Details · Position: Senior Data Engineer · Location: Detroit, MI (Hybrid) · Duration: Long-Term · Description: · Experience in developing data architectures including data ingestion, pipeline design, data mining and advanced data processing frameworks. · Able to as ...