Azure Databricks Lead - Decatur

Job Title- Azure Databricks Lead · Location- Decatur, IL (Onsite) · Job Summary · We are looking for a Data Replication Specialist with expertise in Microsoft Fabric and Databricks Lake flow connect to design implement and manage data replication and synchronization solutions acr ...

Job description

Job Title- Azure Databricks Lead
Location- Decatur, IL (Onsite)
Job Summary
We are looking for a Data Replication Specialist with expertise in Microsoft Fabric and Databricks Lake flow connect to design implement and manage data replication and synchronization solutions across multiple data environments The ideal candidate will ensure that business critical data is consistently available accurate and up to date across systems for analytics reporting and operational needs
Key Responsibilities
Design and implement data replication strategies using Microsoft Fabric Copy Data and Mirroring features
Configure and maintain replication pipelines between on-premises and cloud based data sources eg SQL Server Azure SQL Synapse Data Lake Power BI
Monitor and optimize replication performance latency and reliability
Troubleshoot replication failures data drift and synchronization issues
Collaborate with data engineers architects and analytics teams to align replication processes with business and data governance requirements
Develop and maintain automation scripts for replication monitoring and ing using PowerShell Python or REST APIs
Document replication architectures data flow diagrams and operational procedures
Implement security compliance and data governance standards within replicated environments
Stay updated on Microsoft Fabric enhancements and best practices for data replication and integration
Design develop and maintain data ingestion pipelines using Databricks and Lake flow Connect
Integrate data from various structured and unstructured sources into Delta Lake and other data storage systems
Implement real-time and batch ingestion workflows to support analytics and reporting needs
Optimize data ingestion performance ensuring scalability reliability and cost efficiency
Collaborate with data architects analysts and business stakeholders to define data requirements and ingestion strategies
Ensure data quality lineage and governance compliance across the ingestion process
Automate data ingestion monitoring and error handling mechanisms
Stay up to date with emerging Databricks Lakehouse and data integration technologies and best practices
Required Qualifications
Bachelors or Masters degree in Computer Science Information Systems Data Engineering or a related field
4 years of experience in data engineering or ETL development
Handson experience with Databricks SQL PySpark Delta Lake
Proficiency with Lake flow Connect for building and managing data ingestion workflows
Strong understanding of data integration patterns data modelling and data Lakehouse architectures
Experience with cloud platforms Azure AWS or GCP and associated data services
Knowledge of CICD version control Git and infrastructure-as-code practices
Familiarity with data governance security and compliance standards
Required Skills Qualifications
Bachelors degree in Computer Science Information Systems or related field
3-5 years of experience in data replication ETLELT or data integration
Handson experience with Microsoft Fabric Azure Data Factory or Synapse Pipelines
Strong understanding of Copy Data and Mirroring capabilities in Microsoft Fabric
Proficiency in SQL and scripting languages PowerShell Python
Experience working with Azure Data Lake OneLake and Power BI
Knowledge of change data capture CDC incremental loads and realtime replication
Familiarity with data governance security and compliance standards eg GDPR HIPAA
Preferred Qualifications
Microsoft Certified Fabric Analytics Engineer Associate or Azure Data Engineer Associate
Experience with hybrid data environments on-premises cloud
Knowledge of data mesh or lake house architectures
Exposure to DevOps or CICD practices for data pipelines