Databricks Architect - New York, New York, United States
3 days ago
Job description
Role: Databricks Architect
Location: Remote
Domain Experience: Insurance
Responsibilities:
- Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
- Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
- Continuously improve systems through performance enhancements and cost reductions in compute and storage
- Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
- Lead Data Engineering Projects to manage and implement data-driven communication systems
- Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
- Integrate data across different systems and platforms
- Strong verbal and written communication skills to manage client discussions
Candidate Profile:
- 8+ years experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud
- Some experience in insurance domain/ data is must
- Programming Languages – SQL, Python
- Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
- Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure DevOps
- Project Management using Agile, Scrum
- B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.
- Excellent communication & leadership skills, with the ability to lead and motivate team members
- Ability to work independently with some level of ambiguity and juggle multiple demands
Similar jobs
+ Lead technical architecture design and delivery of high-performance integrated systems including low-latency trading platforms · + Re-architect and optimize Databricks platforms ensuring efficient integration with existing services · + Design develop sophisticated REST API pl ...
1 month ago
Integration Architect with over 12 years of experience to lead design development and implementation of highly available scalable low-latency integrated systems. · ...
1 month ago
We are seeking an experienced Integration Architect to lead the design, development, and implementation of highly available, scalable, and low-latency integrated systems. · ...
1 month ago
Lead technical architecture design and delivery of high-performance integrated systems including low-latency trading platforms. · ...
1 month ago
Databricks Architect to re-architect Databricks platforms and manage related integrations. · ...
1 month ago
Title: Databricks Architect · Key Requirements: Hands on experience in Databricks, SQL, PySpark. Must have Insurance domain experience. · Location: Jersey City, NJ · Job Type: Hybrid (2 days work from office) · Duration: 12+ Months · Start Date: Immediate · Number of roles: 1 · R ...
17 hours ago
We are looking for an experienced Databricks Data Architect to join a high-performing Data & AI team on an initial 6-month contract, · with a strong chance of extension.Data Platform Architecture ...
3 weeks ago
We are looking for an experienced Databricks Data Architect to join a high-performing Data & AI team on an initial 6-month contract with a strong chance of extension. · Design and own end-to-end Databricks Lakehouse architectures (batch & streaming). · ...
3 weeks ago
· ...
3 weeks ago
We are seeking an Azure Databricks Architect to architect and deliver largescale data platforms using Azure and Databricks. · Architect and implement Azure based data solutions using Databricks Data Factory Synapse Analytics and Data Lake services. · Design build and optimize a M ...
3 weeks ago
· Role & Responsibilities Overview: · Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.) · Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for ...
17 hours ago
We are seeking a skilled Azure Databricks Architect to build and maintain scalable ETL/ELT pipelines using Databricks. · Build and maintain scalable ETL/ELT pipelines using Databricks. · Leverage PySpark/Spark and SQL to transform and process large datasets. · ...
4 days ago
We are seeking an Integration Architect to lead the design development and implementation of highly available scalable and low-latency integrated systems. You will collaborate closely with engineering teams and stakeholders to deliver modern integration solutions focusing on both ...
1 month ago
Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining actively hands-on in design, prototyping, coding, and performance tuning. · ...
1 week ago
Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining actively hands-on in design, prototyping, coding, and performance tuning. Provide high-level technical consultancy and solution design to internal business stakeh ...
1 week ago
We are seeking a highly skilled and motivated Technical Team Lead with extensive experience in Azure Databricks to join our dynamic team. · The ideal candidate will possess a strong technical background, exceptional leadership abilities, and a passion for driving innovative solut ...
3 days ago
We are seeking highly skilled Databrick Architect with strong expertise in MDM SQL Python Datawarehouse Cloud ETL tools to join our data team. · Design implement and optimize large-scale data pipeline ensuring scalability reliability and performance. · ...
1 month ago
Databricks Architect designing develop and maintain data ingestion pipelines using Databricks Lake flow Connect integrating data from various structured unstructured sources into Delta Lake other data storage systems. · ...
3 days ago
We are looking for a highly senior Databricks Architect to lead a large-scale Oracle-to-Databricks migration.The ideal candidate has extensive experience building enterprise-grade data platforms on Databricks and is exceptionally strong in PySpark, Spark SQL. · ...
3 weeks ago
We are seeking highly skilled Databrick Architect with strong expertise in MDM SQL Python Datawarehouse Cloud ETL tools to join our data team. · The ideal candidate will design implement and optimize large-scale data pipeline ensuring scalability reliability and performance. · Th ...
1 month ago