Data Engineer@Minneapolis, MN - Diverse Lynx

    Diverse Lynx
    Diverse Lynx Minneapolis, United States

    2 weeks ago

    Default job background
    Description
    Title Data Engineer (Streaming Platforms ) Duration 9+ Months Location Minneapolis, MN
    customer is doing near real-time data integration. They do streaming analytics and they emphasize on Live Table Structured streaming, Kafka, Event Hub, Apache Spark . These technologies are definitely required to do near real-time data integration.

    Job Description :
    • Streaming Platforms Mandatory
    • BS, MS, or PhD in Computer Science, Information Technology, Management Information Systems (MIS), Data Science or related field
    • 2+ years of experience in Apache Spark (PySpark / Spark SQL)
    • 2+ years of experience in Python
    • Experience with large scale streaming platforms (e.g. Kafka, Event Hub, Databricks Live Table / Structured Streaming), processing frameworks (e.g. Spark, Databricks) and storage engines stream analytics
    • 3+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes to provide quality data and analytics solutions
    • 3+ years of experience in SQL with designing complex data schemas and query performance optimization
    • Highly proficient working in Azure cloud environment (e.g. Blob / ADLS, Databricks, Azure Data Factory, Event Hub)
    • Excellent communication skills ability to communicate technical concepts to both technical and non-technical audiences
    • Experience in Regular Expression, CI/CD technology, Terraform and Git