Data Engineer - Fort Worth, United States - Motion Recruitment

    Motion Recruitment
    Motion Recruitment Fort Worth, United States

    1 month ago

    Motion Recruitment background
    Accounting / Finance
    Description

    Data Engineer

    Fort Worth Texas (HYBRID)

    Contract to perm role

    Job Description

    JOB DUTIES


    • Code, test, deploy, Orchestrate, monitor, document and implement data pipelines utilizing the Microsoft azure data bricks platform


    • Utilize technologies like Spark, Delta Lake and Unity Catalog for Efficient data processing and governance


    • Implement the Medallion architecture pattern for robust and modular data pipelines


    • Implement CI/CD pipelines for automated testing and deployment of data pipelines using azure devops.


    • Configure and manage Azure Data Lake Storage for secure and scalable data storage.


    • Collaborate and work closely with ABS data analysts, accounting business partners to identify, capture, collect and format data from the external sources, Internal systems and data warehouse to extract features of interest


    • Monitor and optimize data pipelines for performance and reliability


    • Stay up-to-date with the latest advancements in Databricks, Azure, and big data technologies


    • Educate and Develop ETL developers on data engineering cloud based initiatives so as to enable transition to data engineer and practice


    • Monitor the production reporting environment on daily and monthly basis Data Analytics to analyze data and make business decisions on the data

    Qualifications:

    What makes you a dream candidate?


    • Experience with Databricks platform on Microsoft Azure


    • Experience with processing large datasets using Hadoop, Spark, Kafka or similar distributed systems


    • Experience with PySpark and distributed data processing frameworks


    • Understanding of cloud computing technologies, business drivers and emerging computing trends


    • Experience with ingesting various source data formats such as JASON, Parquet, Sequencefile, Cloud databases, MQ, Relational databases such as Oracle


    • Working knowledge of Object Storage technologies to include but not limited to Data Lake Storage Gen2, ADLS etc.


    • Working knowledge of unity catalog and data governance principles


    • Working knowledge of Agile development /SAFe, Scrum and Application Lifecycle Management


    • Strong background with source control management systems (GIT or Subversion); Build Systems (Maven, Gradle, Webpack); Code Quality (Sonar); Artifact Repository Managers (Artifactory), Continuous Integration/ Continuous Deployment (Azure DevOps)


    • Experience with Python and SQL


    • Creating and maintaining ETL processes and data pipelines


    • Knowledgeable of best practices in information technology governance and privacy compliance


    • Knowledge on auto finance industries core business functions


    • Ability to establish priorities and meet tight deadlines.


    • Ability to interact collaboratively with internal customers


    • Demonstrated quantitative skills and data analytical skills


    • Solid design, coding, testing and debugging skills

    Experience and Education


    • 4-6 years of hands-on experience with data engineering required


    • 3-5 years of hands-on experience with processing large data sets required


    • 3-5 years of hands-on experience with SQL, data modeling, relational databases and/or no SQL databases required


    • Bachelor's Degree in related field or equivalent work experience required.

    Thanks,

    Safina