Data Bricks Engineer - Jackson, United States - Resource Logistics

    Resource Logistics
    Resource Logistics Jackson, United States

    1 month ago

    Default job background
    Description
    Data Bricks Engineer

    Location :
    Jackson Michigan (onsite)

    Contract


    Note:
    Primary Skill is Data Bricks (It include all the skill sets like Python, Py-Spark, SQL, ADF). Refer the Technical areas/responsibilities defined for this role.

    Additionally, the skill sets have been mentioned explicitly (ADF, SQL, Python, Py-spark) so that it will be easy for you to look at the high level skill sets in the profile of the Candidates.


    Sr.
    No Skill Technical Area Must Have/Good to have Proficiency Rating (1 to 5)     1 Data Bricks

    Strong hands on in Pyspark and Apache Spark


    Strong hands on in Medallion Clienthitecture


    Experience in Native Spark Migration to Databricks.


    Experience in Building Data Governance Solutions like Unity Catalog, Clienture Purview etc.


    Highly experienced in Usability Optimization (Auto Compaction, ZOrdering, Vaccuming), Cost Optimization and Performance Optimization.


    Build Very Strong Orchestration Layer in Databricks/ADF.... Workflows.


    Build CICD for Databricks in Clienture Devops.


    Process near Real time Data thru Auto Loader, DLT Pipelines.


    Implement Security Layer in Delta Lake.


    Implement Massive Parallel Processing Layers in Spark SQL and PySpark.


    Implement Cost effective Infrastructure in Databricks.


    Experience In extracting logic and from on prem layers, SAP, ADLS into Pyspark/ADLS using ADF/Databricks. Must Have (Primary Skill Clienture Synapse Analytics/Clienture data Factory (ADF) - Hands on Experience in Clienture Synapse Analytics, Clienture Data Factory and Data Bricks, Clienture Storage, Clienture Key Vault, SQL Pools CI/CD Pipeline Designing and other Clienture services like functions, logic apps

    • Linked services, Various Runtimes, Datasets, Pipelines, Activities
    • Strong Hands on Experience in Various Activites like Control flow logic and conditions (For Each, if, switch, until), Lookup, Stored procedure, scripts, validations, Copy Data, Data flow, Clienture functions, Notebooks, SQL Pool Stored procedures and etc
    • Strong hands on exp in deployment of code through out landscape (Dev > QA > Prod), Git Hub, CI/CD pipelines and etc Must Have 45 3 SQL Server stored procedures strong hands on creating the SQL stored procedures
    • Functions, Stored Procedures, how to call one SP into another, How to process recordbyrecord
    • Dynamic SQL Must have 45 4 Python
    • Must have strong background about the Python libraries like PySpark, Pandas, NumPy, pymysql, Oracle, Pyspark libraries
    • Must have strong hands on to get data through APIs
    • Must be able to install libraries and help users to troubleshoot issues
    • Must have knowledge to get the data through stored procedures via Python
    • Should be able to debug the Python code Must have 45 5 Sparks
    • Hands on experioence in Spark Pools, PySpark
    • Should be able to merge data/delta loads through Notebooks
    • Must have strong background about the Python libraries and PySpark Must have 45