Data Engineer - Hartford, United States - Highbrow

Description

• This job opportunity is for employment type W2 .
• Candidates who are willing to apply must hold US work authorization.

Job Title:
Data Engineer


Job ID:

Job Location:
Hartford, CT

Job Travel Location(s):

# Positions:

2


Employment Type:
W2


Candidate Constraints:
# of Layers:

0


Work Eligibility:
All Work Authorizations are Permitted – No Visa Transfers


Key Technology:
Pyspark, Python, AWS services, Teradata Vantage, CI/CD technologies, Terraform, SQL


Skills and Experience Required:
Good experience on designing and developing data pipelines for data ingestion and transformation using Spark.

• Distributed computing experience using Pyspark.
• Good understanding of spark framework and spark architecture.
• Experience working in Cloud based big data infrastructure.
• Excellent in trouble shooting the performance and data skew issues.
• Must have good understanding of spark run time metrics and tune applications based on metrics.
• Deep knowledge in partitioning, bucketing concepts of data ingestion.
• Good understanding of AWS services like Glue, Athena, S3, Lambda, Cloud formation.
• Preferred working knowledge on the implementation of datalake ETL using AWS glue, Databricks etc.
• Experience with data modelling techniques for cloud data stores and on prem databases like Teradata, Teradata Vantage (TDV)etc
• Preferred working experience in ETL development in Teradata vantage and data migration from on prem to Teradata vantage.
• Proficiency in SQL, relational and non-relational databases, query optimization and data modelling.
• Experience with source code control systems like Gitlab.
• Experience with large scale distributed relational and NoSQL database systems.

Technologies:
Pyspark, Python, AWS services, Teradata Vantage, CI/CD technologies, Terraform, SQL

#J-18808-Ljbffr