No more applications are being accepted for this job
- GCP, BigQuery, Cloud Dataproc, Composer, CloudSQL, Airflow
- PySpark, Scala, Python, SQL
- HDFS, Hive
- Job title SENIOR DATA ENGINEER 2.Job summary Perform the Data Engineer activities 3.Experience 8to12Yrs 4.Required Skills ,BigQuery,Cloud Dataproc,Airflow 5.Nice to have skills ,PySpark,GCP CloudSQL 6.Technology -Not Applicable 7.Shift Day 8.Roles & Responsibilities Analyzes complex data structures from disparate data sources and design large scale data engineering pipeline
- Develops large scale data structures and pipelines to organize, collect and standardize data that helps generate insights and addresses reporting needs
- Collaborates with product business and data science team to collect user stories and translate into technical specifications
- Uses knowledge in Cloud & Hadoop architecture, HDFS commands and experience designing & optimizing queries to build data pipelines
- Uses strong programming skills in PySpark, Python, Java or any of the major languages to build robust data pipelines and dynamic systems
- Builds highly scalable and extensible data marts and data models to support Data Science and other internal customers on Cloud. Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards.
- Analyzes current information technology environments to identify and assess critical capabilities and recommend solutions
- Experiments with available tools and advice on new tools to determine optimal solution given the requirements dictated by the model/use case Required Qualifications
- 3+ years of progressively complex related experience in cloud data engineering and data analysis
- Knowledge in programing languages such as PySpark, Java, Python, Hive, SQL
- Knowledge in Cloud Technology, Hadoop architecture, HDFS commands and experience designing & optimizing queries against data in the HDFS environment
- Strong knowledge of large-scale search applications and building high volume data pipelines, preferably using Dataproc, composer services on GCP or other Cloud Platforms 2+ years of development experience in GCP and native tools & services such as DataProc, Composer, BigQuery, Airflow 2 to 3+ years of PySpark or Scala or Python programing proficient in SQL 3+ years of working with data & data warehouse experience
senior data engineer - Dallas, United States - Diverse Lynx
Description
Onsite - Dallas, TXAll applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role.
We promote and support a diverse workforce across all levels in the company.