- Work closely with cross-functional teams, including product managers, data scientists and engineers to understand project requirements and objectives ensuring alignment with overall business goals.
- Build data ingestion framework and data pipelines to ingest unstructured and structured data from various data sources such as SharePoint, Confluence, Chat Bots, Jira, External Sites, etc. into our existing OneData platform.
- Design a scalable target state architecture for data processing-based on document content (Data types may include, but are not limited to: XML, HTML, DOC, PDF, XLS, JPEG, TIFF, and PPT) including PII/CII handling, policy-based hierarchy rules and Metadata tagging.
- Design, development, and deployment of optimal data pipelines including incremental data ingestion strategy by taking advantage of leading-edge technologies through experimentation and iterative refinement.
- Design and implement vector databases to efficiently store and retrieve high-dimensional vectors.
- Conducting research to stay up to date with the latest advancements in generative AI services and identify opportunities to integrate them into our products and services.
- Implement data quality and validation checks to ensure accuracy and consistency of data.
- Build automation that effectively and repeatably ensures quality, security, integrity, and maintainability of our solutions.
- Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks and issues.
- Define and implement data access policies; implement and maintain data security measures and access policies for cloud storage buckets and vector databases. Experience:
- + years relevant industry and functional experience in Database and Cloud-based technologies
- Experience in working with Machine learning and AI concepts related to RAG architecture, LLMSs, embedding and data insertion into a Vector data store.
- Experience in building data ingestion pipelines for Structured and Unstructured data both for storage and optimal retrieval
- Experience working with Cloud data stores, noSQL, Graph and Vector databases.Proficiency with languages such as Python, SQL, and PySpark
- Experience working with Databricks and Snowflake technologies.
- Experience with relevant code repository and project tools such as GitHub, JIRA and Confluence
- Working experience with Continuous Integration & Continuous Deployment with hands-on expertise on Jenkins, Terraform, Splunk and Dynatrace.
- Highly innovative with aptitude for foresight, systems thinking and design thinking, with a bias towards simplifying processes.
- Detail oriented individual with strong analytical, problem-solving, and organizational skills
- Ability to clearly communicate to both technical and business teams.LLM, RAG, Data engineer, Python, Pyspark, Machine learning, Gen AI, AI/ML, Vector database, Databricks, Snowflake, Education:
- Bachelor's degree in engineering, Computer Science or a related field; Master's degree is a plus.
-
Data Engineering
2 weeks ago
Understood New York, United States**About Understood**: · Understood is a nonprofit focused on shaping the world for difference. We raise awareness of the challenges and strengths of people who learn and think differently. Our resources help people navigate challenges, gain confidence, and find support and commun ...
-
Data Engineer
3 weeks ago
Wallenius Wilhelmsen Parsippany, United StatesOverview · DATA ENGINEER · Dreaming of a career that combines your passion for data with cutting-edge technology? Look no further · Wallenius Wilhelmsen is seeking a Data Engineer to join our dynamic Data Services Team. · *Please note this is a hybrid role requiring the employ ...
-
Data Engineer
3 weeks ago
Diverse Lynx Parsippany, United StatesJob Title Data Engineer Skill Snowflake Minimum Experience 4 - 6 Years Qualification BACHELOR OF COMPUTER SCIENCE Location Newport Beach, C · Job Description : · Skill: Python, Snowflake, AWS, git, docker, azure DevOps, Control-M Enhance and maintain Python based ETL fr ...
-
Engineering Contractor, Data
1 week ago
EQUINOX New York, United States Freelance**Engineering Contractor, Data**: · Overview · **OUR STORY** · Equinox Group is a high growth collective of the world's most influential, experiential, and differentiated lifestyle brands. We restlessly seek what is next for maximizing life - and boldly grow the lifestyle brands ...
-
Specialist - Data Engineering
2 weeks ago
LTIMindtree New York, United States**Job Req Id**:1287937 · - **About Us**: · - **Job Title**:Specialist - Data Engineering (Geneva Developer) · - **Work Location**: New York, NY days onsite/week) · experience as developer. · - Strong functional experience in financial industry and Portfolio Accounting System Gene ...
-
Data Engineering Director
2 weeks ago
FanDuel New York, United States**ABOUT FANDUEL** · FanDuel Group ("FanDuel") is an innovative sports-tech entertainment company that is changing the way consumers engage with their favorite sports, teams, and leagues. The premier gaming destination in the United States, FanDuel consists of a portfolio of leadi ...
-
GCP Data Engineer
2 weeks ago
Datalore Technologies Inc Parsippany, United StatesPosition: GCP Data Engineer · Location: Parsippany NJ (Hybrid) – weekly 3 days at client location · Duration: Long Term contract · Need Data Engineer with GCP Experience along with the below Primary Skills: · Python · Airflow · Cloud experience preferably with GCP · Version contr ...
-
GCP Data Engineer
1 week ago
Saxon Global Parsippany, United StatesHello , · Greetings of the day · Hope you are doing great · We have a job opportunity for the position GCP Data Engineer with one of our client. · If you are interested, please share your latest resume to or reach me · It is highly appreciated if you refer anyone who is availab ...
-
AWS Data Engineer
2 weeks ago
Iris Software Inc. Parsippany, United StatesIRIS direct end client which is one of the leading American provider of human resources management software and services is urgently looking to hire AWS Data Engineer – Parsippany, NJ Hybrid - 3 days per week). This is a Contract to hire Role opportunity. · AWS Data Engineer · Pa ...
-
Principal Data Engineer
2 weeks ago
US Tech Solutions Parsippany, United StatesResponsibilities: · Work closely with cross-functional teams, including product managers, data scientists and engineers to understand project requirements and objectives ensuring alignment with overall business goals. · Build data ingestion framework and data pipelines to ingest ...
-
GCP Data Engineer
2 weeks ago
Avance Consulting Parsippany, United StatesJob Description · Job DescriptionGCP Data Engineer · Location: Parsippany / New Jersey · Tech stack: BigQuery , any ETL tool (Informatica, Talend, DataStage), Dataflow, Dataproc · • 5 years Experience in Data warehouse and Data lake implementation · • 3 years of experience in Goo ...
-
Principal Data Engineer
3 days ago
ADP Parsippany, United StatesJob Description · ADP is hiring a Principal Data Engineer. · Are you empathetic to client needs and inspired by transformation and impacting the lives of millions of people every day? · Are you looking to join a dynamic, inclusive team environment with a culture of collaboration ...
-
Principal Data Engineer
3 weeks ago
Apex Systems Parsippany-Troy Hills, United States ContractJob#: · Job Description: · Apex Systems combines with parent company ASGN Inc. to make it the 2nd largest IT staffing agency in the country. · Apex has an opportunity open for a Principal Data Engineer. If interested in discussing the position further, please send an MS Word v ...
-
Cloud Data Engineer
2 weeks ago
QData Parsippany, United States contractData gets extracted using DataStage using FTP. The predictive modeling is done using R Python and Linux scripting.Primary Skills Python R SQL Datastage Linux scripting API knowledge SSIS. Supporting skills - Azure platform knowledge and Partykit. Experience on Azure Data factory ...
-
Junior Data Engineering Specialist
1 week ago
STV Group New York, United StatesWe are seeking a talented and motivated Junior Data Engineering Specialist to join our dynamic Digital Advisory team. As a Junior Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines and systems to support our data-driven ...
-
Cloud Data Engineer
6 days ago
QData Parsippany, United States contractHiHope you are doing good...We have an urgent requirement below please go through Job description and send your updated profile and expected rate ASAP.Please reach me at .comJob Title Cloud Data EngineerJob Location Parsippany NJJob Description Data gets extracted using DataStage ...
-
Cloud Data Engineer
2 weeks ago
QData Parsippany, United States contractResponsibilities Create and maintain optimal data pipeline architecture Assemble large complex data sets that meet functional / non-functional business requirements. Identify design and implement internal process improvements automating manual processes optimizing data delivery r ...
-
Data Engineer
1 week ago
MDMS Recruiting LLC Morristown, United StatesJob Description · Job DescriptionThis is a remote position. · Data Engineer · Key Objectives/Responsibilities: · A strong understanding of common data integration challenges within the insurance industry, as well as the best practice technical approaches & solutions available thr ...
-
Engineering/data Analytics Manager
5 days ago
Renaissance Global Services New Brunswick, United StatesOverview: · **Responsibilities**: · v **Ops Readiness/KPI Responsibilities**: Engineering/Business Analytics/Business degree. · - Review current state of the Ops Readiness program and identify fundamental principles to keep in place. · - Develop deliverables for site Ops Readines ...
-
Lead Data and AI Platform Engineer
15 hours ago
ADP Parsippany, United States Full timeADP is hiring a Lead Data and AI Platform Engineer. · Are you empathetic to client needs and inspired by transformation and impacting the lives of millions of people every day? · Are you looking to join a dynamic, inclusive team environment with a culture of collaboration and bel ...
Principal Data Engineer - Parsippany-Troy Hills, United States - US Tech Solutions
