- Defines data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.
- Supports the standardization, customization and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.
- Creates data policy and develop interfaces and retention models which requires synthesizing or anonymizing data.
- Implements statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.
- Develops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.
- Works closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.
- Builds data pipelines that clean, transform, and aggregate data from disparate sources.
- Employs a variety of languages and tools (e.g. scripting languages) to marry systems together.
- Applies knowledge of Data Architecture components, leads project teams from requirements to implementation.
- 3-5 years of being in a Data Engineering role manipulating, processing, and extracting value from large datasets.
- 3+ years of Big Data tools like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS.
- 3+ years building, testing, and optimizing 'Big Data' data ingestion pipelines, architectures, and data sets with Tibco, IBM or others.
- High-velocity high-volume stream processing with Apache Kafka and Spark Streaming.
- Strong SQL skills with ability to write intermediate complexity queries.
- ETL experience with PySpark, Spark SQL or similar.
- Agile Scrum, Kanban or SAFe experience.
- Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog.
- Python (and/or Scala) and PySpark/Scala-Spark.
- Database solutions like Snowflake, Kudu/Impala, Delta Lake or BigQuery.
- NoSQL databases, including HBASE and/or Cassandra.
- Azure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.
- Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
-
Data Engineer
1 week ago
Crimson Solutions Fulton, United StatesREMONDIS IT Services is the central IT service provider for the REMONDIS Group, as well as a manufacturer and provider of software for internal and external customers. The task of REMONDIS IT Services is to support all companies in the group with digital solutions to fulfill thei ...
-
Sr Data Warehouse Engineer
4 days ago
AssuranceAmerica Atlanta, United StatesJOIN THE ASSURANCEAMERICA TEAM · Do you want to be part of an organization where you are valued, and your ideas and opinions have an impact? Join the AssuranceAmerica team. · For more than 25 years, AssuranceAmerica has provided superior property and casualty insurance products t ...
-
Data Engineer
3 days ago
TechnoGen Inc Atlanta, United StatesTECHNOGEN, Inc. is a Proven Leader in providing full IT Services, Software Development and Solutions for 15 years. · TECHNOGEN is a Small and Woman Owned Minority Business with GSA Advantage Certification. We have offices in VA; MD and Offshore development centers in India. We h ...
-
Data Engineer
5 days ago
Georgia IT Inc Atlanta, United StatesJob Title - Azure Data Engineer with Python & Scala · Location - tlanta GA (Remote) · Duration - 6 months to 1 year · Rate - DOE · Work Authorization - USC, GC, GC-EAD,H4EAD , L2EAD, TN · Job Description · Azure Data Engineer with Python & Scala · We are looking for an e ...
-
Data Engineer
17 hours ago
Visionaire Partners Atlanta, United StatesSr. Data Analytics Engineer (Microsoft stack / Power BI) · Exciting opportunity to join 1st class organization that offers a great work-life balance, outstanding company culture, and amazing benefits Most of our client's employees retire from here, which is the reason for this op ...
-
Data Engineer
5 days ago
9th Way Insignia Atlanta, United States"Company Introduction 9th Way Insignia is a service-disabled veteran-owned small business bringing transformative technology to our government customers so they can achieve their missions. Our specialties include cybersecurity, cloud modernization, software development, data anal ...
-
Data Engineer
1 week ago
United Digestive Atlanta, United StatesJob Details · Job Location · CBO Office - Atlanta, GA · Position Type · Full Time · Education Level · 4 Year Degree · Travel Percentage · As needed for business requirements. · Job Category · Operations · Description · GENERAL SUMMARY OF DUTIES: United Digestive's Data ...
-
Data Engineer
5 days ago
Kion Group AG Atlanta, United StatesWhat we offer: · What We Offer: Career Development · Competitive Compensation and Benefits · Pay Transparency · Global Opportunities · Learn More Here: · Dematic provides equal employment opportunities to all employees and applicants for employment and prohibits discrimina ...
-
Data Engineer
1 week ago
Microsoft Corporation Atlanta, United StatesCloud Operations + Innovation (CO+I) is the team behind one of the World's largest Cloud Infrastructures, responsible for powering all Microsoft online Products and Services as well as powering Microsoft's "Cloud First" mission. Our focus is on smart growth, high efficiency, and ...
-
Data Engineer
1 week ago
Comprehensive Staffing Solutions of Georgia Atlanta, United StatesJob Description · Requirements · 5-6 years of data engineering experience · Experience working on a data warehouse to cloud migration project · Experience with Oracle and SQL server · Experience designing, developing, and maintaining software solutions on ETL process using I ...
-
Data Engineer
1 week ago
Nlb Services Atlanta, United StatesBusiness Unit: ITS/Enterprise Data Services · Department: 874 · Location: ATL/Admin · Number of approved openings: x · Job Title & Code: Data Analyst (Data Engineer) · Job Description: · The Opportunity · The Data Analyst will be a key part of the Enterprise Data Services team, w ...
-
Data Engineer
5 days ago
Zenput Atlanta, United StatesGlobal restaurant brands run their operation on the Crunchtime platform. Delivering a consistent guest experience across every location and managing food and labor costs are at the core of how Crunchtime's software is used today in over 125,000 locations across 100+ countries by ...
-
Data Engineer
1 day ago
AbbVie Atlanta, United StatesJob Description · Job DescriptionCompany Description · AbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives acr ...
-
Data Engineer
2 weeks ago
Armavel, LLC Atlanta, United StatesJob Description · Job Description · Key Responsibilities: · Meet with clients or management to understand data requirements and needs. · Collaborate with cross-functional teams, including data engineers and business analysts, to define data requirements and ensure data-driven d ...
-
Data Engineer
1 week ago
Xebia Atlanta, United StatesThe Opportunity The Data Analyst will be a key part of the Enterprise Data Services team, which is responsible for transforming data from disparate systems to provide insights and analytics for business stakeholders. Specifically, this role will support a multi-year project to mi ...
-
Data Engineers
3 days ago
Further Atlanta, United StatesJob Description · Job DescriptionIf you love data and are looking for unlimited growth opportunities, we would love to talk with you about joining Further in the near future. · Being a professional services organization, we are always one week away from our next potential deal cl ...
-
Data Engineer
4 days ago
Currency Cloud Atlanta, GA, United StatesCompany DescriptionVisa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to c ...
-
Data Engineer
1 week ago
Pyramid Consulting, Inc Atlanta, United StatesImmediate need for a talented Data Engineer. This is a 12+ Months Contract opportunity with long-term potential and is located in Atlanta GA(Hybrid). Please review the job description below and contact me ASAP if you are interested. · Job ID: · Pay Range: $70 - $75/hour. Employe ...
-
Data Engineer
1 week ago
Visionaire Partners Atlanta, United StatesSr. Data Analytics Engineer (Microsoft stack / Power BI) · Exciting opportunity to join 1st class organization that offers a great work-life balance, outstanding company culture, and amazing benefits As a part of the Analytics Engineering team, you will be responsible for advance ...
-
Data Engineer
2 weeks ago
Visionaire Partners Atlanta, United StatesSr. Data Analytics Engineer (Microsoft stack / Power BI) · Exciting opportunity to join 1st class organization that offers a great work-life balance, outstanding company culture, and amazing benefits As a part of the Analytics Engineering team, you will be responsible for advanc ...
Big Data Engineer - Atlanta, United States - Norfolk Southern Corp.
Description
Norfolk Southern offers a unique opportunity to be part of our proud legacy that spans nearly 200 years. We are a customer-centric, operations-driven team dedicated to advancing safety, serving communities, and driving innovation for tomorrow's rail. As part of Norfolk Southern, you'll join a collaborative team where there are opportunities for growth across the organization. We are building a culture where everyone can thrive by owning and driving exceptional results, being humble and leading with trust, serving our customers with excellence, and collaborating and coaching to win.
Job Description
Norfolk Southern Corporation is currently seeking a Big Data Engineer with an affinity for working with others to create successful solutions. Join a smart, highly skilled team with a passion for technology, where you will work on our state-of-the-art Big Data Platforms. This Data Engineer with a balance of theoretical business intelligence and data analytics knowledge and hands-on experience and exposure to real-world BI problem-solving. In this role, you will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties.
In this role, you will work with people from many areas of the company to understand their data and BI needs and perform detailed requirements gathering along with analysis in partnership with our team of Data Modelers. Engage our BI development team to agree on effort estimates and then oversee the progress of the project through its completion.
Responsibilities:
Skills Required
Skills Desired
Education Desired
Bachelor's Degree, preferably in Information Systems, Computer Science, Computer Information Systems or related field.
Work Conditions
Environment: Office Remote, ideal candidate must be based within 60 miles Atlanta.
Shift Work: No
On-Call: Yes
Weekend Work: No
Company Overview
Since 1827, Norfolk Southern Corporation (NYSE: NSC) and its predecessor companies have safely moved the goods and materials that drive the U.S. economy. Today, it operates a customer-centric and operations-driven freight transportation network. Committed to furthering sustainability, Norfolk Southern helps its customers avoid 15 million tons of yearly carbon emissions by shipping via rail. Its dedicated team members deliver more than 7 million carloads annually, from agriculture to consumer goods, and is the largest rail shipper of auto products and metals in North America. Norfolk Southern also has the most extensive intermodal network in the eastern U.S., serving a majority of the country's population and manufacturing base, with connections to every major container port on the Atlantic coast as well as the Gulf of Mexico and Great Lakes. Learn more by visiting
At Norfolk Southern, we believe in celebrating our individuality. By leveraging the unique backgrounds and viewpoints of our employees, we can create a culture of innovation, respect, and inclusion. We know that employees thrive in a workplace where differing viewpoints, ideas, and experiences are freely shared and valued. As such, we encourage all employees to contribute their distinctive skills and capabilities to our organization.
Equal employment opportunities are available to all applicants regardless of race, color, religion, age, sex, national origin, disability status, genetic information, veteran status, sexual orientation, and gender identity. Together, we power progress.