Data Lake Engineer - Champaign, United States - NVIDIA

Description

We are seeking experienced Distributed Systems Engineers to accelerate Apache Spark and related frameworks on GPUs. Apache Spark is the most popular distributed data processing engine in data centers. It is used for a wide variety of workloads, from data preparation, feature generation, reporting, analytics, and more. Data scientists spend a considerable amount of time exploring data and iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features and fit ML algorithms impedes an efficient business workflow. At NVIDIA, we are passionate about working on hard problems that have an impact. You will work with the open source community to enable Apache Spark data processing with GPUs. Data workflows can benefit tremendously from being accelerated, enabling data scientists to explore many more and larger datasets to achieve their business goals, faster and more efficiently.

What you'll be doing:

• Extending the RAPIDS Accelerator for Apache Spark to work seamlessly with data lake table formats – including Delta Lake, Apache Iceberg, and Apache Hudi
• Benchmarking and optimizing data lake table formats at scale with NVIDIA GPUs
• Optimizing I/O operations on table layout formats and backing stores (distributed filesystems, object stores)
• Engaging open source communities, including Apache Spark, RAPIDS, and for technical discussions and contributions around data lake table formats
• Creating a collection of GPU accelerated libraries for data processing, data analytics and ML for compatibility with data lake table formats
• Working with NVIDIA strategic partners on deploying sophisticated data analytics solutions in public cloud or on-premise clusters
• Presenting technical solutions in industry conferences and meetups

What we need to see:

• BS, MS, or PhD in Computer Science, Computer Engineering, or equivalent experience
• 15+ years of work or research experience in software development
• 5+ years working with key open source big-data projects as a contributor or committer including Delta Lake, Apache Iceberg, Apache Hudi, Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka, Apache Storm and Apache Hive
• Understanding of data management architecture and experience with table formats at large scale
• Experience with Apache Parquet, Apache ORC, Apache Arrow
• Outstanding technical skills in crafting and implementing high-quality distributed systems
• Excellent programming skills in C++, Java, and/or Scala
• Knowledge of distributed system schedulers: Kubernetes, Hadoop YARN, Spark standalone
• Able to work successfully with multi-functional teams across organizational boundaries and geographies
• Highly motivated with strong communication skills

Ways to stand out from the crowd:

• Development work on one or more of Delta Lake, Apache Iceberg, Apache Hudi, preferably with contributions back to the open source community
• Track record of contributions to major open source projects (such as Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka, Apache Arrow)
• Working experience with acceleration libraries (CUDA, RAPIDS, UCX)

​We are widely considered to be one of the technology world's most desirable employers, and as a result have some of the most forward-thinking and hardworking people in the world working for us. If you're passionate, creative, and driven, we'd love to have you join the team. With competitive salaries and a generous benefits package, we are widely considered to be one of the technology world's most desirable employers. We have some of the most forward-thinking and hardworking people in the world working for us and, due to unprecedented growth, our exclusive engineering teams are rapidly growing. If you're a creative and autonomous engineer with a real passion for technology, we want to hear from you.

The base salary range is 268,000 USD - 414,000 USD. Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

You will also be eligible for equity and benefits. NVIDIA accepts applications on an ongoing basis.

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.