Lead Data Infrastructure Engineer - Cupertino
1 month ago

Job summary
We're looking for an engineer to help lead the scaling and reliability of our data infrastructure, which is core to the ML work we do at Gridmatic.Forecasting energy prices is challenging. We have very effective price forecasting models, but we'd like to go much further — scaling the amount of data we can use in our ML models by a factor of 10-100x by incorporating petabyte-scale weather data, increasing spatial granularity of our price forecasting, and more.
We'd also like someone who can tackle the challenge of scaling and improving reliability of our data platform. We deal with a lot of real-world problems when ingesting data from external sources — downtime, late-arriving data, changing schemas.
Responsibilities
- Experience building the infrastructure for large-scale data processing pipelines (both batch and streaming) using tools like Spark,Kafka,Apollo Flink,and Apache Beam.
- Experience designing and implementing large-scale data storage systems (feature store,timeseries DBs) for ML use cases.Strong familiarity with relational databases,data warehouses(object storage,timeseries
- Experience building,data pipelines for external ,sources that are observable,,debuggable,and verifiably correct.Have dealt with challenges
like ,data versioning point-in-time correctness,evolving schemas.- Owning ,and scalingour dataninfrastructureby several ordersof magnitude.Thisincludesour datapipelines,distributed
data processing,and datastorage.Builidinga unified featurestorefor allourMLmodels.Efficiently storingalandloading hundredsOf terabytes Ofweatherdatavforusein AI-basedweathermodels.Processingandstoringpredictionsandevaluationmetricsforlarge-scalegforecastingmodels.Interviewprocessusuallyhaschatwiththehiringsmanagerorsomeonetontheteamaboutyourbackgroundandexperiencenextdependingontherole,you'llhaveatechnicalphone screenwithanengineer orworkona take-homeproject
Job description
Lorem ipsum dolor sit amet
, consectetur adipiscing elit. Nullam tempor vestibulum ex, eget consequat quam pellentesque vel. Etiam congue sed elit nec elementum. Morbi diam metus, rutrum id eleifend ac, porta in lectus. Sed scelerisque a augue et ornare.
Donec lacinia nisi nec odio ultricies imperdiet.
Morbi a dolor dignissim, tristique enim et, semper lacus. Morbi laoreet sollicitudin justo eget eleifend. Donec felis augue, accumsan in dapibus a, mattis sed ligula.
Vestibulum at aliquet erat. Curabitur rhoncus urna vitae quam suscipit
, at pulvinar turpis lacinia. Mauris magna sem, dignissim finibus fermentum ac, placerat at ex. Pellentesque aliquet, lorem pulvinar mollis ornare, orci turpis fermentum urna, non ullamcorper ligula enim a ante. Duis dolor est, consectetur ut sapien lacinia, tempor condimentum purus.Get full accessAccess all high-level positions and get the job of your dreams.
- Experience building,data pipelines for external ,sources that are observable,,debuggable,and verifiably correct.Have dealt with challenges
Similar jobs
Infrastructure Engineer
1 day ago
We are looking for a hands-on Infrastructure Engineer to join our team and support our high-performance, on-premise server and networking infrastructure. · Maintain, provision, and troubleshoot hardware and Linux systems. · Physically install rack cable maintain blade servers and ...
Infrastructure Engineer
1 week ago
A summary of the job is not available in this data · ...
ML Infrastructure Engineer
3 weeks ago
Are you passionate about Generative AI? · We are the Intelligence System Experience (ISE) team within Apple's software organization. · ...
Design Infrastructure Engineer
1 day ago
Etched is seeking an Design Infrastructure Engineer to join our rapidly growing team. · ...
Software Engineer, Infrastructure
3 weeks ago
Embedded within Apple's Security Engineering u0026amp; Architecture organization, this role is on an engineering team that builds security infrastructure into Apple's production build system and provides security engineering expertise and leadership across Apple to create securit ...
Infrastructure Engineer
1 month ago
We're looking for an Infrastructure Engineer to help us design, build, and scale the foundational architecture that powers our next-generation AI systems. · ...
ML Infrastructure Engineer
2 weeks ago
We are seeking engineers experienced in building tools for training, adapting. · ...
Software Engineer, Infrastructure
3 weeks ago
We design and build software that implements security features into Apple's build systems that affect the software that is shipped to Apple's customers. · In addition to writing software, we perform security analysis on these internal systems to assess threats and security risks. ...
DevOps Infrastructure Engineer
19 hours ago
Designing and writing software for new ASICs is hard. · ...
ML Infrastructure Engineer
3 days ago
+We are seeking engineers experienced in building tools for training, adapting and deploying large-scale generative models. · +Contributing towards tools for large generative model training including diffusion & autoregressive workflowsBenchmarking, Analysing and Improving traini ...
Software Engineer, Infrastructure
3 weeks ago
This is a software engineer role within Apple's Security Engineering & Architecture organization. · We design and build software that implements security features into Apple's build systems that affect the software that is shipped to our customers. · We perform security analysis ...
Infrastructure Lead Engineer
16 hours ago
This is a full-time opportunity for an experienced Infrastructure Lead Engineer to join our team in Cupertino, CA. The ideal candidate will have at least 7 years of experience in Business Process Consulting and Architecture/Design/Detailing of Processes. · • At least 7 years of e ...
Infrastructure Engineer
3 weeks ago
We're hiring an Infrastructure & Embedded Software Engineer to help own and scale the embedded infrastructure stack behind next-generation radio and networking systems. · Develop and maintain embedded Linux and bare-metal firmware for radio and networking hardware. · Ownand impro ...
Infrastructure Engineer
3 weeks ago
+We're hiring an Infrastructure & Embedded Software Engineer to help own and scale the embedded infrastructure stack behind next-generation radio and networking systems. · +Develop and maintain embedded Linux and bare-metal firmware for radio and networking hardware. · Own and im ...
WebKit Performance Infrastructure Engineer
3 weeks ago
We're looking for an engineer to evolve our performance monitoring and testing infrastructure.You'll own and improve the systems that power our performance testing and analysis workflows, · and your work will directly impact how WebKit engineers detect and diagnose regressions. · ...
On-device ML Infrastructure Engineer
1 month ago
Imagine being at the forefront of an evolution where modern AI meets the elegance of Apple silicon. · ...
On-Device ML Infrastructure Engineer
3 weeks ago
The On-Device Machine Learning team transforms groundbreaking research into practical applications. · ...
Senior ML Infrastructure Engineer
3 weeks ago
We are seeking engineers experienced in building infrastructure for training, adapting and deploying large-scale generative models.We are the Intelligence System Experience (ISE) team within Apple's software organization. · ...
On-device ML Infrastructure Engineer
3 weeks ago
The On-Device Machine Learning team transforms groundbreaking research into practical applications, enabling billions of Apple devices to run powerful AI models locally, privately, and efficiently. · ...
Software Engineer, Test Infrastructure
3 weeks ago
This is your opportunity to join a hands-on team that cultivates innovation and helps define the future of quality for Apple's entire enterprise ecosystem. · Design, build, and operate internal infrastructure that simulates core enterprise device management and identity protocols ...
Cloud Infrastructure Software Engineer
3 weeks ago
+Job summary · The Apple Services Engineering(ASE) team builds and provides systems and infrastructure that power Apple's services (such as iCloud, iTunes, Siri, and Maps). We are the foundation on which Apple's software developers build the products that our customers love. Our ...