Senior HPC Engineer - Mountain View, CA, United States - ASRC Federal Holding Company
Description
Job Description ASRC Federal InuTeq provides High Performance Computing services throughout the HPC lifecycle for computational requirements, architecture, acquisition, and operations to federal government customers.
Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement, and assimilation of industry best practices.
We are seeking to fill a role that primarily provides development for Supercomputing Batch Scheduling with Supercomputing Systems Administration secondary support for our NASA NACS High Performance Computing (HPC) contract.
Summary :
The successful candidate will be an active supporting member of the ASRC Federal team reporting directly to the Manager of the Application Performance and Productivity (APP) group and matrixed directly to the Supercomputing Systems Team Manager.
An individual at this skill level should have demonstrated extensive experience working with common HPC batch schedulers e.g.(PBS, Slurm, or Moab/Torque) while contributing to the support of users of HPC resources on the various issues they might have getting applications to run efficiently.
This individual should demonstrate experience installing, maintaining, and upgrading HPC systems.The individual, along with the entire HPC team, will be engaged in the day-to-day operations and support of the HPC resources.
Activities may include system patching, OS upgrades, deploying new systems, writing scripts, and troubleshooting system issues on the HPC system.
The ability to interact with users to determine symptoms, and then reproduce their issues to isolate the causes is critical skills for this work.
There will also be activities in testing, benchmarking, user tool scripting, and analyzing trouble tickets to find patterns indicating system or user education issues.
Duties and Responsibilities:
Designs, deploys and maintains HPC clusters with over 2000+ nodes with InfiniBand, 100+ petabytes of data storage in production.
Write and shepherd scalable feature designs through the entire software development process, from requirements and use cases to releaseDesigns and develops scripts for system administration, monitoring and usage reporting.
Modify existing software to correct errors and/or improve performanceDesigns and develops scripts for system regression test and performance (file systems (Luster), scheduler (PBS), interconnect (HDR/NDR, Slingshot, ), high availability, etc.).Troubleshoots, isolates and resolves application, system and other technical problems (hardware, software, and network).Understands research use cases, researches and deploys new technologies, defining cost, performance and other trade-offs.
Manages and maintains tools for configuration management (HPCM, Ansible GIT), resource management, scheduling and all necessary aspects of HPC in accordance with best practices.
Researches, deploys and manages networking and security infrastructure, including development of policies and procedures.Assists in developing and writing proposals and publications.
Creates and provides clear documentation.
Mentoring junior staff and cross training peersAfter hours/weekend support as requiredModerate Supercomputing System Administration that contributes to:
Day-to-day operations of the Linux HPC clusters and storage systemsProactive monitoring, analyze, and correct system issuesDevelopment of scripts to automate repetitive tasks or tools to enhance support of the HPC systemsSystem performance analysis and tuningBuilding, installing, and supporting user-requested softwareSupporting evaluation and assessment of new HPC technologyResolving user report issues and manage support tickets requests in
Remedy Requirements Requirements:
Bachelor's degree in computer science or related fieldStrong computer science background with in-depth systems-level knowledge in operating systems and networkingSolid understanding of the software development process, including requirements, use cases, design, coding, documentation and testing of scalable, distributed applications in a Linux environmentA minimum of 5 years experience working with HPC systems and scheduling software (PBS, Slurm, or Moab/Torque)
A minimum of 10 years of experience developing system software in heterogeneous, multi-platform HPC environmentsStrong ability to analyze, debug and maintain the integrity of an existing code baseDemonstrated equivalence of 5 years of Linux/UNIX user support experience and hands-on experience with administration of Linux systemsExperience working with HPC applications and proficiency in at least C, C++, or FortranSuperior scripting skills and excellent attention to detail; proficiency in at least Python, Perl, or BashStrong ability to interact with customers to understand needs, elicit requirements, and get feedback on prototype solutionsExcellent communication and people skills; excellent time management and organizational skillsExperience with system configuration management tools e.g.
CVS, SVN, GitTrack record of delivering commercial quality software on schedule with excellent quality through multiple release cyclesProficiency at technical writingPreferred Skills (Requesting Manager Defines):
Proficiency with analysis and problem-solving skills for debugging and optimization of applicationsFamiliarity/proficiency with OpenMP and Message Passing Interface (MPI) programmingExperience with Lustre, and InfiniBandExperience with cloud technologies (AWS, Azure, GCP), OpenStack or Kubernetes is a plus EEO Statement ASRC Federal and its Subsidiaries are Equal Opportunity / Affirmative Action employers.
All qualified applicants will receive consideration for employment without regard to race, gender, color, age, sexual orientation, gender identification, national origin, religion, marital status, ancestry, citizenship, disability, protected veteran status, or any other factor prohibited by applicable law.
#J-18808-Ljbffr