Lead DevOps Engineer - Phoenix, AZ

Full-time · Description · Design, scale, and secure mission-critical AI/ML systems · Prime Solutions Group (PSG), Inc. is seeking a Lead DevOps Engineer (AI/ML Ops) to serve as a hybrid senior technical contributor and team leader, responsible for designing, implementing, and ope ...

Job description

Full-time

Description

Design, scale, and secure mission-critical AI/ML systems

Prime Solutions Group (PSG), Inc. is seeking a Lead DevOps Engineer (AI/ML Ops) to serve as a hybrid senior technical contributor and team leader, responsible for designing, implementing, and operating secure, automated machine learning and data pipelines across cloud and on-premise environments.

In this role, you will sit at the intersection of machine learning, data engineering, and DevSecOps, ensuring ML models and data-driven services are scalable, secure, observable, and compliant across their full lifecycle—from data ingestion and feature engineering through training, deployment, monitoring, and retraining.

You will guide technical execution, mentor engineers, and make key architectural and tooling decisions for PSG's MLOps platforms. Building on PSG's established DevSecOps foundation (CI/CD, Infrastructure-as-Code, security baselines), you will extend capabilities to include experiment tracking, model registries, drift detection, and model performance monitoring.

This is a fast-paced, high-impact opportunity to deliver enterprise-scale AI/ML solutions while directly supporting U.S. national security missions.

Key Responsibilities

• Lead the design, implementation, and operation of ML-focused CI/CD pipelines supporting data ingestion, feature engineering, model training, evaluation, and deployment across dev, test, staging, and production environments.
• Apply and adapt MLOps best practices within existing DevSecOps workflows, including:
  • Data quality checks and schema validation
  • Model validation and promotion gates
  • Model performance and drift monitoring
• Architect and oversee training and inference platforms, including experiment tracking, model registries, and automated retraining pipelines.
• Oversee secure integration of Infrastructure-as-Code, containerization, and orchestration (Docker, Kubernetes) for ML and data workloads, including GPU and high-performance compute resources.
• Mentor and guide engineers in MLOps and DevSecOps practices, promoting automation, observability, and security-first design.
• Collaborate with cross-functional teams (data science, software engineering, research, IT, cybersecurity, systems engineering) to ensure ML system reliability, performance, and compliance.
• Lead technical risk assessments and contribute to incident response for ML and data systems (e.g., model degradation, data quality issues, pipeline failures).
• Serve in a hybrid role as both:
  • A senior hands-on engineer contributing to pipelines, infrastructure, and monitoring
  • A technical leader guiding small to mid-sized MLOps initiatives
• Make informed technical decisions across ML, data, security, and operations domains, resolving complex multi-disciplinary challenges.
• Evaluate ethical and operational considerations in AI/ML deployment (e.g., bias, data constraints, mission risk) and recommend appropriate mitigations.
• Stay current on emerging MLOps, AI platform, and data engineering technologies, recommending adoption where beneficial.

Requirements

• U.S. Citizenship
• Active Top Secret clearance or higher
• Bachelor's degree in Computer Science, Engineering, Data Science, Applied Mathematics, or related field
• 5–9+ years of experience in one or more of the following:
  • MLOps or ML platform engineering
  • DevOps / DevSecOps / SRE supporting data or ML workloads
  • Data engineering with production ML integration
  • Applied machine learning in production environments
• Strong experience with CI/CD tools (Jenkins, GitLab CI, GitHub Actions, CircleCI) and modern Git workflows
• Hands-on experience with Infrastructure-as-Code (Terraform, Ansible, CloudFormation) and Kubernetes
• Proficiency with ML and data technologies, including:
  • Python and ML/data libraries (NumPy, pandas, scikit-learn, PyTorch, TensorFlow)
  • Workflow/orchestration tools (Airflow, Kubeflow, Prefect, Dagster)
  • Experiment tracking and model registries (MLflow, Weights & Biases, SageMaker)
• Experience integrating security and governance into ML environments (image/dependency scanning, SBOMs, secrets management, IAM)
• Familiarity with NIST, FedRAMP, and DoD RMF compliance frameworks as applied to ML and data systems
• Strong scripting or programming skills (Python, Bash, Go, or similar)
• Demonstrated experience leading technical efforts and mentoring engineers
• Ability to communicate clearly with both technical and non-technical stakeholders
  

Preferred Qualifications

• Security, cloud, or ML certifications (e.g., CISSP, AWS Security Specialty, AWS ML Specialty, CKS, GIAC)
• Experience implementing Zero Trust architectures
• Experience with observability and monitoring tools (Prometheus, Grafana, ELK/EFK, OpenTelemetry) for ML services
• Hands-on experience with:
  • Feature stores and data validation frameworks (e.g., Great Expectations)
  • Data governance and lineage tooling
  • Policy-as-code for ML environments (OPA, Kyverno, admission controllers)
• Prior experience supporting defense, aerospace, or government-secured AI/ML programs
• Experience operating enterprise-scale or mission-critical ML systems, including high-availability inference and rigorous performance monitoring
  

Why Join PSG?

At PSG, you're not just filling a role—you're shaping the future of AI/ML-enabled digital engineering. We combine the agility of a small business with the opportunity to support some of the government's most advanced and impactful technology programs. We offer:

• Competitive compensation and benefits
• Professional development and tuition assistance
• A collaborative, mission-driven culture
• Direct impact on national security through secure AI/ML solutions

Bring your AI/MLOps expertise to PSG and help build the next generation of secure, intelligent, data- and model-driven platforms.

Salary Description

Salary range starts at $116,806 with the potential for higher compensation based on experience, skills, and mission needs.