AI Platform Operations Analyst 207604 - Orlando - Aquent Talent

Description

Here's a dynamic and engaging job posting for the AI/ML Ops Analyst role:

**Ignite Innovation: Drive the Future of AI at a Global Leader**

Imagine being at the forefront of innovation for a global leader, shaping the future of entertainment and technology. This organization is building a cutting-edge Generative AI platform and Center of Excellence, a critical hub powering all AI capabilities across its vast operations, from marketing content generation to core business functions. This is your chance to make a profound impact, ensuring the reliability, efficiency, and cost-effectiveness of advanced AI systems that touch millions. As an integral partner with Aquent, you will play a pivotal role in this transformative journey.

**Your Impact:**

As an AI/ML Ops Analyst, you will be the operational nerve center of this groundbreaking AI platform. You'll directly influence the performance, cost, and stability of models, agents, and knowledge bases that drive critical business processes. Your expertise will ensure seamless operations, optimize cloud spend, and provide crucial insights that guide strategic decisions, making a tangible difference in how AI is leveraged across the entire organization. You will be instrumental in building a robust, scalable, and responsible AI ecosystem, contributing to projects that redefine how AI is integrated into daily operations and strategic initiatives.

**Key Responsibilities:**

* **AI/ML Operations:** Manage operational workflows for model deployments, updates, and versioning across multi-cloud environments. Monitor critical model performance metrics including latency, throughput, error rates, token usage, and inference quality. Proactively track model drift, accuracy degradation, and performance anomalies, escalating issues to engineering teams as needed. Support knowledge base operations, ensuring the health of vector embedding pipelines, chunk quality, and timely refresh cycles within cloud AI services. Maintain a comprehensive model inventory and documentation across diverse cloud environments. Coordinate model evaluation cycles with Responsible AI and Core Engineering teams.

* **Agent & Server Operations:** Monitor the health, performance, and reliability of AI agents. Track agent execution metrics such as task completion rates, tool call success/failure, latency, and error patterns. Support agent deployment and configuration management workflows. Document agent behaviors, known issues, and operational runbooks. Coordinate with Core Engineering on agent updates, testing, and rollouts. Monitor the availability, connection health, and integration status of context management servers.

* **FinOps & Cost Management:** Track and meticulously analyze AI/ML cloud spend across various cloud AI services. Develop insightful cost dashboards with breakdowns by model, application team, use case, and environment. Monitor token consumption, inference costs, and embedding/storage costs to identify trends. Identify and implement critical cost optimization opportunities through model selection, caching strategies, batching, and rightsizing. Provide accurate cost allocation reporting for chargeback/showback to consuming application teams. Forecast spend trends and proactively flag budget anomalies, partnering closely with Infrastructure and Finance teams on AI cost governance.

* **Monitoring, Dashboarding & Reporting:** Build and maintain comprehensive dashboards for platform performance, model health, agent metrics, and key operational KPIs. Create executive and stakeholder reports on platform adoption, usage trends, and detailed cost allocation. Develop Responsible AI dashboards tracking hallucination rates, accuracy metrics, guardrail triggers, and safety incidents. Monitor API gateway traffic patterns and API consumption trends. Provide regular reporting to product management on use case performance and impact.

* **Release Operations Support:** Support release management processes with rigorous pre- and post-deployment validation checks. Track release health metrics for models, agents, and platform components. Maintain up-to-date release documentation, runbooks, and operational playbooks. Coordinate seamlessly with QA, Performance Engineering, and Infrastructure teams during releases.

* **Responsible AI Operations:** Monitor guardrail effectiveness and flag anomalies to the Responsible AI team. Track and report on hallucination detection, content safety triggers, and accuracy trends. Support LLM Red Teaming efforts by collecting and organizing critical evaluation data. Maintain audit logs and compliance documentation for robust AI governance.

* **Cross-Functional Coordination:** Serve as the primary operational point of contact for application teams consuming AI APIs. Coordinate with Corporate Security on audit requests and compliance reporting. Partner with the Infrastructure team on capacity tracking and resource utilization planning. Support Performance Engineering with load test analysis and results documentation.

**Must-Have Qualifications:**

* 2-4 years of experience in an Ops, Analytics, or Technical Operations role (MLOps, AIOps, DataOps, Platform Ops, or similar).

* Solid understanding of AI/ML concepts, including models, inference, embeddings, vector databases, Large Language Models (LLMs), tokens, and prompts.

* Proven experience with cloud cost management and FinOps—tracking, analyzing, and optimizing cloud spend.

* Strong proficiency with dashboarding and visualization tools (e.g., Looker, Tableau, Grafana, or similar).

* Working knowledge of a major cloud platform (required); familiarity with other major cloud platforms is a significant plus.

* Comfortable with SQL and basic Python for data analysis and scripting.

* Experience with monitoring and observability platforms (e.g., Datadog, Prometheus/Grafana, Cloud Monitoring, or similar).

* Understanding of APIs and API gateways—ability to read logs, trace requests, and analyze traffic patterns.

* Strong analytical and problem-solving skills with meticulous attention to detail.

* Excellent communication skills, with the ability to translate complex technical metrics into actionable stakeholder insights.

* Bachelor's Degree in Computer Science, Business Information Systems, Management Information Systems, Electrical Engineering, Mechanical Engineering, or a similar field is required.

**Nice-to-Have Qualifications:**

* Hands-on experience with LLM platforms from major cloud providers.

* Familiarity with AI agents and agentic architectures (e.g., LangChain or similar).

* Exposure to agent-tool integration patterns.

* Experience with vector databases and Retrieval-Augmented Generation (RAG) operations.

* Understanding of the MLOps lifecycle: model registry, versioning, deployment patterns, and A/B testing.

* Experience with API management platforms.

* Familiarity with Responsible AI metrics—hallucination, bias, content safety, and guardrails.

* FinOps certification or formal cloud cost management experience.

* Experience supporting enterprise platform teams with multiple consuming applications.

* Familiarity with ML pipeline tools.

* Exposure to prompt management and evaluation frameworks.

* ITIL or other operational process framework experience.

* Experience creating comprehensive runbooks and operational documentation.

**About Aquent Talent:**

Aquent Talent connects the best talent in marketing, creative, and design with the world's biggest brands.

Our eligible talent get access to amazing benefits like subsidized health, vision, and dental plans, paid sick leave, and retirement plans with a match. More information on our awesome benefits

Aquent is an equal-opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics. We're about creating an inclusive environment—one where different backgrounds, experiences, and perspectives are valued, and everyone can contribute, grow their careers, and thrive.

#LI-Onsite