LLMOps for AI Agents: How to Run Agents Reliably in Production

57% of teams surveyed by LangChain now have AI agents in production, and quality is the biggest thing holding them back: 32% cite it as a top barrier, according to the State of Agent Engineering report, based on more than 1,300 practitioners. The research backs that worry up. On the τ-bench benchmark, even a strong function-calling model like gpt-4o completed fewer than half of realistic customer-service tasks, and when the same task was run eight times, it succeeded on all eight attempts less than 25% of the time.

LLMOps is the set of practices that keeps applications built on large language models reliable after launch: versioning what changes their behaviour, releasing changes safely, supervising what they do, controlling cost, and responding when something goes wrong.

Most LLMOps guides were written for chatbots and RAG apps, where the job is managing prompts and models. Agents break that frame. They call tools, update records and spend money on their own, many steps in a row. This guide gives you a five-stage operating loop built for that reality, with a risk tiering table, an incident runbook and a checklist you can use this week.

In a hurry? The short version: reliability lives in the operations layer, not in the framework. Run your first agent (we call them coworkers) with live logs and approvals, free for 5 days.

Key takeaways

• Agents fail quietly. A broken script throws an error. A broken agent keeps going, confidently.
• Reliability is measured across repeated runs, not one lucky demo. Track pass^k, not pass@1.
• LLMOps for agents is a loop: Version, Release, Supervise, Budget, Respond. Every incident feeds the next version.
• Approvals are a policy, not a feeling. Tier actions by risk and put a human in front of the irreversible ones.
• LLMOps for agents is AgentOps. The unit you operate is a worker that acts, not a model that answers.
• Your framework does not do this for you. LangGraph, CrewAI or raw API calls all need the same operations layer around them.

What is LLMOps?

Stanford HAI describes LLMOps as the practices and tooling used to manage the lifecycle of large language model applications, from development to deployment and monitoring. It grew out of MLOps, the discipline that turned machine learning models from notebooks into production systems.

The difference is in what you manage. In MLOps you train a model, measure its accuracy on a labelled test set and redeploy when the data drifts. In LLMOps you rarely train anything. Most teams use a hosted model and shape its behaviour with prompts, retrieved context and tools. In a separate finding, LangChain's survey shows that 57% of organisations are not fine-tuning at all, so "LLMOps is only for teams that train models" is a myth worth dropping early.

LLMOps vs MLOps

From LLMOps to AgentOps

Classic LLMOps assumes one input and one output. An agent takes a goal and decides its own path: it plans, calls a tool, reads the result, calls another tool, and keeps going. That changes four things at once:

1. Paths are not known in advance. You cannot review a flow you have never seen.
2. Actions have side effects. A wrong answer is embarrassing. A wrong refund is expensive.
3. Errors compound. One bad step early in a long task poisons everything after it.
4. Cost keeps running. A looping agent spends tokens until something stops it.

That is why the industry is starting to say AgentOps or agent ops. MLflow frames its LLMOps material the same way, with agents as the hard case. In this article, LLMOps means LLMOps for agents.

A broken Zap stops. A broken agent continues, confidently.

Why AI agents fail in production

Agents rarely crash. They drift, loop, or do slightly the wrong thing with total conviction. The failure modes worth planning for:

• Silent failure. The run finishes, the status is green, the output is wrong. No exception, no alert.
• Non-determinism. The same input produces different paths. τ-bench introduced pass^k, the probability an agent succeeds on all k attempts, precisely because single-run success hides this.
• Long-task decay. METR's research shows the length of tasks models can complete is doubling roughly every seven months, but reliability still drops sharply as tasks get longer.
• Excessive agency. OWASP lists Excessive Agency (LLM06) in its Top 10 for LLM applications: agents holding more permissions, functions or autonomy than the job needs.
• Runaway loops and spend. An agent retrying a failing tool call can burn a day's budget in minutes.

The textbook case happened in July 2025, when a coding agent on Replit deleted a company's production database during an explicit code freeze, then misreported what it had done. The model was not the root cause. The missing operations layer was: no separation between environments, write access to production, and no human gate in front of an irreversible action.

Vibe coding service Replit deleted production database: AI ignored instruction to freeze code, forgot it could roll back errors, and generally made a terrible hash of thingstheregister

The LLMOps lifecycle for agents: a 5-stage loop

Here is the loop. It wraps any framework and any model, and it never really ends: stage five feeds stage one.

1. Version everything that changes behaviour

An agent's behaviour comes from more than its prompt. If any of these change, you have a new agent, whether you meant to ship one or not:

• the system prompt and instructions
• the tools it can call, and the permissions on each one
• the model and its exact version
• memory, knowledge sources and retrieval settings
• config such as temperature, step limits and timeouts

Treat the whole bundle as one versioned artifact and tag every run with that version ID. When a customer complains on Tuesday, you need to know which agent they talked to, not which prompt file was in the repo that morning.

agent: invoice-chaser
version: 2026-10-17.3
model: claude-sonnet-4-5-20250929
prompt: prompts/invoice-chaser@a41f9c
tools:
  stripe.read_invoices: allow
  gmail.send_email: require_approval
  stripe.refund: deny
limits:
  max_steps: 25
  max_cost_per_run_usd: 0.80
owner: finance-ops

2. Release in stages: shadow, canary, full rollout

Never swap an agent version for everyone at once. Borrow the playbook software teams have used for years with feature toggles:

1. Shadow. The new version runs on real inputs but its actions are not executed. Compare its decisions with the current version.
2. Canary. Route 5 to 10% of real work to the new version, with tighter limits and more approvals.
3. Full rollout. Promote only when task success, cost per task and intervention rate hold steady.
4. Rollback. Write the rollback criteria before you release, not during the incident.

Pre-release checks belong here too, but building the test suite itself is a separate job, covered in our guide to AI agent testing. If you are shipping an agent for the first time rather than updating one, start with how to deploy AI agents.

3. Supervise: decide what needs a human

Supervision is where agents and chatbots part ways. A chatbot's worst case is a bad sentence. An agent's worst case is a bad action. So the core LLMOps decision is a policy: which actions run alone, which get logged, and which wait for a person.

This is not only good practice. For high-risk systems in Europe, Article 14 of the EU AI Act makes human oversight a legal requirement, including the ability to interrupt the system. LangChain's survey shows teams already lean this way: 59.8% rely on human review for nuanced or high-stakes cases. We go deeper on approval design in our guide to human-in-the-loop AI agents.

The minimum bar for supervision is seeing every action as it happens. How you collect traces, metrics and dashboards is its own topic, covered in AI agent observability. Here, the point is the policy those traces serve.

A human approval is a blocking interrupt, not a notification. If the agent keeps going while the message sits unread, you do not have oversight, you have a log.

4. Budget: put guardrails on tokens, cost and loops

Google's SRE teams manage reliability with error budgets: an agreed amount of failure a service may spend before releases slow down. Agents need the same idea for three resources:

• Steps. A hard cap per run, so a confused agent stops instead of looping.
• Cost. Per-run and per-day ceilings, with an alert well before the ceiling.
• Failure. An agreed error budget per agent. Burn through it and the agent moves back to canary with more approvals.

The point of a budget is not saving money. It is turning "the agent went wild overnight" into "the agent hit its cap at 2:14 and paused". The pricing maths and ROI side live in our breakdown of AI agent cost, and model routing, failover and provider fallback are covered in LLM gateway.

5. Respond: incidents, rollback and the feedback loop

Something will go wrong. The teams that recover fast decided what to do before it happened. A minimal agent incident runbook:

• Detect. An alert fires on cost, error rate, an unusual tool call or a user report.
• Pause. One click stops the agent. No new runs, pending actions held.
• Scope. Pull the runs of the affected version and list every external action they took.
• Roll back. Restore the last good version from stage 1.
• Repair. Undo or compensate the side effects, with a human signing off.
• Learn. Add the failing case to your regression set and write a blameless review.

That last step closes the loop. The incident becomes a test, the fix becomes a new version, and the new version goes back through staged release. That is what makes LLMOps a lifecycle rather than a dashboard.

We learned the silent-failure lesson on our own blog. The coworkers that research and score these articles depend on an SEO data provider. When its credentials expired, nothing crashed: the research coworker quietly fell back to older cached figures and kept producing topic lists. What told us the inputs had changed was its run log, and the fact that its report named its data source. The lesson was not about prompts. A credential check belongs in the runbook, and every agent report should say where its numbers came from.

LLMOps tools and stack for agents

LLMOps tooling sorts into four layers. They are complementary, and most production teams end up with something in each:

The operate layer is the one most stacks skip, because frameworks stop at "the agent works on my laptop". Anthropic's guide to building effective agents makes a related point: start with the simplest system that works, and add autonomy only when it pays for itself. Operations is how you pay for it safely.

Where Rerun fits, and where it does not

Rerun is one example of that operations layer. On Rerun, agents are called coworkers. Each workspace gets its own private, always-on machine, every action a coworker takes shows up live with its tokens and cost, and sensitive steps wait for a human approval from the app or Slack before they run. It runs on Claude, ChatGPT, Gemini and more, either hosted by Rerun or on your own subscription or API key.

To be clear about the limits: Rerun is not an agent framework, and it is not a dedicated eval or tracing platform. If you need offline eval suites with LLM-as-judge scoring, you will still want a tool built for that.

It is also not a flowchart tool. Zapier, Make and n8n run paths you drew in advance, so most of LLMOps barely applies to them. And a chatbot answers but does not act. LLMOps for agents matters precisely because the path is not known in advance.

LLMOps checklist for production agents

Copy this list into your next release review. If you cannot tick an item, that is your next piece of work.

• Every run is tagged with an agent version ID
• Prompt, tools, permissions, model and config are versioned as one bundle
• New versions go through shadow or canary before full rollout
• Rollback criteria are written down before release
• Every tool call is visible, with inputs and outputs
• Actions are tiered by risk, and irreversible ones need approval
• Each agent has a step cap and a cost ceiling per run and per day
• Alerts fire before a budget is exhausted, not after
• There is a one-click way to pause an agent
• A written runbook exists and someone owns it
• Every incident adds a case to the regression set
• Reliability is tracked across repeated runs (pass^k), not single successes

For the wider governance picture, from ownership to audit trails, see our guide to AI agent governance, and frame risk with the NIST AI Risk Management Framework.

Run agents like workers, not like demos

The model will keep getting better. The METR curve says so. But better models do not remove the need to know which version ran, what it did, who approved it and how to stop it. That is LLMOps for agents: the boring, durable layer that lets you hand real work to software that thinks for itself.

Written by

Clément Janssens