Practice: LLMOps: evals, tracing, guardrails and AI-assisted development with Claude Code and Cursor

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Main tool: Cursor · Cursor puts the things a platform team governs for an AI coding assistant (rules, subagents, MCP servers, tool approval, model choice and agent environments) into one editor, so the whole evaluate, trace and guard loop can be practised on a laptop. · Alternatives: Claude Code is a terminal-first agent configured through its own settings and memory files rather than an editor, Copilot is an assistant plugin inside existing editors with lighter agent controls, Aider is an open source terminal pair programmer driven by git commits, and Continue is an open source editor extension where the team wires up its own models.

Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1

Setup (one time)

1. Install Cursor, sign in and open an empty practice repository that will hold a small LLM app with a stubbed model, so no paid API or GPU is needed.
Every later step runs inside this one workspace. A stubbed model keeps the evals deterministic and free.
Explained in: Cursor Docs — Agent, Rules, MCP, Skills & CLI (cursor.com), checked 2026-10-12
Done when: Cursor opens the practice folder and the agent panel accepts a prompt.
2. Pick the model the agent will use for this path and learn where to switch it.
Model choice changes cost, speed and behavior, and a platform team pins it deliberately rather than leaving the default.
Explained in: Prompting agents | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The model selector shows the model you chose and you can switch it and switch back.
3. Read how MCP servers are installed in Cursor and make sure a local runtime for a stdio server and a container engine are available on the laptop.
The tracing and guardrail steps connect the agent to local tools through MCP. Having the prerequisites ready avoids stalling in the middle of the path.
Explained in: Model Context Protocol (MCP) | Cursor Docs (cursor.com), checked 2026-10-12
Done when: You can name the two ways to add a server and the MCP settings screen opens with no servers failing.

Steps

1. Run a first Agent task that scaffolds a tiny LLM app with a stubbed model and a pytest eval suite of about ten cases.
This is the first run of the tool and it produces the app and eval set that every later step gates, traces and guards.
Explained in: Overview | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The agent has created the files and the eval suite runs in the terminal with a visible pass count.
2. Open Design Mode on a small local page that displays the eval results, select one element and ask the agent to change it, then inspect what context the agent received.
Seeing exactly what the agent is given teaches the core model of the tool: the agent acts only on the context you hand it.
Explained in: Design Mode | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The selected element changes on the page and you can list what the agent saw for that request.
3. Restore a checkpoint after asking the agent for a prompt change that lowers the eval score, and compare the eval results before and after the restore.
Checkpoints are how agent work is persisted and rolled back locally. Comparing scores across them is the smallest form of an eval regression check.
Explained in: Overview | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The files return to the earlier state and the eval pass count matches the pre-change run.
4. Compare your installed Cursor version with the latest agent and chat release notes and record the version in the repository readme.
Agent behavior shifts between releases, so a team needs a recorded version to explain why the same prompt behaves differently on two laptops.
Explained in: Release notes | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The readme states the version in use and one release note change that affects agent behavior.
5. Create a project rule stating that any change to prompts, retrieval or tool definitions must be followed by a full eval run that passes before the work is called done.
A rule packages the release gate as versioned configuration in the repository, so every teammate's agent follows the same eval policy.
Explained in: Rules | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The rule file is committed and a new agent chat that edits a prompt runs the evals without being asked.
6. Create a custom verification subagent whose only job is to run the eval suite and report pass or fail against a threshold you set.
A separate verifier keeps the release gate independent of the agent that wrote the change, which is the same separation an eval gate has in a pipeline.
Explained in: Subagents | Cursor Docs (cursor.com), checked 2026-10-12
Done when: Invoking the subagent returns a pass or fail verdict with the score, and a deliberately broken prompt gets a fail.
7. Run Agent Review on the uncommitted diff as a second check after the eval gate passes.
Evals catch behavior regressions while review catches issues that the eval set does not cover, so the two together act as checks before release.
Explained in: Agent Review | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The review lists findings for the diff and you have fixed or dismissed each one.
8. Configure a local stdio MCP server that exposes traces from a tracing backend running in a container, then query the trace of one app run from the agent chat.
Tracing LLM and agent runs is only useful when engineers can reach the traces while debugging. MCP brings them into the editor.
Explained in: Model Context Protocol (MCP) | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The server shows as connected and the agent returns the spans, inputs and outputs of a run you just made.
9. Enable tool approval for the MCP tools, plant an instruction inside a file that tells the agent to call a tool, and confirm the call is held for approval.
Prompt injection and excessive agency lead the OWASP LLM risk list. Approval on tool calls is the guardrail that stops injected text from acting on its own.
Explained in: Model Context Protocol (MCP) | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The injected tool call appears as a pending approval and does not run until you decide.
10. Write the data retention terms of the model you pinned into the team rule, together with the kinds of data that must never be pasted into a prompt.
Sensitive information disclosure is another OWASP LLM risk. The retention terms of the chosen model decide what a team may send to it.
Explained in: Claude Fable 5.1 | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The rule names the pinned model, its retention behavior and a short list of forbidden data types.
11. Write an AGENTS.md file that holds the eval gate, tracing and guardrail conventions, so Cursor and Claude Code both read the same instructions.
A platform team governs several coding assistants at once. One shared instruction file keeps their behavior reproducible across tools and laptops.
Explained in: Rules | Cursor Docs (cursor.com), checked 2026-10-12
Done when: A fresh Cursor agent chat follows the conventions from the file without the project rule being mentioned.
12. Write an environment.json and a Dockerfile that define the agent environment in code. Cloud agents may need a paid plan, so as the local substitute build and run that Dockerfile with a container engine and run the evals inside it.
Defining the environment in code is the production-like setup: every agent run gets the same pinned dependencies and the eval gate behaves the same everywhere.
Explained in: Cloud Environment Setup | Cursor Docs (cursor.com), checked 2026-10-12
Done when: The container builds from the committed files and the eval suite passes inside it with the same score as on the laptop.

Back to LLMOps: evals, tracing, guardrails and AI-assisted development with Claude Code and Cursor