How to test code that calls an LLM (without a live model every run)

Testing an AI feature means running a loop that calls a model, maybe executes a tool, feeds the result back, and calls the model again. Every run of that loop is slow, costs money, and returns something slightly different. So teams end up in one of two bad places: nobody runs the tests, or everybody mocks the model so heavily the test no longer resembles what ships. Neither catches the bug that wakes you up.

The way out isn't "mock the model" or "don't mock the model" — it's choosing *which layer* to mock at, because that decision quietly determines what your test actually exercises. Below are the four approaches, the trade-off that matters, and a working record/replay setup using our open-source @duskel/cassette. The one-line version: record the model's answers once, replay them offline in milliseconds, and let your own code keep running for real.

Why LLM test suites rot

Three properties of a real model call make it hostile to a normal test suite, and they compound:

• Slow. A multi-step agent loop is several sequential round-trips to a model. A suite that takes 90 seconds instead of 90 milliseconds is a suite developers stop running before every commit.
• Expensive. Tests that hit the API on every CI run put your test suite on the same bill as production. Teams respond by running them rarely, which defeats the point.
• Non-deterministic. The same input returns different words each time. Assert on the exact text and the test flakes; assert on nothing and it passes while broken. Flaky tests get muted, and a muted test is a deleted test that still shows green.

Four ways to test code that calls an LLM

These are the options in practice, from naive to robust. Most teams cycle through the top three before landing on the fourth.

The layer you mock at decides what you actually test

This is the insight that most LLM testing advice skips. An agent's interesting behaviour is not in the network call — it's in the loop *around* it. The model returns a `tool_use` block; your code has to parse it, dispatch the right tool, coerce the arguments, run the tool, handle its errors, and format the result back into the next request. That parsing-and-dispatch path is where agents actually break.

Now look at what each mocking layer does to that path:

• Mock at the HTTP layer (replace `fetch`/the socket) and you replay a response body — but your tool never runs. The dispatch code, the argument coercion, the error handling inside the tool: all untested. If your `convertTemperature` has an off-by-one, an HTTP mock cannot catch it, because it never called the function.
• Record at the SDK boundary (patch `client.messages.create`, not `fetch`) and only the *model's answer* comes off the tape. Your agent loop, your tool dispatch, and your tools all still execute for real. The off-by-one fails the test, exactly as it would in production.

Record/replay in practice

Here's the whole pattern with cassette. You wrap the run once; on the first pass it records the model's responses to a file, and every run after that replays them — no network, no API key, deterministic.

import Anthropic from '@anthropic-ai/sdk';
import { withCassette, anthropicAdapter } from '@duskel/cassette';
import { expect, test } from 'vitest';

test('the agent converts the temperature it looked up', async () => {
  const client = new Anthropic();

  const run = await withCassette('weather-agent', async (tape) => {
    tape.use(anthropicAdapter, client);            // patch the SDK boundary
    return runAgent(client, 'Weather in Paris, in fahrenheit?');
  });

  // runAgent's tool dispatch + tools ran for real; only the model replied from tape
  expect(run.answer).toContain('69.8');
});

Record once with the mode set to `record` (and an API key present); commit the resulting cassette file; then CI runs in `replay` with no key and no network. Because the model's answers are frozen, the test is deterministic — and because your loop still runs, it's a real test, not a mock theatre. When you genuinely change the prompt or the expected behaviour, you re-record on purpose. Two practical notes: secrets in recorded requests should be redacted before the cassette is committed (cassette does this), and for tools that are themselves slow, paid or non-deterministic you can opt to record the tool boundary too.

When record/replay is the wrong tool

Being honest about the boundary matters, because record/replay solves one problem and not another. It makes your *integration* deterministic — does my code correctly handle the model's output? It does not tell you whether the model's output is any *good*.

• Integration correctness → record/replay. Does the loop parse the tool call, run the tool, and assemble the answer correctly? Freeze the model, run your code, assert. This is what belongs in your normal CI suite.
• Output quality → evals, not replay. Is the model choosing the right tool, staying on policy, refusing the bad request? That needs an eval set scored against live (or recent) model output, run on a different cadence — not a frozen tape. Replaying a good answer proves nothing about whether the model still produces good answers.

Doing this on real projects

We built cassette because we ship production AI systems and kept watching agent test suites die the same death — too slow and flaky to run, then quietly abandoned. Getting the testing layer right is part of what separates an agent that survives real traffic from a demo, which is the same reason we care about evals and guardrails; we wrote that up in how to choose an AI agent development company.

If you're building an AI feature and want it tested so the suite actually gets run — deterministic integration tests plus an eval set that measures quality — tell us what you're building and we'll scope it. Or just `npm i -D @duskel/cassette` and freeze your first flaky test.