Unit tests break for LLM applications. Learn the methods, metrics, and architecture that make llm testing work: datasets, LLM judges, traces, and runtime enforcement.
A unit test assumes one input produces one correct output. LLMs do not work that way. Due to their probabilistic nature, the same prompt can lead to slightly varied responses on different occasions. A single response might not be definitively wrong, but rather less desirable compared to other possible outputs. The entire pass/fail model breaks.
As of 2025, 67% of organizations worldwide are using LLMs to support their operations with generative AI. Most of them are testing these systems with tools designed for deterministic software. The result: outputs are open-ended, and no assertion library can distinguish between correct, creative, or confidently wrong answers.
This article covers what llm testing actually requires when unit tests stop being useful: the methods, the metrics, the traces that make evaluation possible, and where runtime enforcement fits in.
Before going further, a vocabulary distinction that saves hours of confusion.
LLM evaluation measures how well a model performs on standardized datasets or benchmarks. The result is a numerical score representing accuracy, reasoning capacity, or language understanding. LLM testing is more applied and scenario-related: it involves how the model will perform in specific scenarios related to your product. Evaluation picks the model. Testing verifies it is ready for production.
The gap between these two is where teams lose time. A model might score impressively on SQuAD 2.0, but if you feed it a finance-related prompt in a multi-step pipeline, it may struggle without domain-specific prompt engineering. Raw benchmark scores do not predict production behavior. Application-centric testing considers prompts, multi-step processing, domain-specific constraints, and memory usage as part of the evaluation scope.
LLM testing spans four distinct dimensions: functionality, system performance, security, and alignment. Each requires different methods, different data, and different expertise.
| Dimension | What it covers | Example checks |
|---|---|---|
| Functionality | Correct task completion, format adherence, tool usage | Does the agent call the right API? Does the summary include key facts? |
| Performance | Latency, throughput, cost per request | P95 response time under load, token cost per conversation |
| Security | Prompt injection resistance, data leakage, guardrail evasion | Adversarial prompts, PII extraction attempts |
| Alignment | Tone, policy adherence, harmful content avoidance | Brand voice consistency, refusal of out-of-scope requests |
No single testing technique covers all four. A regex check catches format violations but says nothing about alignment. An LLM judge scores helpfulness but cannot measure latency. A robust testing data suite should encompass real user queries, synthetic prompt generation, annotation, and adversarial prompts to stress all four dimensions.
Five evaluation method families exist, and most teams mix and match from them:
1. Code evaluators. The cheapest and most reliable method wherever a check can be expressed as logic: exact match, regex validation, JSON parseability, schema validation, keyword checks, tool-call assertions, and custom business rules. Start here. If you can write a deterministic rule for it, do not use a judge.
2. LLM-as-a-judge. For subjective criteria like helpfulness and faithfulness, the most reliable method is using an LLM to evaluate with natural language rubrics, though it requires techniques like G-Eval. Automated evaluators also need ongoing calibration to stay aligned with human judgment on a sample of annotated examples. Without that upkeep, judges drift.
3. Human annotation. Deep, trustworthy judgments from domain experts. Expensive and slow, but necessary for establishing ground truth that automated methods calibrate against.
4. User feedback. The clearest signal of whether output hit the mark. Thumbs up/down, ratings, corrections.
5. Implicit feedback. Behavioral signals: retries, link clicks, conversation abandonment. These reveal failure modes users never bother to report.
One common mistake: relying on traditional scorers like BLEU and ROUGE. These miss semantic nuance in LLM outputs. A paraphrase that perfectly answers the question scores poorly on BLEU because the words differ. A response that matches surface tokens but hallucinates a key detail scores well. For llm testing, metrics must be quantitative, reliable, and aligned with human judgment, not just string similarity.
None of these methods work at scale without structured traces. A trace is a detailed record of one interaction with your application, including the input, intermediate steps like retrievals and tool calls, and the final output. Traces let you pinpoint which step failed, compare outputs against expected results, and see which inputs cause the most problems.
This matters especially for multi-step pipelines. In a RAG system, a bad answer could come from the router, the retriever, or the generator. If you only score the final answer, you cannot tell whether a bad response came from routing, retrieval, or generation. Each stage needs its own evaluation criteria.
Traces are also what convert production traffic into test data. Every traced interaction is a potential test case. Every failure is a potential regression test. Without traces, you are guessing which scenarios to test. With them, you know.
For deeper coverage on production trace collection, see our guide to LLM observability.
LLM testing splits into two modes that serve different purposes.
Offline evaluation runs on curated datasets before you ship a change. It catches regressions against known test cases. Think of it as CI gating: before a prompt change or model swap goes live, run it against your dataset and compare scores.
Online evaluation runs continuously on live production traces. It catches model drift, unexpected user inputs, and new failure modes. Things your dataset never covered because you did not know to test for them.
Each catches errors the other misses. And they form a loop: offline experiments validate a change before deployment, online evaluation catches the edge cases your dataset did not cover, and those edge cases go back into the dataset so the next experiment catches them.
This loop is what separates teams that improve their LLM applications over time from teams that ship and hope. The dataset grows from production reality, not from what engineers imagined users might ask.
Metric selection should cover both the LLM use case and the LLM system architecture:
The recurring challenges are real: defining clear evaluation goals, subjective interpretation, cost and latency, calibration of automated evaluators, and cross-team coordination. Traditional metrics like error rates do not translate well when your application might encounter any question under the sun. You need to pin down what "good" looks like before you measure anything.
The testing evolution, from basic manual inspection to automated large-scale evaluation using LLM-as-a-judge, synthetically generated testing data, and real-time monitoring, points toward a structural conclusion: CI pipelines test the scenarios you anticipated, but production surfaces the ones you did not.
The offline/online loop described above only works if you have infrastructure capturing traces and running evaluators on live traffic. CI gates prevent known regressions. Production monitoring catches what your test suite never imagined. The two complement each other.
For teams already running AI agent testing in CI, adding production-level evaluation closes the gap between what you tested and what your users experience. Production traces become tomorrow's test cases, and test results inform tomorrow's evaluation policies.
That feedback loop, not a larger test suite, is what makes llm testing work at production scale.
shim sits in the request path across every model and tool call, capturing the traces your evals need and enforcing guardrails at runtime so production failures your test suite never imagined get caught before users see them.