LLM Evaluation

LLM evaluation is how teams measure whether a language-model feature behaves well enough to ship and stay reliable after it ships. It covers prompt quality, retrieval accuracy, grounded answers, agent task success, policy compliance, latency, and cost.

Evaluation is not a one-time benchmark score. It is a loop that runs before release and continues in production as models, prompts, documents, and tools change.

Offline vs online evaluation

Offline evaluation catches regressions early. Online evaluation tells you whether the lab results still match reality.

flowchart LR
Change["Prompt, model, RAG, or tool change"] --> Offline["Offline eval"]
Offline -->|"Pass"| Release["Release"]
Offline -->|"Fail"| Fix["Fix or block"]
Release --> Online["Online monitoring"]
Online --> Review["Human or automated review"]
Review --> Dataset["Update golden set"]
Dataset --> Offline

Both sides need ownership. A green offline run does not remove the need to watch production.

Golden sets and regression tests

A golden set is a small, curated collection of representative inputs and expected qualities. It should include:

• Common happy paths users actually ask for.
• Missing-context cases where the product should refuse or ask for more information.
• Ambiguous requests that test judgment and formatting.
• Adversarial cases for prompt injection, missing context, malformed JSON, and policy edge cases.
• Workflow cases for AI agents, not only single-turn chat.

Store prompts, retrieval settings, tool definitions, and model versions with each test case. A passing result without that context is hard to reproduce.

Treat golden sets as product assets. Review failures from logs, support tickets, and incident reports, then turn repeated failures into permanent tests.

What to measure by layer

Different AI features fail in different places. Evaluate the layer that owns the risk.

Prompt-level quality

For prompt engineering, measure:

• Format validity when structured output is required.
• Tone, length, and completeness against a rubric.
• Correct refusal when evidence is missing.
• Stability across paraphrased user inputs.

RAG quality

For retrieval-augmented generation, measure in layers:

• Retrieval recall, did the retriever find the chunk that contains the answer?
• Retrieval precision, are top results relevant, current, and allowed for this user?
• Groundedness, does the answer stay within retrieved evidence?
• Answer usefulness, is the response complete enough for the task?
• Operational quality, latency, index freshness, cost, and failure rates.

Retrieval and generation should be tested separately. A fluent wrong answer is still a failure.

Agent quality

For AI agents, measure the whole loop:

• Task success or partial success against a clear definition of done.
• Tool error rate, retry count, and invalid argument rate.
• Human intervention rate and approval turnaround.
• Policy violations blocked by guardrails.
• Cost, latency, and iteration count per task.

Keep representative traces so a failed run can be replayed after a model, prompt, or tool change.

Automated, human, and model-based review

Teams usually combine methods:

• Deterministic checks, schema validation, citation presence, regex or rule checks, allowlists.
• Human review, best for nuanced quality, safety, and new product areas.
• Model-based grading, useful at scale when rubrics are explicit, but risky as the only judge.

Model-based evaluation can inherit the same blind spots as the system under test. Use it for triage and trend detection, not as the sole source of truth for safety-critical behavior.

Production signals to track

Online evaluation should answer whether quality is drifting without a code deploy:

• User corrections, thumbs down, re-asks, and escalations to support.
• Refusal rate and false-refusal rate.
• Citation or evidence miss rate in grounded flows.
• Retrieval empty-rate and permission-filter drops.
• Agent loop length, tool failure rate, and spend per successful task.
• Latency p50 and p95, token usage, and cost per successful outcome.

Set thresholds for alerts, not only dashboards. A metric nobody acts on is decoration.

When evaluation must rerun

Quality can shift even when application code does not change:

• Base model upgrades or provider changes.
• Prompt or tool-definition edits.
• Corpus updates, embedding model changes, or chunking strategy changes.
• Authorization or metadata filter changes in retrieval.
• Guardrail rule changes.

Version these inputs together. “Same feature, new model” is a release and should pass the same evaluation gate.

Common pitfalls

• Measuring only final answer fluency while retrieval or tools fail quietly.
• Using a tiny golden set that only covers demo questions.
• Skipping failure-case tests because they are harder to grade.
• Treating offline success as proof that production is safe.
• Changing prompts, models, and retrieval settings in one deploy with no way to attribute regressions.
• Ignoring cost and latency until the feature becomes too expensive to operate.

Trade-offs

• More evaluation slows iteration unless tests are automated and fast.
• Human review is high signal but does not scale without sampling strategy.
• Overfitting to a golden set creates a product that passes tests but fails real users.
• Under-testing creates silent drift, especially in RAG systems where documents and embeddings change often.

The goal is not perfect scores. The goal is to know when behavior got worse, why it got worse, and whether the product is still fit to ship.