How to Evaluate LLMs Before You Ship to Production
GitHub's team shares a practical framework for evaluating LLMs before production, drawn from building a secret-scanning false-positive reducer.
The GitHub team published a detailed breakdown of how they evaluate LLM-based systems before pushing them to production, using their secret-scanning false-positive reducer as the working example. The core finding: benchmark scores during prototyping do not reliably predict production behavior, because real inputs are noisier, labels are inconsistent, and edge cases that rarely appear in test sets can become the main source of failure once traffic is live. They share a structured, repeatable evaluation framework that any team building on top of an LLM can follow.
What happened
GitHub’s engineering team published a guide on evaluating LLMs before production, drawing directly from the work they did on a system designed to reduce false positives in GitHub Secret Scanning. That product identifies credentials, such as API tokens and keys, that have been accidentally committed to a repository. Some candidate strings look like secrets but are not real credentials, so developers waste time on alerts that need no action. The team needed to know whether an LLM could filter out that noise without accidentally suppressing a genuine credential.
| Concept | Detail |
|---|---|
| Use case | Reducing false positives in GitHub Secret Scanning |
| Primary metric | False-positive reduction and precision |
| Safety constraint | Recall must stay within a predefined acceptable range |
| Operational guardrails | Latency, cost, reliability, production compatibility |
| Evaluation trigger | Any meaningful change to prompt, model, inputs, or business logic |
Why it matters
The problem the GitHub team describes is not unique to secret scanning. Any LLM-based feature, whether it is a support bot, a document classifier, or a code reviewer, faces the same gap between prototype results and production reality. Offline metrics can improve while actual user-facing behavior gets worse, because the evaluation set may not reflect the real input distribution.
The practical consequence is that teams keep shipping LLM features with misplaced confidence, then scramble to diagnose regressions after launch. A repeatable evaluation framework closes that gap before it becomes a customer-facing problem.
If you are building AI integrations for clients, this is the kind of discipline that separates a prototype from something you can actually stand behind in production.
Start with the product decision, not the model
The GitHub team’s first rule is to define what success looks like for the user before touching any technical component. When an LLM system underperforms, the instinct is to rewrite the prompt or switch models. But without a clear decision framework, you end up optimizing a metric that does not actually matter to the product.
They organized their criteria into three tiers:
- Primary outcome: the user benefit being improved (false-positive reduction, precision).
- Safety constraint: a metric that cannot drop below a set threshold, even if the primary outcome improves (recall, in their case).
- Operational guardrails: practical limits on latency, cost, reliability, and production compatibility.
The tiered structure means a change that dramatically improves precision but drops recall below the safety threshold is automatically rejected, not debated. The team illustrated this with two hypothetical experiments:
| Experiment | Precision | Recall | Latency | Decision |
|---|---|---|---|---|
| Experiment A | Large improvement | Falls below safety guardrail | Acceptable | Do not advance |
| Experiment B | Moderate improvement | Remains within guardrail | Acceptable | Continue testing |
Experiment A looks better in isolation. Experiment B is the right call because it improves the developer experience without introducing unacceptable security risk.
Treat offline evaluation like integration testing
The second principle is that evaluation is not a one-time event. Prompts get revised, models get swapped, input construction changes. Each of those changes can introduce a regression. The GitHub team ran their offline evaluation after every meaningful change, comparing each result against a recorded baseline.
For every run, they logged the prompt, model version, dataset version, and full system configuration. That logging discipline lets you answer questions like: did the new prompt improve precision without hurting recall, or did it just shift where errors appear?
They also stress changing only one major variable per experiment. Without that discipline, you can attribute an improvement to the wrong cause and spend weeks chasing a dead end.
Why benchmark scores alone are not enough
Curated benchmarks are useful early on. They help teams compare models quickly and test whether an idea is technically feasible. But according to the GitHub team, several things go wrong as a system moves toward production:
- Real inputs are often ambiguous or missing context.
- Labels in the evaluation set may be inconsistent.
- Edge cases that are rare in benchmarks can become common in production traffic.
- The evaluation set may not match the actual distribution of inputs the system will see.
This is why the team’s practices apply broadly to code analysis, developer tools, security workflows, data analysis, and other production systems, not just secret scanning.
For teams tracking multi-agent AI rollouts, this evaluation gap is one of the key reasons so few organizations have successfully scaled beyond pilots.
Our take
The GitHub team’s framework is genuinely useful and refreshingly concrete. The tiered metric structure (primary outcome, safety constraint, guardrails) is something most teams skip because it requires a product conversation before the engineering work starts. That conversation is exactly what makes the difference between a demo and a deployed feature.
The weakest part of the article is that it cuts off before covering how to handle label quality and distribution shift in detail. Those are the hard problems in practice. But the foundation it lays, clear success criteria, versioned experiments, one variable at a time, is solid and directly applicable.
If your team is prototyping an LLM feature right now, the simplest thing you can do today is write down one safety constraint that cannot be violated, before you touch the prompt.
What to do about it
- Define your primary outcome metric and at least one hard safety constraint before writing or revising any prompt.
- Set up a repeatable offline evaluation: log the prompt, model version, dataset version, and system config for every run.
- Change one major variable per experiment (prompt, model, input construction, or business logic) so you can attribute results correctly.
- Run the evaluation again after every meaningful system change, not just at the start of the project.
- Check whether your evaluation dataset actually reflects the distribution of inputs your system will see in production, and update it if it does not.
A good evaluation framework is the difference between a prototype you demo and a system you trust. If you want help building that discipline into a project, talk to the Lumien team.
Frequently asked questions
Why do LLMs that perform well on benchmarks fail in production?
Benchmarks use curated, clean data that often does not match real production inputs. In production, inputs are ambiguous, context may be missing or truncated, labels can be inconsistent, and edge cases that are rare in test sets become common sources of failure.
What metrics should I use to evaluate an LLM system?
GitHub recommends organizing metrics into three tiers: a primary outcome metric tied to user benefit, a safety constraint metric that must stay above a set threshold, and operational guardrails covering latency, cost, and reliability. Treat these as separate tiers, not interchangeable numbers.
How often should you run offline evaluation for an LLM system?
According to GitHub's team, you should rerun your offline evaluation after every meaningful change to the prompt, model, input construction, or surrounding business logic. It should function like an end-to-end integration test, not a one-time exercise.
What is the difference between precision and recall in LLM evaluation?
Precision measures how often the model's positive predictions are correct. Recall measures how many actual positives the model correctly identifies. In security-sensitive systems like GitHub's secret scanning, recall is treated as a hard safety constraint because missing a real credential is more dangerous than a false alarm.
