Context
Review capacity is the bottleneck on a platform team serving 250+ engineers. Every pull request needs someone to check it does what the ticket said, and that nothing sensitive slipped into the diff. Both are mechanical checks that consume senior attention.
The problem
The obvious move, putting an LLM on pull requests, fails in a specific way: a model that always produces a confident review is worse than no review. Reviewers learn to ignore it, or worse, trust it. The hard requirement was not accuracy on average, it was knowing when the model could not verify something.
- step: Pull requestopened
- step: GitHub Actiontriggers review
- data: Jiraacceptance criteria
- step: Claude Codereviews diff vs criteria
- control: Verdictpass · fail · unknown
- result: Human reviewersees the summary
Approach
- 01
Ground the review in the ticket, not the diff alone
The reviewer pulls the linked Jira issue and checks the change against its acceptance criteria. Without that anchor the model reviews style; with it, the model reviews intent.
- 02
Calibrate the prompt against real reviews before widening
Rolled out incrementally rather than switched on everywhere, across roughly five major prompt versions. The first was too generic and produced noise, so the scope narrowed to what actually needs context to judge: missing Jira acceptance criteria, exposed secrets, broken authorization, unsafe configuration, and material implementation risk. Formatting, imports and style were left to linters deliberately. A reviewer that comments on everything trains people to skim it.
- 03
Make Unknown a first-class verdict
Every acceptance criterion comes back Verified, Not Satisfied, or Unknown. Unknown is not a failure mode, it is the point: it fires when satisfying a requirement depends on something the reviewer genuinely cannot see, such as configuration in another repository, external infrastructure, another service, or runtime state. The number of Unknowns was never something to minimise. An agent that says it cannot verify something, and hands that judgement back, is worth more than one that manufactures confidence.
- 04
Scan for exposed credentials on the same pass
Secrets in a diff are cheap for a model to spot and expensive for a human to miss. Folding it into the same review means no extra pipeline stage.
- 05
Ship it as a GitHub Action, not a service
Running as an Action meant adoption per repository was a file, not an onboarding. That is what took it from 30 repositories to 70+.