Case study · Agentic systems in production
AI code review, zero to production
A GitHub App that reviews every pull request in the context of the whole repository — with an eval harness gating every prompt and model change.
Context
Reviewing a pull request from the diff alone is the reason most automated review is noise. The change that looks fine in isolation is the one that breaks a caller two directories away, or contradicts a test nobody opened.
The system had to read a change the way a senior engineer does — following it out into its neighbours and its tests — and it had to be cheap enough per PR to leave switched on.
Scale
Runtime
A single AWS Lambda — no database, no framework, no LangChain
Model
Claude on Amazon Bedrock, native tool-use loop
Status
Zero to production
What I decided and built
- ›Wrote the agent as a native Claude-on-Bedrock tool-use loop rather than adopting a framework — the control flow is small enough that a framework would have cost more than it returned.
- ›Moved review from diff context to repo context: the reviewer checks out the whole repository and reads each change against the application around it — the changed files, their neighbours, and their tests. This was the single change that moved review quality most.
- ›Wrote a language-agnostic review rubric, extended per repository by a config file the repo owns, so teams tune the reviewer without touching the service.
- ›Kept the whole service in one Lambda with no database, so there is no state to operate and the per-PR cost stays predictable.
- ›Built an offline eval harness scoring review quality on recall, precision and stability, and gated every prompt and model change behind it — the change ships only if the scores hold.
- ›Deployed as infrastructure as code with AWS CDK and least-privilege IAM.
Outcome
- ◆Posts small, friendly, line-anchored comments on each pull request rather than a wall of prose.
- ◆Prompt and model changes are gated on measured review quality instead of impressions.
What I'd do differently
- ›Start with the repository, not the diff. The first version reviewed the changed lines and the files immediately around them, and it read like every other automated reviewer: technically correct, contextually useless.
- ›Having it check out the repo and reason about the change inside the whole application transformed review quality. That should have been the first commit, not a correction — reviewing a diff in isolation is the thing that makes AI review noise, and I built the noisy version first.