AI code review, zero to production
A GitHub App that reviews every pull request in the context of the whole repository, with an eval harness gating every prompt and model change.
At a glance
- Role
- Designed and built it end to end: agent loop, review rubric, GitHub App, AWS infrastructure and eval harness
- Engagement
- Independent contractor, remote from Dubai
- Timeframe
- 2024 – present
- Team
- Solo build; used by the frontend team I lead
- Stack
- TypeScript · Claude on Amazon Bedrock (native tool use, no framework) · AWS Lambda · AWS CDK · Secrets Manager · Octokit · Zod
- ◆100+ real pull requests reviewed in production
- ◆0 false positives on the eval fixture set (TypeScript / C# / Swift)
- ◆~$0.02 per pull-request review
Context
Reviewing a pull request from the diff alone is why most automated review is noise. The change that looks fine in isolation is the one that breaks a caller two directories away, or contradicts a test nobody opened.
The reviewer had to read a change the way a senior engineer does, and it had to be cheap enough per PR to leave switched on. It also had to stay changeable: a reviewer built on a non-deterministic model is one prompt edit away from getting quietly worse, and nobody notices until the team stops trusting its comments.
Constraints
- ›Trust: one noisy comment costs more than one missed finding, because a team that learns to ignore the bot ignores all of it.
- ›Non-determinism: the same prompt can review the same code differently on two runs, so a single passing run proves nothing.
- ›Security: repository code must never execute inside the reviewer, and there must be no static keys to leak.
- ›Cost: cheap enough per PR that nobody has a reason to switch it off.
Architecture
A pull request opens or updates; GitHub calls the App's webhook on a single AWS Lambda behind a Function URL, with no database. The Lambda reaches Claude on Amazon Bedrock through its IAM role, with no static model keys, and reads the GitHub App's private key from AWS Secrets Manager. The agent is a native tool-use loop with three tools: read_file and list_dir, which read the repository through the GitHub API with no checkout and no clone, and submit_review, which posts one review per run with line-anchored comments and a status check. Prompt caching keeps repeated context cheap, a model fallback keeps reviews flowing when the primary model is unavailable, and per-app cost attribution makes the bill visible. Everything deploys with AWS CDK under least-privilege IAM.
export function AlarmList({ siteId }: Props) { const [alarms, setAlarms] = useState<Alarm[]>([]); useEffect(() => { fetch(`/api/sites/${siteId}/alarms`).then((r) => r.json()).then(setAlarms); }, [siteId]);Key decisions
| Decision | Why | Trade-off |
|---|---|---|
| A native Claude tool-use loop, no framework | The control flow is small enough that a framework would cost more than it returns. | Retries, fallback and caching are code I own and test myself. |
| Read the repository, not just the diff, through the GitHub API (read_file, list_dir), with no checkout | The agent follows each change into its neighbours and its tests. This moved review quality more than any prompt change, and with no clone the Lambda stays stateless and never runs repository code. | More GitHub API calls per review, and the agent only sees what it chooses to read. |
| One submit_review per run | A PR gets one coherent review and one status check instead of a stream of comments. | Nothing appears until the run finishes. |
| One Lambda, no database | No state to operate, and a predictable per-PR cost. | No memory across runs; anything persistent lives in GitHub. |
| A language-agnostic rubric, extended per repository by a config file the repo owns | Teams tune the reviewer for TypeScript, C#, Swift or Python without touching the service. | A repo can tune its own reviewer badly; the eval gate only covers the shared rubric. |
Evals
A fixture is a small repository with deliberately planted violations of the rubric, in TypeScript, C# or Swift. The planted violations are the labels: each fixture knows exactly which findings a correct review contains, so the harness scores recall (planted findings caught), precision (comments that match a planted finding) and stability across repeated runs. It runs against a local filesystem copy of each fixture, with no AWS and no GitHub in the loop.
Because the model is non-deterministic, every fixture is replayed several times and scored on the spread, not one lucky pass. Every prompt and model change has to hold those scores to ship, and on the current fixture set the reviewer produces zero false positives.
The gate is what makes the reviewer safe to improve: a prompt edit that raises recall by finding more real problems ships, and one that raises it by commenting on everything fails on precision.
Outcome
- ◆100+ real pull requests reviewed in production.
- ◆~$0.02 per pull-request review, with cost attributed per app.
- ◆0 false positives on the eval fixture set, across TypeScript, C# and Swift fixtures.
- ◆Prompt and model changes ship on measured review quality, not impressions, which is why the rubric and the model choice can keep moving.
What I'd do differently
- ›Start with the repository, not the diff. The first version reviewed the changed lines and the files right around them, and read like every other automated reviewer: technically correct, contextually useless.
- ›The fix was giving the agent tools to walk the repository and reason about the change inside the whole application. It transformed review quality, and it should have been the first commit, not a correction.
Data fetching inside the component. src/features/alarms/useAlarms.ts already loads this list with cancellation and error state, and AlarmPanel uses it. Call useAlarms(siteId) here instead, so both views share one cache and one failure path.
Rubric: data fetching belongs in hooks, not components.