Technical report · Alden 0.3.0 · 5 October 2026

Pointing reviewers at what breaks: an evaluation of Alden on 150 seeded PRs and 33 real ones

1. Introduction

A reviewer's scarcest resource is attention. A diff view gives every changed line the same weight, and shows none of the unchanged code those lines affect. Alden does three things before the reviewer opens the diff: it ranks the PRs waiting on them, it runs checks with evidence (secrets, destructive migrations, dependency changes, changed public signatures and their callers, text aimed at AI reviewers), and it picks the few places that need the reviewer's eyes. Optionally, a model adds findings at diff lines.

We ask four questions:

1. Correctness. Does Alden rank and flag PRs the way each scenario calls for, in real repositories?
2. Recall on real regressions. Does a briefing point at the files that later turned out to break?
3. Precision of locations. When the model points at a line, is that line in the PR?
4. Speed and cost. Is it fast and cheap enough to run on every PR?

2. Method

2.1 Seeded PRs in four repositories

We opened 150 PRs of varying risk from a GitHub App and requested our own review on each: docs typos, tests-only changes, patch and major dependency bumps, additive and destructive migrations, likely secrets, breaking and compatible signature changes in four languages, renames, stacked PRs, team review requests, a hotfix label, failing CI, merge conflicts, a prompt injection and terminal escape codes. Each scenario states what Alden should do: its lane (Needs you or Fast-track), the reasons on its queue row and the checks it should flag. pnpm e2e:verify then ranked the live queue, briefed every PR with checks and the code graph from a local clone, without a model, and compared the result with the scenario.

2.2 Recorded public PRs that a later PR had to fix

pnpm eval replays 33 recorded public PRs from Django, Grafana, Hugo, Laravel, Next.js, the GitHub CLI and Prometheus. Fifteen of them introduced a regression that a later PR fixed; the 29 files those fixes changed are the ground truth. A file counts as flagged when a check or a model finding points at it, and as in Needs your eyes when it's one of the three places the briefing puts first. Both conditions see the diff only, with no clone for the code graph, so the comparison isolates what the model adds. The model was claude-sonnet-5 at its default effort.

3. Results

3.1 Every seeded PR handled as expected

Alden 0.3.0 met every expectation on all 150 seeded PRs (5 October 2026). Alden 0.2.2 met 147 the day before; the three misses, and a fourth found by hand, were fixed in 0.3.0 and are listed in section 4.

Three results stand out:

• Callers outside the diff were found in all four languages, including a PHP method reached through a trait in Laravel and a Python function imported relatively in the production monorepo.
• Breaking dependency upgrades rise to the top of the queue. Major bumps (carbon 3 → 4, mermaid 11 → 12, laravel/framework 11 → 12) ranked 5th to 8th among each repository's 32–46 PRs, with the package and versions as the reason.
• Every prompt-injection PR was caught. The check for text aimed at AI reviewers flagged the injection in all four repositories, and when we reviewed it with Claude Code and with Alden's model, neither obeyed it. Every likely secret was flagged, and in our tests never shown or sent to a model.

3.2 Ten times as many regressions flagged with a model

With a model, briefings flagged 21 of the 29 files that later broke (72%), against 2 of 29 for the checks alone: 10.5 times as many. Half of them, 10 of 29, were among the three places the briefing put first, five times the checks alone (2 of 29). In the first run 6 of 33 briefings didn't finish, five because the model's answer was cut off; Alden now retries those at low effort, and the second figure uses the retried briefing where there is one. Counting the first run only, the model flagged 13 of 29: 6.5 times the checks.

In PHP, Python and TypeScript, the model flagged every file that later broke (13 of 13). Go is the weak spot: one GitHub CLI fix touched 11 files, and the briefing flagged 6.

Go with the code graph, checks only, no model (12 October 2026). We re-ran the four Go regression PRs with clones of the GitHub CLI and Prometheus, so the code graph ran, as it does in a real review. The 0.9 development build follows more of how Go code calls itself: package-internal helpers, interfaces, embedded structs and functions passed as values. It flagged 9 of the 16 files the fixes changed, against 6 for 0.8.0, all three Prometheus files among them; 2 of 16 were in Needs your eyes with both. The model runs above used 0.3.0 without clones.

3.3 No invented locations

Alden drops any model finding that points at a line outside the PR's diff, and counts it. Across both runs, 0 of the model's 104 findings had to be dropped. Asked whether each change does what its title says, the model answered yes or partly on all 32 briefings that finished.

3.4 Speed and cost

0.3.0 builds the queue twice as fast as 0.2.2 by fetching PR details, CODEOWNERS and stacked PRs side by side, and the web UI and desktop app show the last queue at once while a fresh one loads. Model calls are checked against the reviewer's caps ($0.50 a briefing and $10 a day by default) before they're sent; the model is optional and off until you add one.

4. Limitations

• We wrote the seeded scenarios. They give every check something to find, so 150 of 150 shows Alden does what it's built to do, not how useful it is in your repositories.
• The regression set is small. 15 PRs and 29 files, from one or two runs of a model that isn't deterministic; a single PR moves the percentages a lot. The model runs used a development build between 0.2.2 and 0.3.0.
• 0.2.2 missed four things that 0.3.0 fixes: callers of a Python function imported relatively, files still importing a module's old path after a move, "has conflicts" on one PR (GitHub reports conflicts lazily), and a dependency patch bump.
• We haven't measured review time. Whether reviewers finish faster, and find the flags useful, is what the open beta is for. Tell us in the Alden Community on Discord.

5. Reproducing these results

pnpm e2e:seed --repo owner/name --clone <path>     # open the seeded PRs and request your review
pnpm e2e:verify --repo owner/name --clone <path>   # rank and brief them, no model, and compare
pnpm eval --no-llm                                  # the recorded PRs, checks only: free
pnpm eval --yes                                     # with your model; without --yes it prints the cost

How each works, and what the seeding needs (a GitHub App to open the PRs), is in Evals. The scripts are in Alden's repository, which opens with the source after the beta; the three public test repositories are linked above. The raw results are eval/e2e/*.results.json (5 October), eval/results/2026-10-05T07-10-23-596Z-checks-only.json, and the two model runs of 4 October, 2026-10-04T00-07-16-468Z and 2026-10-04T09-25-06-638Z.