One Label In, Merge-Ready Out: Automating Community PRs

Every open source maintainer knows the queue. A contributor you have never met opens a pull request. It might be a one-line typo fix or a 900-line refactor with a two-sentence description. Either way, someone has to read it, rebase it, run it, argue with CI, ask for changes, wait a week, and then do most of that again. The contributor's time was free. The maintainer's was not.

Coding agents have made this worse before they make it better. More people can now produce a plausible PR, so the queue grows, while the cost of actually adopting a PR into the project has barely moved. The obvious response is to point an agent at the queue. The less obvious part is deciding what the agent is allowed to do, what it is never allowed to do, and where a human has to stay in the loop.

We have been studying a production workflow that one open source project runs on exactly this problem. We will not name it, but the design is worth walking through, because most of its decisions generalize to any project that wants to automate community PRs without handing the merge button to a model. Here is one PR's run, start to finish.

The whole interface is one label

A maintainer looks at a contributor's PR and applies a label: status/needs-review. That is the entire interface. A small poller watches for the label, records a durable request so nothing is lost if the system restarts, launches a run, comments on the PR with a link to the live run, swaps the label for status/reviewing, and gets out of the way.

Notice three things about that opening move.

The trigger is a human decision. The automation does not pick PRs up on its own. A maintainer still glances at the PR and decides it is worth the project's compute. That one glance filters out spam, off-roadmap features, and PRs that need a conversation before they need a review.

Labels are the state machine. The PR moves through needs-review, reviewing, awaiting-approval, merge-ready, and merge-queued, and every one of those is visible to the contributor, the maintainer, and any other tooling. There is no dashboard to learn. The PR page is the dashboard.

Every write goes out as a GitHub App. Comments, labels, and pushes are all attributed to the project's bot, never to a maintainer's personal account. Contributors can tell at a glance which actions were automated, and maintainers never wake up to find their name on a push they did not make.

Preflight before anyone spends a token on review

The first real step is not review. It is a preflight check in an isolated worktree. A script verifies the PR's identity, and a model reads the description against the diff and asks one question: does this PR do what it says it does? Clear drift gets a polite comment, a status/needs-info label, and a full stop. Nobody reviews code that does not match its own description, because the review would be arguing with the wrong thing.

If the description holds up, the workflow rebases the PR onto the current main branch and repairs any conflicts locally. Nothing is pushed yet. If the contributor's fork cannot be pushed to, the workflow notes that now, so it can open a maintainer follow-up PR later instead of discovering the problem at the end.

For maintainers, this is the step that pays for itself fastest. A large share of community PRs are stale against main by the time anyone looks at them, and "please rebase" is the single most common reason a contribution dies in the queue.

Review and CI start together, and CI never blocks review

Once the rebase lands, two things start in parallel: the review loop and a CI snapshot.

The CI snapshot is deliberately dumb. It records the state of the rebased head's checks, and it never waits and never repairs. The reason is subtle but important: the reviewers are reading the same worktree at the same time, and a CI repair that edits files mid-review would hand the reviewers a moving target. So red CI is captured as evidence and handed to the fix step instead. Review no longer waits on a 20-minute test suite, and the suite's failures still get fixed.

Two model families, three different jobs

The review itself is not one model reading a diff. It is parallel lanes from different model families, each with a distinct job.

• A reasoning lane checks intent fidelity and invariants, and asks what happens when things fail halfway or interleave. It writes its findings with path:line evidence.
• An evidence lane collects first and judges second. It traces every caller, test, and contract the diff touches before it says anything.
• A cross-file consistency lane is optional, gets a single attempt, and soft-fails. If it is rate-limited or down, the run continues with two lanes.

Then a synthesis step merges the lanes into one decision report. Findings are deduplicated and rated blocker, major, minor, or nit, and where two lanes disagree on severity, the stricter one wins. The synthesis then runs a deterministic scorecard: a 1,000-point budget, a gate at 850, and a hard rule that any open blocker or major means request changes regardless of score.

The lesson for maintainers is not "use three models." It is that review quality comes from disagreement, and disagreement is cheap if you design for it. Different model families miss different things. Giving each lane a different job, rather than three copies of "review this PR," is what makes the overlap useful instead of redundant. And ending in a deterministic scorecard means the pass/fail decision is reproducible, which matters when a contributor asks why their PR was sent back.

The fix loop is bounded and leaves the contributor's commits alone

When the synthesis requests changes, the workflow does not comment and wait. It fixes every blocker and major itself, plus any red CI the snapshot found, in a single bot-authored fixup commit on top of the contributor's commits. Later passes amend that same commit rather than adding more. The contributor's own commits are rebased onto main but never rewritten or squashed, so the history still tells the true story of who did what. And it is still the same PR: the reviewed head is pushed to the contributor's PR branch as the project's GitHub App, which works on a fork as long as the contributor left maintainer edits enabled. Only when the fork cannot be pushed to does the workflow fall back to a maintainer follow-up PR carrying the same commits.

Then the loop runs again, from the top, with a fresh full review. It is capped at five passes. Each pass ends with a check that reads the review verdict: iterate starts another pass, done exits. But before done exits, the check attests the exact commit the reviewers saw. If the fix step touched code after the clean synthesis, done quietly becomes one more pass. From that point on, every push, approval, and merge-ready handoff has to match the attested head. The project calls this the review coverage gate, and it closes the oldest hole in automated review: approving one version of the code and merging another.

Before a human is asked, CI has to be green

A second gate sits between the review loop and the maintainer. The reviewed head must still match what is on GitHub, and the required checks must pass. If they are red, a repair step fixes only what the CI evidence supports, and failures that already exist on main are allowed through rather than looped on forever.

A repair that changes reviewed code does not trigger a full re-review. It triggers a delta review of just that change, by the reasoning lane. If the delta review requests changes, the PR goes back to the full loop. Pending CI waits without using up repair attempts. Five passes, then it fails hard.

The human is a gate, not a spectator

Only now does the workflow stop and ask. It posts an approval packet on the PR: what the PR does, the review decision, a short summary of the review, the fixes it made, the CI state, and the merge path. It applies status/awaiting-approval and emails the maintainer. Only an explicit "approve" or "proceed" from a maintainer closes the gate.

An auto-approve mode exists for the project's own end-to-end tests. It is never used on contributor PRs.

This is the design choice we would most encourage other projects to copy. The automation has done the reading, the rebasing, the fixing, and the arguing with CI. What is left for the human is one decision with everything needed to make it on a single PR page. That is a review a maintainer can do in two minutes between other things, instead of the forty-minute context switch that community PRs used to demand.

Finalize marks merge-ready. It never merges

The last body step is the only one allowed to change merge state, and even it does not merge. It re-verifies the approval, refuses any head the review does not cover, refreshes onto the latest main, pushes with the contributor's commits intact, waits for green CI, posts a maintainer adoption review, and applies status/merge-ready.

A separate merge queue, serial per repository, takes it from there. Keeping merge out of the adoption workflow entirely means a bug anywhere in the review pipeline can, at worst, mark a PR ready. It cannot land one.

Failure is designed, not handled

Everything described above is wrapped in a scope. If any body step fails hard, the whole scope aborts and the run goes straight to cleanup. Every loop is bounded. Every retry has a count. The optional lane soft-fails. And on failure, cleanup keeps the worktree so a human can see exactly what the agent saw.

Model routing follows the same discipline. Heavy steps like rebase, fixes, and finalize get the most capable model at maximum effort. Infrastructure steps like recording CI or cleaning up get a cheap configuration. Gates and checks run as plain shell scripts with no model at all. The result is that the expensive thinking happens exactly where it changes the outcome.

What to copy

If you maintain an open source project and want to automate community PRs, the pattern reduces to a short list.

1. Make the trigger a human label, and make the labels your state machine.
2. Preflight for description drift and rebase before you review anything.
3. Run review and CI in parallel, and treat red CI as evidence, not a blocker.
4. Review with more than one model family, give each a different job, and end in a deterministic score.
5. Fix in one bounded loop, never rewrite the contributor's commits, and attest the reviewed head.
6. Put a human gate after CI is green and give them a complete packet to decide from.
7. Mark merge-ready. Never merge from the review pipeline.
8. Bound every loop, keep the worktree on failure, and spend the expensive model only where it matters.

None of this removes the maintainer. It moves them from the beginning of the process, where they used to do all the work, to the end, where they make the one decision that should still be theirs.

Actual AI builds an architecture agent that manages architectural context for coding agents and software factories, so the rules your review lanes enforce stay in step with the codebase they are reviewing.