Skip to main content
Back to Blog

AI Pull Request Review: Why the Newest Objection Vetoes

Shay

AI agents write almost all of the code at Scheduler Systems. I don't review every pull request myself. A separate agent reviews and scores each one, a gate written in plain code decides whether it can merge, and I sign off on the few changes that are risky.

One day that gate passed a pull request scored 92 out of 100 while its newest review said "Blocking". The score was high, so the gate let it through. The objection that arrived after the score didn't count.

This post covers what we changed: three rules for AI pull request review, three ways our own automation slipped past a block, and a five-point checklist you can apply to any GitHub repo.

How our AI pull request review works

Every change goes through four steps:

  1. An author agent writes the change and opens the pull request.
  2. A reviewer agent that did not write the change reads the diff and fills in a weighted scorecard out of 100. The scorecard covers correctness, security, tests and CI, evidence, maintainability, scope, documentation and operational risk, with correctness and security weighted most. If the total clears our pass mark, the reviewer approves that exact commit.
  3. A gate decides. It makes no model call and no judgment about quality. It answers one question: has an independent reviewer approved this exact commit, and is anyone still blocking it?
  4. A human, me, signs off on the changes that can do the most damage.

The gate is boring on purpose. Judgment lives in the review, and the gate checks only facts that GitHub records.

Rule 1: the newest objection vetoes

The 92/100 failure came from treating the score as the verdict. To the gate, a blocking review that arrived later was just one more review.

Now a blocking review outranks any earlier score, however high. Only the reviewer who raised the block can clear it. The author can't clear it, another bot can't, and a new commit can't.

Rule 2: a review counts only for the exact commit it reviewed

An approval of yesterday's commit says nothing about the code you merge today. GitHub records a commit_id on every pull request review, so you can check this instead of assuming it.

Our gate compares the approval's commit with the pull request's current head. If you change the patch, even by whitespace, you need a new review. We allow one exception: a new head whose patch against the base is identical, such as a clean rebase.

Notice the asymmetry, because it matters later: approvals bind to one commit, and blocks do not.

Rule 3: an author never scores its own work

On 2026-08-12 we counted. Across our last 40 pull requests there were 50 comments, all written by the pull request's own author, and zero formal reviews. Our policy said every change got reviewed, but nothing checked that it did.

Now the gate compares identities and ignores any review by the author. One GitHub detail caught us here. The same app can appear as app/name in the GraphQL API and as name[bot] in the REST API. If you compare the raw strings, an app's review of its own pull request looks independent. Normalize logins before you compare them.

Three ways automation slipped past a block

Once rule 1 existed, our own automation found three ways around it. None was malicious. Each was a shortcut that looked reasonable when it was written.

  1. A comment after the block. The gate read only the newest review. A human requested changes and the gate reported "needs a review". A bot then posted a plain comment, so the newest review was a comment, and the gate passed.
  2. A new commit. To enforce rule 2, the gate first filtered reviews down to the head commit. A block on commit A disappeared as soon as commit B was pushed, so the author could clear a human's block just by pushing.
  3. The blocker's own follow-up. Then the gate switched to "latest review per reviewer". A reviewer who blocked and then commented to answer the author turned their own block into a pass.

One fix covers all three. We track each reviewer's state across every commit, and only an approval or a fresh block from that same reviewer changes it. Comments change nothing. This matches how GitHub itself behaves: a reviewer stays in "changes requested" until they approve or the review is dismissed.

A comment is not a vote.

Fail closed

If the gate can't read the pull request, its reviews or its author, the answer is "unmeasured", and unmeasured blocks. An error inside the gate blocks too. A crash must never look like "no review yet", or the automation will go and try to satisfy the gate.

A gate that opens when its inputs are missing treats silence as success. That is the class of bug we keep removing.

What stays human in the loop

Agents write the code, and agents review it. Some changes still wait for me: production deploys, live billing changes, security deploys and anything that affects a paying customer. Those need a human sign-off on top of the review and the gate.

That is our working answer to human in the loop for AI agents. We don't put a human on every pull request. We put one on the changes that are expensive to undo.

A checklist for AI pull request review on GitHub

If agents open pull requests in your repo, these five checks are cheap to write against GitHub's pull request reviews API:

  1. The review's commit_id equals the pull request's head commit.
  2. The reviewer is not the author, after normalizing both logins.
  3. For each reviewer, take the latest APPROVED or CHANGES_REQUESTED across all commits, and ignore comments.
  4. A COMMENTED review is never an approval.
  5. Missing or unreadable data means block.

FAQ

Should AI review AI-written code? AI code review can work when the reviewer is independent of the author, its verdict is bound to the exact commit, and a human still owns the risky changes. If the reviewer shares the author's identity, it isn't a review.

Can a comment clear a requested change? Not on GitHub. A reviewer stays in "changes requested" until they approve or the review is dismissed. Your gate should behave the same way.

Where GAL fits

The gate above is how we run our own repo. Independent review scores on agent-written pull requests and human sign-off gates for risky changes are early access only, not shipped features.

GAL, a Scheduler Systems product, works on the input side of the same problem: the instructions and permissions your agents run with. It discovers agent configs across your repos, lets you set one approved standard, and syncs it to your developers with gal sync --pull. It covers Claude Code, Cursor, GitHub Copilot, Gemini Code Assist and OpenAI Codex.

You can read more about GAL at gal.run.

Tags:ai code review, agent governance, human in the loop
Back to Blog