Building a Reviewer That Still Thinks

"Reviewing code is a skill: Reviewing code solves many simultaneous tasks like spotting bugs, questioning design decisions, understanding unfamiliar code, transferring knowledge and maintaining project nuances. Especially in the age of AI, being able to reason deeply about code is valuable and getting better at reviewing code is a high-value skill" by Velda Kiara.

So when I built a small agent to review pull requests automatically, I tried to make it review the way a careful human would — not skim.

The project is a GitHub webhook service. When a PR is opened or updated, it verifies the request's signature, fetches the changed files, and filters out lockfiles and images. It then scans the diff for newly added imports Python, TypeScript, Rust, Go and asks the Exa search API when it needs more context for recent security issues and API gotchas about those exact libraries. That live context is pasted next to the diff into a prompt for JetBrains' Mellum2, a small reasoning model served over vLLM.

How the Webhook Guards the Gate

Every request GitHub sends is signed. Before any review work happens, the gateway confirms the HMAC signature and ignores anything that isn't a fresh pull_request event:

if not verify_signature(
        settings.github_webhook_secret,
        payload_bytes,
        x_hub_signature_256,
):
    raise HTTPException(status_code=403, detail="Invalid GitHub signature")

if x_github_event != "pull_request":
    return {"status": "ignored"}

if payload.get("action") not in {"opened", "synchronize"}:
    return {"status": "ignored"}

How the Orchestrator Recovers

Mellum2 must answer with a strict JSON list of comments, each tied to a real line. If a thinking model leaves its answer buried in the reasoning trace, the code recovers it; if GitHub rejects a line number, the findings fall back to a visible PR comment:

elif reasoning:
    # Thinking models may leave the JSON array in their trace with content empty.
    review_comments = extract_review_from_reasoning(reasoning)
    if review_comments is None:
        return

# If GitHub rejects the line numbers (422), surface findings as a PR comment.
if isinstance(review_result, dict) and "error" in review_result:
    fallback_msg = " **AI Code Review Found Issues (Line numbers were outside the diff):**\n"
    await post_general_comment(owner, repo, pr_number, fallback_msg, settings.github_token)

The result is a reviewer that cites sources and stays grounded augmenting, not replacing, the human skill of slow, careful reading.