Skip to content

jiachen: add MCode Review, PR review that has to prove its findings - #15

Open
yujiachen-y wants to merge 2 commits into
MiniMax-AI:mainfrom
yujiachen-y:jiachen/pr-review
Open

yujiachen-y wants to merge 2 commits into
MiniMax-AI:mainfrom
yujiachen-y:jiachen/pr-review

Conversation

@yujiachen-y

@yujiachen-y yujiachen-y commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What this adds or changes

Adds MCode Review to stacks/jiachen (stack version 0.1.1 → 0.2.0): the PR reviewer our team runs on its own merge requests, as a public edition that works with any GitHub or GitLab repository.

MCode Review overview

One rule set, used in two places, plus a loop that tunes it:

  • In your agent, no setup. pr-review reviews a branch or a PR (GitHub pull request or GitLab merge request) with a severity scale, an evidence ladder and one checklist per review direction; pr-description drafts the PR description. Your own session's model, nothing posted.
  • In CI, on every PR. One workflow file for GitHub Actions (ci/github/mcode-review.yml), or a dispatch job plus a child pipeline for GitLab CI, pinned to a commit of this repository. Tiered review directions in parallel, a verifier pass that tries to refute every P0/P1, a verdict decided by a script, one thread per finding or question, re-review of only the files a push changed, an optional author quiz. Runs MiniMax-M3.1-Flash through the MiniMax Code CLI by default.
  • It tunes itself. pr-review-tuning is the weekly loop: collect what the bot did on last week's PRs, measure it, propose at most three rule changes, backtest them on past PRs with known bugs (writes blocked, token budget agreed first), have an independent pass recompute the numbers, and change nothing until a person approves.

Projects keep their own rules in .mcode-review.json (risk paths and their questions) and .mcode-review.md (known failure patterns), which CI reads from the merge base so a PR cannot rewrite the rules that review it. Output is English by default; MCODE_REVIEW_LANGUAGE=zh switches everything the bot posts to Simplified Chinese.

New in stacks/jiachen:

Path What it is
skills/pr-review, pr-description, pr-review-ci, pr-review-tuning, pr-quiz The five skills
ci/ The pipeline: review.mjs (prepare, lens, quiz, publish, grade), GitHub and GitLab clients, prompts, schemas, templates, tests. Node 22, no dependencies
skills/pr-review-tuning/scripts/ collect.mjs, metrics.mjs, backtest.mjs
docs/pr-review.md Full reference: how it decides, both CI setups, every setting, the security model
docs/pr-review-overview.png The diagram above, with its HTML source

Also: the stack README now leads with MCode Review, WORKFLOW.md covers it, manifests and both marketplace entries carry the new description, a row in stacks/README.md and lines in both top-level READMEs. .github/workflows/validate.yml now also runs the stack's tests.

Check output

$ CHECK_BASE=origin/main MINIMAX_CODE_DIR=../minimax-code node scripts/check.mjs
validate-plugin: stacks/jiachen passed the public Plugin V1 checks.
validate-plugin: stacks/req-tpm passed the public Plugin V1 checks.
validate-plugin: stacks/ronny passed the public Plugin V1 checks.
validate-plugin: stacks/youfang passed the public Plugin V1 checks.
validate-plugin: templates/stack-template passed the public Plugin V1 checks.
check: 5 stack(s) passed: stacks/jiachen, stacks/req-tpm, stacks/ronny, stacks/youfang, templates/stack-template
  • node --test ci/test/*.test.mjs skills/pr-review-tuning/scripts/test/*.test.mjs in stacks/jiachen: 230/230 on Node 22 and on Node 26. They include end-to-end rounds against a fake GitLab server and a fake GitHub REST + GraphQL server.
  • actionlint 1.7.12 (with shellcheck) on ci/github/mcode-review.yml: no findings.
  • Relative links in the changed Markdown: 0 broken.
  • A scan of every changed file for internal hostnames, project and people's names, model names other than MiniMax's, and secret-shaped strings: none.

Real usage record

MCode Review has reviewed our team's own merge requests in GitLab CI since late October, in shadow mode (comments only); pr-review-tuning is how we tune it every week. For this public edition I replaced our internal rule packs and infrastructure with the per-repository files above, then backtested it with MiniMax-M3.1-Flash on six merged PRs of one of our repositories, with every write to GitLab blocked:

  1. Baseline. Of the first two cases, one review direction ran its full 25 minutes and returned nothing (69 model steps, five time reminders ignored); the other answered twice in a shape the output schema rejected. Cause: mcode exec --output-schema validates the final answer but never shows the schema to the model, and the prompt gave no time budget.
  2. Fix. The prompts now carry the schema and the time budget. A direction that still times out is asked once, in its own session, for the findings it has already checked; those are posted, and the round counts as unfinished (it closes no thread and cannot pass).
  3. Rerun of all six. Five finished with valid answers and no time-outs; the sixth was lost to a 529 from the model service. No blocking finding on the PR that merged clean. Of the four bugs that a later fix PR traced back to the reviewed PR, one was raised, as a question for a person that matches the later fix; two of the other three were also missed by our internal reviewer. A case cost 1.6M to 8.4M tokens.
  4. GitHub. The new GitHub client ran read-only against a real public PR (Apply issue triage labels directly cli/cli#15001): the PR, its merge base, comments and review threads came back through REST and GraphQL in the shapes the core code expects, and publish rendered the summary as a dry run. Writing comments, threads and resolutions is covered by the fake-server tests, not yet by a live repository.

If this skill were deleted, where would the agent go wrong?

Asked to review a PR, the agent comments on style and possible risks with no bar for evidence and no notion of what should block. In CI the verdict moves with the model's wording, every push re-raises questions already answered, and nobody can tell whether the reviewer is getting better or worse. MCode Review sets the bar (evidence ladder, a verifier that tries to refute blockers, a verdict decided by a script), keeps its memory in threads people resolve, and measures itself before anyone changes its rules.

Checklist

  • No keys, tokens, internal hostnames or private links
  • Third-party content is credited with its license (none included)
  • I agree to release my own work here under the repository's MIT License (third-party material keeps its own license, noted in the stack's NOTICE.md)

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Adds the PR review skills (pr-review, pr-description, pr-review-ci,
pr-review-tuning, pr-quiz) and the review pipeline for GitHub Actions and
GitLab CI to stacks/jiachen, with a reference doc and an overview diagram.
Projects keep their own rules in .mcode-review.json and .mcode-review.md,
read from the merge base in CI. Bumps the stack to 0.2.0, updates the
marketplace entries and READMEs, and runs the stack's tests in validate.
The model in the review and quiz jobs has a shell, so it can read the model
key its job holds. The security model said secrets are kept from the model;
it now says the write token is, names who can put text in front of the model
(on a public repository, anyone who can reply in the bot's threads), and
notes that redaction only catches exact copies and that the runner does not
block network access. It advises a model key used only by the reviewer, with
a spend limit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant