Repository navigation
jiachen: add MCode Review, PR review that has to prove its findings - #15
Open
yujiachen-y wants to merge 2 commits into
Open
yujiachen-y wants to merge 2 commits into
yujiachen-y wants to merge 2 commits into
Conversation
Adds the PR review skills (pr-review, pr-description, pr-review-ci, pr-review-tuning, pr-quiz) and the review pipeline for GitHub Actions and GitLab CI to stacks/jiachen, with a reference doc and an overview diagram. Projects keep their own rules in .mcode-review.json and .mcode-review.md, read from the merge base in CI. Bumps the stack to 0.2.0, updates the marketplace entries and READMEs, and runs the stack's tests in validate.
The model in the review and quiz jobs has a shell, so it can read the model key its job holds. The security model said secrets are kept from the model; it now says the write token is, names who can put text in front of the model (on a public repository, anyone who can reply in the bot's threads), and notes that redaction only catches exact copies and that the runner does not block network access. It advises a model key used only by the reviewer, with a spend limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds or changes
Adds MCode Review to
stacks/jiachen(stack version 0.1.1 → 0.2.0): the PR reviewer our team runs on its own merge requests, as a public edition that works with any GitHub or GitLab repository.One rule set, used in two places, plus a loop that tunes it:
pr-reviewreviews a branch or a PR (GitHub pull request or GitLab merge request) with a severity scale, an evidence ladder and one checklist per review direction;pr-descriptiondrafts the PR description. Your own session's model, nothing posted.ci/github/mcode-review.yml), or a dispatch job plus a child pipeline for GitLab CI, pinned to a commit of this repository. Tiered review directions in parallel, a verifier pass that tries to refute every P0/P1, a verdict decided by a script, one thread per finding or question, re-review of only the files a push changed, an optional author quiz. RunsMiniMax-M3.1-Flashthrough the MiniMax Code CLI by default.pr-review-tuningis the weekly loop: collect what the bot did on last week's PRs, measure it, propose at most three rule changes, backtest them on past PRs with known bugs (writes blocked, token budget agreed first), have an independent pass recompute the numbers, and change nothing until a person approves.Projects keep their own rules in
.mcode-review.json(risk paths and their questions) and.mcode-review.md(known failure patterns), which CI reads from the merge base so a PR cannot rewrite the rules that review it. Output is English by default;MCODE_REVIEW_LANGUAGE=zhswitches everything the bot posts to Simplified Chinese.New in
stacks/jiachen:skills/pr-review,pr-description,pr-review-ci,pr-review-tuning,pr-quizci/review.mjs(prepare, lens, quiz, publish, grade), GitHub and GitLab clients, prompts, schemas, templates, tests. Node 22, no dependenciesskills/pr-review-tuning/scripts/collect.mjs,metrics.mjs,backtest.mjsdocs/pr-review.mddocs/pr-review-overview.pngAlso: the stack README now leads with MCode Review,
WORKFLOW.mdcovers it, manifests and both marketplace entries carry the new description, a row instacks/README.mdand lines in both top-level READMEs..github/workflows/validate.ymlnow also runs the stack's tests.Check output
node --test ci/test/*.test.mjs skills/pr-review-tuning/scripts/test/*.test.mjsinstacks/jiachen: 230/230 on Node 22 and on Node 26. They include end-to-end rounds against a fake GitLab server and a fake GitHub REST + GraphQL server.actionlint1.7.12 (with shellcheck) onci/github/mcode-review.yml: no findings.Real usage record
MCode Review has reviewed our team's own merge requests in GitLab CI since late October, in shadow mode (comments only);
pr-review-tuningis how we tune it every week. For this public edition I replaced our internal rule packs and infrastructure with the per-repository files above, then backtested it withMiniMax-M3.1-Flashon six merged PRs of one of our repositories, with every write to GitLab blocked:mcode exec --output-schemavalidates the final answer but never shows the schema to the model, and the prompt gave no time budget.If this skill were deleted, where would the agent go wrong?
Asked to review a PR, the agent comments on style and possible risks with no bar for evidence and no notion of what should block. In CI the verdict moves with the model's wording, every push re-raises questions already answered, and nobody can tell whether the reviewer is getting better or worse. MCode Review sets the bar (evidence ladder, a verifier that tries to refute blockers, a verdict decided by a script), keeps its memory in threads people resolve, and measures itself before anyone changes its rules.
Checklist
NOTICE.md)Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.