[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-19

[github-actions[bot]]

Summary

• Date of test: 2026-04-19
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: The docs have a clean structure and the Quick Start is well-organized with clear steps and estimated time. However, as a total beginner, I repeatedly hit walls around authentication setup — specifically around COPILOT_GITHUB_TOKEN — and found several jargon terms used without inline definitions that would stop a newcomer cold.

Note: Playwright screenshots were unavailable due to network isolation in the test environment. Analysis was performed via curl + HTML parsing.

---

🔴 Critical Issues Found

1. Authentication is the #1 blocker — and it's offloaded too quickly

In Quick Start Step 2, the authentication secret is mentioned like this:

Set up the required secret — COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)

Then it says "See Authentication for setup instructions" with a link.

Problem: The Authentication page explains this requires a "fine-grained PAT with Copilot Requests permission" — but a brand new user doesn't know what a PAT is, what "fine-grained" means, or why they need two different tokens. This is the most common reason beginners give up on auth-heavy tools. The Quick Start should either:

• Include a 2-line inline summary: "You need a GitHub Personal Access Token. Go to github.com/settings/tokens → Fine-grained → enable Copilot Requests scope."
• Or embed a collapsed <details> block with the minimum steps right on the Quick Start page.

2. githubnext/agentics/ registry path — no explanation

The core install command in Step 2 is:

gh aw add-wizard githubnext/agentics/daily-repo-status

A new user has no idea what githubnext/agentics/ refers to. Is it a GitHub repo? An npm registry? A built-in catalog? There's no explanation of this namespace format anywhere on the Quick Start page. There's a link to "The Agentics collection" on the CLI Commands page, but not on Quick Start.

Recommendation: Add one sentence like: "This installs a pre-built workflow from the Agentics collection, a curated library of ready-to-use workflows."

3. No sidebar navigation visible on the Quick Start page

The left-side navigation panel (typical for Starlight docs) was not visible from the HTML structure. A noob navigating docs expects a sidebar with section structure like "Getting Started → Quick Start → Creating Workflows". Without it (or if it only loads via JS), the user's only navigation is the top bar: Quick Start | Create | Examples | Docs | FAQ | Blog.

This means a beginner who lands on Quick Start has no sense of where it fits in the bigger picture.

---

🟡 Confusing Areas

4. "frontmatter" used without inline definition

Quick Start Step 4 says:

If you have changed the frontmatter (the YAML configuration block between --- markers...)

The parenthetical is helpful — but this appears only in Step 4, which is the optional customization step. The Overview page explains frontmatter well, but the Quick Start assumes you'll know the term. The link to /reference/frontmatter/ is buried in the full navigation, not at the point of first mention.

Recommendation: Add an inline link on first use: "the frontmatter (YAML config block between --- markers)"

5. Multiple engine choices with no recommended path

The Prerequisites list shows three options for Step 1's AI Account:

• GitHub Copilot
• Anthropic Claude
• OpenAI Codex

But there's no "most people start with X" guidance. For a beginner, this is a decision paralysis point. The add-wizard command selects the engine interactively, which is good — but the Quick Start doesn't say that, so the user worries they need to pick before starting.

Recommendation: Add a short callout: "💡 Not sure which to use? Most users start with Copilot — you may already have access via GitHub."

6. gh aw compile in Step 4 is under-explained

Step 4 says:

If you have changed the frontmatter... regenerate the workflow YAML by running: gh aw compile

For a beginner, "regenerate the workflow YAML" is unclear. Why does changing the markdown header require running a command? The overview explains the compilation concept well, but Quick Start doesn't link to it here.

Recommendation: Add a brief aside: "The workflow markdown compiles to a .lock.yml file — if you change the YAML headers (--- section), you must recompile."

7. "Peli's Agent Factory" in the top navigation

The main nav bar shows: Quick Start | Create | Examples | Docs | FAQ | Blog | Peli's Agent Factory

A new user has no context for who "Peli" is or what "Agent Factory" means. It looks like a team-internal link. Either rename it to something more descriptive ("Example Workflows" or "Community Showcase"), or move it inside the Blog section.

8. CLI Commands page is overwhelming for beginners

The CLI Commands page is comprehensive (excellent as a reference!) but leads with every command equally. A beginner looking for "what do I actually need to run?" gets lost in --no-mcp, --no-gitattributes, --stop-after, GHES configuration, GitHub Enterprise, etc.

Recommendation: Add a "Beginner's Cheat Sheet" callout at the very top:

Just getting started? You only need 3 commands:
  gh aw add-wizard <workflow>   # Add a workflow
  gh aw compile                 # Build the workflow YAML
  gh aw run <workflow>          # Trigger a run

---

🟢 What Worked Well

• ✅ "Estimated time: 10 minutes" at the top of Quick Start — sets expectations, reduces anxiety
• ✅ Prerequisite checklist is comprehensive and has links for each item
• ✅ "Most Common Commands" table at the top of CLI Commands is excellent
• ✅ Security first narrative on the home page — the five-layer guardrails explanation builds confidence
• ✅ Video on Quick Start page — a walkthrough video is very helpful for visual learners
• ✅ Step-by-step format with clear section headers in Quick Start
• ✅ gh aw add-wizard interactive flow — the fact that it guides you through everything is great; just needs to be stated more clearly up front
• ✅ Authentication page has a pre-filled PAT link — this is a clever UX touch that reduces friction
• ✅ All critical internal pages return 200 — no broken links found in navigation

---

Recommendations

Quick Wins (High impact, low effort)

1. Add a 2-3 sentence auth summary directly in Quick Start Step 2 — don't make beginners navigate away before they even start
2. Explain the githubnext/agentics/ namespace with one sentence and a link to the Agentics catalog
3. Rename or contextualize "Peli's Agent Factory" in the nav (e.g., "🏭 Workflow Gallery")
4. Add a "Just getting started?" callout at the top of the CLI Commands page with the 3 essential commands
5. Link "frontmatter" to its reference page on first mention in Quick Start

Medium-Term Improvements

6. Add a "Choose your engine" recommendation callout in Prerequisites — suggest Copilot as the default path
7. Add an inline explanation of compilation when Step 4 mentions gh aw compile
8. Ensure sidebar navigation loads without JavaScript (or add a "On this page" / "In this section" fallback for users with JS disabled or slow connections)

Longer-Term

9. Consider a "true beginner" pathway — a separate "5-minute Hello World" that does exactly one thing end-to-end with minimal options, before the full Quick Start
10. Add a glossary tooltip for terms like "frontmatter", "lock file", "agentic workflow", "safe outputs" that appear on intro pages — beginners shouldn't need to navigate away to understand key terms

---

References:

• §24629924341

Generated by Documentation Noob Tester · ● 3.5M · ◷

• expires on Apr 20, 2026, 1:28 PM UTC