docs: prepare Fern docs workflow

[andreatgretel]

📋 Summary

This PR prepares DataDesigner's Fern docs workflow for local validation, preview, and publishing while keeping the existing MkDocs docs path active. The goal is to run both docs stacks for a few releases, validate Fern with real releases/dev notes, then cut over once the workflow is mature.

🔗 Related Issue

N/A

🔄 Changes

Added

• Fern CI/publish workflow for release/manual publishing via DOCS_FERN_TOKEN: build-fern-docs.yml.
• Fern dev-notes publish workflow that only runs when Dev Notes change: publish-fern-devnotes.yml.
• Fern preview publishing alongside the existing MkDocs preview, with hosted preview/deploy steps limited to same-repo PRs.
• Fern-local docs generation targets for API reference and notebooks, including source-only notebook fallback and executed-notebook support.
• Fern v0.5.9, v0.5.8, latest, and Older versions entries, plus redirects for legacy MkDocs archives.
• VLM long-document Dev Note and recipe cards in Fern.
• Transition notes in README.md, CONTRIBUTING.md, and fern/README.md.

Changed

• Generated Fern notebook artifacts are now gitignored and regenerated locally/CI from docs/notebook_source/*.py.
• Notebook build caching now matches the MkDocs behavior more closely for docs publishing.
• Dev Notes sidebar is trimmed on latest, with older posts still discoverable through the Dev Notes index.
• Older Fern versions route now uses /nemo/datadesigner/older-versions, with the previous doubled route redirected.
• Fern docs skill guidance now reflects the current versioning, Dev Notes, notebook, and API-reference workflow.

Fixed

• Generated MDX quoting for MCP provider docstrings.
• Dev Notes author metadata rendering.
• Fern preview comment URL formatting.
• Recipe cards and VLM recipe pages missing from the Fern port.

🔍 Attention Areas

• We are temporarily relying on pinned py2fern==0.1.6 so OSS contributors can run the local Fern API reference path without a Fern login. Andrew Schilling said Fern has promised a local-build solution without login; we should revisit this once available.
• MkDocs should remain published in parallel for the next few releases while Fern output, CI, release publishing, and contributor workflows settle.
• Hosted Fern previews and publishing use org secret DOCS_FERN_TOKEN; preview publishing is gated to same-repo PRs so fork PRs do not receive secrets.

🧪 Testing

• .venv/bin/ruff check --fix .
• .venv/bin/ruff format .
• make check-fern-docs
• make serve-fern-docs-locally with browser spot-checks for versions, Dev Notes, notebooks, recipes, API reference, and older-version routing
• npx -y fern-api@4.106.0 generate --docs
• PR CI is passing
• Docs preview workflow is passing
• make test passes - not run locally; CI test matrix is passing
• Unit tests added/updated - N/A, docs workflow/content changes
• E2E tests added/updated - N/A, docs workflow/content changes

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO) - DCO Assistant is passing, but branch commits do not include Signed-off-by trailers
• Architecture docs updated - N/A; docs workflow notes were added instead

---

Description updated with AI

[github-actions]

MkDocs preview: https://66bdd336.dd-docs-preview.pages.dev

Fern preview: https://nvidia-preview-pr-622.docs.buildwithfern.com/nemo/datadesigner

Notebook tutorials are placeholder-only in previews.

[github-actions]

PR #622 Review — docs: prepare Fern docs workflow

Summary

This PR lands the scaffolding needed to run the Fern docs stack alongside MkDocs for the next few releases: a Fern CI/publish workflow, a Dev-Notes-only publish workflow, Fern preview publishing in the existing docs-preview workflow, local-build Make targets (prepare-fern-docs, check-fern-docs, serve-fern-docs-locally), and the content updates (v0.5.9 version entry, VLM long-doc recipes + dev note, older-versions redirect page). Generated Fern notebook artifacts move from committed snapshots to gitignored outputs regenerated locally/in CI. The only non-docs production code change is a two-line cosmetic tweak to packages/data-designer-config/src/data_designer/config/mcp.py (docstring quotes) and a py2fern==0.1.6 pin in pyproject.toml. Scope is well-contained and matches the PR description.

55 files changed; ~7k additions are dominated by new MDX recipe pages and regenerated notebook JSON/TS. Non-docs code is small.

Findings

Workflows

• .github/workflows/build-fern-docs.yml (new) — Clean structure: reusable build-notebooks.yml produces the notebooks artifact, publish job downloads it and runs make check-fern-docs → fern generate --docs. Top-level permissions: {} plus narrow per-job perms is correct. The explicit FERN_TOKEN empty-guard before npx fern-api is good practice. Version is resolved from fern.config.json with jq so the CLI stays pinned — nice.
• .github/workflows/publish-fern-devnotes.yml (new) — Reuses notebook artifacts from the last successful build-fern-docs.yml or build-docs.yml run. Two things to verify:
  1. Fallback target build-docs.yml — make sure a workflow by that exact filename exists in main. If not, the fallback silently contributes nothing and the step errors only after exhausting both, which is still safe but the loop would be redundant. Consider simplifying to just build-fern-docs.yml, or confirm build-docs.yml is a real workflow on main.
  2. gh run download ... --name notebooks exits non-zero when the artifact is missing/expired (artifacts default to 90 days). A freshly-cut release window that relies on a 90+ day old build would fail. Not blocking, but worth flagging so someone can trigger build-fern-docs.yml manually if this ever happens.
• .github/workflows/docs-preview.yml — Preview URL is hand-constructed: https://nvidia-preview-pr-${PR}.docs.buildwithfern.com/nemo/datadesigner. If Fern ever changes its preview URL scheme this silently goes stale. Not a blocker since you can confirm from a real preview run, but a future-proofing option would be to capture the URL from fern generate --docs --preview stdout. The same-repo gate (github.event.pull_request.head.repo.full_name == github.repository) is applied consistently to Fern deploy, Cloudflare deploy, and both comment steps — good.
• Concurrency group docs-preview-${{ github.event.pull_request.number }} with cancel-in-progress: true is the right pattern for PR-push churn.
• build-notebooks.yml adds an explicit uv sync --all-packages --group notebooks --group docs step; matches the Makefile's shift to .venv/bin/* invocations. Consistent.

Makefile

• Replacing uv run --python $(DOCS_PYTHON) with .venv/bin/<tool> paths is a meaningful behavioral shift: the docs targets now require make install-dev-notebooks (or equivalent uv sync --group docs --group notebooks) to have been run first. The build_notebooks_cached.sh precondition check ([ ! -x "$DOCS_JUPYTEXT" ]) catches this for the caching path, but generate-fern-notebooks, generate-fern-api-reference, convert-execute-notebooks (non-cache branch), and generate-colab-notebooks do not have an equivalent guard — they'll fail with a generic "No such file" if the venv is missing or stale. Minor: consider adding a one-line check or at least documenting the precondition in the help text.
• convert-execute-notebooks (non-cache branch) now exit 1s on any notebook failure, up from the previous "warn and continue." The SKILL.md and CONTRIBUTING.md note this, but contributors missing an API key for a single image notebook will now see the whole target fail. Intended behavior per the changelog, but worth flagging for anyone reading the diff looking for regressions.
• rm -rf docs/notebooks inside convert-execute-notebooks and inside build_notebooks_cached.sh are both destructive-by-design — reasonable for deterministic outputs, but if a contributor was hand-editing a notebook under docs/notebooks/ that work will be blown away. Not a real concern given the source-of-truth is docs/notebook_source/*.py.
• FERN ?= npx -y fern-api@$(FERN_VERSION) with FERN_VERSION resolved via jq from fern/fern.config.json — nicely keeps CI and local in sync.

Notebook pipeline

• fern/scripts/ipynb-to-fern-json.py — is_colab_injected_cell generalization is good: primary check is the nemo_colab_inject metadata flag set by generate_colab_notebooks.py, with regex heuristics as a fallback for notebooks that weren't tagged (e.g., older Colab snapshots). The regex patterns are narrow enough they shouldn't false-positive on legitimate cells. Handling source=None in _join_source is a reasonable defensive change.
• docs/scripts/generate_colab_notebooks.py — mark_colab_injected helper is clean and localized. Good.
• The one purely-cosmetic reformat of _MD assignment and the collapsed f-string in write_ts_export are noise in the diff but harmless.

Fern content & routing

• fern/docs.yml — The ~34 version redirects to nvidia-nemo.github.io/DataDesigner/<ver>/ all use permanent: false (302), which is correct during migration. The :path* parameter pass-through is idiomatic. The self-redirect /older-versions/older-versions → /older-versions defends against Fern's slug doubling when a section's skip-slug: true doesn't fully suppress the nav slug — sensible guard.
• fern/versions/latest.yml and v0.5.9.yml — Both reuse ./v0.5.8/pages/ for almost everything, with ./v0.5.9/pages/ only for recipe cards and VLM content. This matches the SKILL.md's "reuse shared pages, diverge only when needed" story. Watch for drift: any future edit to a shared page will land in both latest and v0.5.9, which is the intended behavior but worth calling out in contributor docs (SKILL.md does address this).
• Older Dev Notes nested section in both latest.yml and v0.5.8.yml — matches the description goal of trimming the sidebar on latest while keeping discoverability via the Dev Notes index.
• VLM recipe MDX pages — 8 new recipe pages + an overview card page. I did not line-by-line check each (4.7k lines of MDX content) but spot checks show consistent frontmatter, no leftover position:/authors: keys that would trip Fern's runtime, and cross-links use absolute paths as required.

Non-docs production code

• packages/data-designer-config/src/data_designer/config/mcp.py — Only change is Literal["sse", "streamable_http"] → Literal['sse', 'streamable_http'] (and the stdio equivalent) in the docstrings. The stated reason is MDX-quoting issues from py2fern. This is benign at runtime (the typed annotation on the class field is unchanged) and scoped to two docstrings. One small consistency nit: grep the rest of data_designer.config for other Literal["..."] docstring patterns and either normalize all of them or leave a short # py2fern: single-quote Literal args comment so a future edit doesn't silently revert this. Non-blocking.

pyproject / dependencies

• py2fern==0.1.6 pinned exactly in the docs group. Pin is appropriate for a known-temporary dependency, and the PR description commits to revisiting this once Fern ships a login-free local build. Lock updates in uv.lock are consistent.

Risks

1. Fern preview URL format is hard-coded; relies on Fern's buildwithfern.com preview scheme staying stable. Low risk during migration, worth a TODO to capture from CLI output later.
2. Devnotes publish workflow depends on reusable-workflow artifacts being available within the 90-day retention window. Low risk under normal release cadence.
3. Makefile targets require make install-dev-notebooks; a contributor running make check-fern-docs on a clean clone without the notebooks group installed will hit opaque failures. Consider a minimal precondition guard or doc update.
4. No new tests — correct, since changes are docs workflow/content only.
5. No structural impact analysis file found at /tmp/structural-impact-622.md; section omitted.

Verdict

Approve (with optional follow-ups). This is a well-scoped docs-infrastructure PR. Workflow permissions are minimal, fork secrets are properly gated, the local-build Make targets are consistent with what CI runs, and the py2fern dependency is explicitly acknowledged as a temporary pin. The one non-docs production change (mcp.py docstring quotes) is cosmetic and limited. The optional follow-ups above are all quality-of-life improvements, not blockers.

[greptile-apps]

Greptile Summary

This PR introduces the Fern docs publishing pipeline alongside the existing MkDocs path, adding CI workflows for full release publishing, a lightweight devnotes-only publish, and PR preview support. Generated Fern artifacts (notebook JSON/TS and API reference) are now gitignored and regenerated in CI or locally via new Make targets.

• New CI workflows (build-fern-docs.yml, publish-fern-devnotes.yml) handle Fern publishing; the existing docs-preview.yml is extended to build and post a Fern preview link alongside the Cloudflare MkDocs preview, gated behind same-repo checks to protect secrets from fork PRs.
• Makefile refactor migrates docs toolchain invocations from uv run --python X to .venv/bin/<tool> paths, adds check-fern-docs, serve-fern-docs-locally, and related targets, and updates generate-fern-notebooks to fall back to source-only notebook conversion when executed notebooks aren't present.
• ipynb-to-fern-json.py is updated to identify Colab-injected cells by metadata tag (nemo_colab_inject) in addition to the previous regex heuristics, giving more reliable filtering of Colab-only setup cells from rendered docs.

Confidence Score: 4/5

Safe to merge after addressing the step-ordering issue in the preview workflow; all other changes are well-structured docs and tooling additions.

The Fern preview step in docs-preview.yml is ordered before the Cloudflare deploy and the PR comment steps, with no continue-on-error guard. A Fern failure silently prevents the MkDocs preview from being deployed and prevents the PR comment from being posted in that run. Everything else — new CI workflows, Makefile refactor, notebook conversion logic, and Fern versioning structure — looks correct and the PR description confirms CI is passing.

.github/workflows/docs-preview.yml — step ordering around the Fern preview deployment.

Important Files Changed

Filename	Overview
.github/workflows/docs-preview.yml	Adds Fern preview deployment, concurrency cancellation, and fork-PR guards; check-fern-docs relies implicitly on the stub-notebook step running first since notebooks group is not installed.
.github/workflows/build-fern-docs.yml	New release/manual workflow that builds notebooks, checks Fern docs, and publishes; uses pinned action SHAs and validates FERN_TOKEN before publishing.
.github/workflows/publish-fern-devnotes.yml	Lightweight devnotes-only publish trigger; hard-fails if no prior successful docs build has notebook artifacts, which blocks initial bootstrap until build-fern-docs.yml runs first.
Makefile	Migrates docs tooling from uv run to .venv/bin/* paths, adds Fern-specific targets (check-fern-docs, serve-fern-docs-locally, generate-fern-api-reference), and changes convert-execute-notebooks to exit 1 on any notebook failure instead of printing a warning.
fern/scripts/ipynb-to-fern-json.py	Expands Colab cell detection from badge-only regex to metadata tag + multiple content patterns; renames is_colab_badge_cell to is_colab_injected_cell for accuracy.
docs/scripts/build_notebooks_cached.sh	Parameterizes jupytext path via DOCS_JUPYTEXT env var, adds pre-flight check for missing executable, and cleans the output dir before rebuilding.
fern/docs.yml	Adds v0.5.9 and 'Older versions' entries, updates display name to 'Latest · v0.5.9', and adds redirect for doubled older-versions slug and historical MkDocs archive routes.
fern/versions/latest.yml	Rolling 'latest' version mixes v0.5.8 pages for core content with v0.5.9 pages for new VLM recipes and latest/pages for dev notes; intentional design for incremental rollout.
pyproject.toml	Adds pinned py2fern==0.1.6 to the docs dependency group so API reference generation works without a Fern login.

Sequence Diagram

sequenceDiagram
    participant Push as Git Push / Release
    participant BN as build-notebooks.yml
    participant BFD as build-fern-docs.yml
    participant PFD as publish-fern-devnotes.yml
    participant DP as docs-preview.yml
    participant Fern as Fern Docs (buildwithfern)
    participant CF as Cloudflare Pages

    Note over BFD,PFD: Release / workflow_dispatch
    Push->>BFD: release:published or manual
    BFD->>BN: reuse (build-notebooks)
    BN-->>BFD: notebooks artifact
    BFD->>BFD: make check-fern-docs
    BFD->>Fern: fern generate --docs

    Note over PFD: push to main (devnotes paths)
    Push->>PFD: path filter matches
    PFD->>BFD: gh run download notebooks artifact
    PFD->>PFD: make check-fern-docs
    PFD->>Fern: fern generate --docs

    Note over DP: PR opened / updated
    Push->>DP: pull_request event
    DP->>DP: uv sync --group docs
    DP->>DP: Create notebook stubs
    DP->>DP: mkdocs build
    DP->>DP: make check-fern-docs
    alt same-repo PR
        DP->>Fern: fern generate --preview (pr-N)
        DP->>CF: wrangler pages deploy
        DP->>DP: Post PR comment (MkDocs + Fern URLs)
    else fork PR
        DP->>DP: notice: skipping hosted previews
    end

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
.github/workflows/docs-preview.yml:93-106
**Fern failure cascades to block MkDocs deploy and PR comment**

`Deploy Fern preview` sits before `Deploy to Cloudflare Pages` with no `continue-on-error` guard. GitHub Actions skips all subsequent steps when any step fails (the implicit `success()` condition applies even to steps with explicit event-based `if:` guards). A Fern API error, timeout, or token issue will therefore prevent the MkDocs site from being deployed *and* prevent the PR comment from being posted — silently dropping the MkDocs preview URL from the PR.

Adding `continue-on-error: true` to the Fern step lets the Cloudflare deploy and comment steps always run for same-repo PRs regardless of Fern's outcome.

_{Reviews (1): Last reviewed commit: "docs: clean older versions route" | Re-trigger Greptile}

[greptile-apps]

Fern failure cascades to block MkDocs deploy and PR comment

Deploy Fern preview sits before Deploy to Cloudflare Pages with no continue-on-error guard. GitHub Actions skips all subsequent steps when any step fails (the implicit success() condition applies even to steps with explicit event-based if: guards). A Fern API error, timeout, or token issue will therefore prevent the MkDocs site from being deployed and prevent the PR comment from being posted — silently dropping the MkDocs preview URL from the PR.

Adding continue-on-error: true to the Fern step lets the Cloudflare deploy and comment steps always run for same-repo PRs regardless of Fern's outcome.

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/docs-preview.yml
Line: 93-106

Comment:
**Fern failure cascades to block MkDocs deploy and PR comment**

`Deploy Fern preview` sits before `Deploy to Cloudflare Pages` with no `continue-on-error` guard. GitHub Actions skips all subsequent steps when any step fails (the implicit `success()` condition applies even to steps with explicit event-based `if:` guards). A Fern API error, timeout, or token issue will therefore prevent the MkDocs site from being deployed *and* prevent the PR comment from being posted — silently dropping the MkDocs preview URL from the PR.

Adding `continue-on-error: true` to the Fern step lets the Cloudflare deploy and comment steps always run for same-repo PRs regardless of Fern's outcome.

How can I resolve this? If you propose a fix, please make it concise.