Skip to content

fix(workflows): keep non-ASCII text readable in written overlay files - #4148

Open
jawwad-ali wants to merge 1 commit into
github:mainfrom
jawwad-ali:fix/overlay-allow-unicode
Open

fix(workflows): keep non-ASCII text readable in written overlay files#4148
jawwad-ali wants to merge 1 commit into
github:mainfrom
jawwad-ali:fix/overlay-allow-unicode

Conversation

@jawwad-ali

Copy link
Copy Markdown
Contributor

Problem

Both overlay writers in overlays/_commands.py (lines 217 and 270 — the only two yaml.safe_dump calls in the module) omit allow_unicode=True:

staged.write_bytes(yaml.safe_dump(data, sort_keys=False).encode("utf-8"))

So every non-ASCII character in a user's overlay is rewritten as a \uXXXX / \xNN escape inside a double-quoted scalar.

Every other YAML writer in the repo already passes allow_unicode=True:

src/specify_cli/agents.py:163
src/specify_cli/bundler/lib/yamlio.py:91
src/specify_cli/extensions/_commands.py:530, 568
src/specify_cli/extensions/__init__.py:4800
src/specify_cli/integrations/base.py:69, 1429, 1440

Why it matters

Overlay files are explicitly hand-authored and hand-editeddocs/reference/workflows.md documents the file format and tells users to write these files. overlay add, enable, disable and set-priority all round-trip the file through safe_dump, so merely toggling an overlay mangles a UTF-8 file the user wrote by hand.

Reproduction on current main

Source overlay written in UTF-8 with message: "Revisar el plan — ¿aprobar? 日本語", then specify workflow overlay add overlay.yml:

id: revisar
extends: wf
priority: 10
edits:
- replace: a
  step:
    id: a
    type: gate
    message: "Revisar el plan — \xBFaprobar? 日本語"
    options:
    - approve

The value still parses back identically, so this is not corruption — it is the loss of a documented, hand-edited file's legibility, the same thing allow_unicode=True is already there to prevent everywhere else.

Verification

  • Fail-before / pass-after: 2 new-vs-baseline failures with the source reverted → 25 passed with the fix.
  • One test per writer: overlay add (the install path) and overlay set-priority (the _update_overlay_field round-trip path).
  • Each asserts both that the text survives readable and that it still round-trips to the identical value through yaml.safe_load.
  • Scoped regression over tests/workflows: no new failures vs a clean-main baseline (10 pre-existing, Windows symlink-privilege).
  • uvx ruff@0.15.0 check src tests → clean

No breaking change. allow_unicode only affects how characters are encoded in the output; the parsed value is byte-identical either way (both new tests assert this explicitly), and files are already written as UTF-8 via .encode("utf-8"). Pure-ASCII overlays — the overwhelming majority — produce identical bytes.


Written with assistance from Claude Code. Bug found, reproduced, and verified by me on current main.

Both overlay writers in `overlays/_commands.py` called
`yaml.safe_dump(data, sort_keys=False)` without `allow_unicode=True`, so
every non-ASCII character was rewritten as a `\uXXXX` / `\xNN` escape inside
a double-quoted scalar. Every other YAML writer in the repo already passes
`allow_unicode=True` (agents.py, bundler/lib/yamlio.py, extensions,
integrations/base.py, ...).

Overlay files are explicitly hand-authored and hand-edited -- the format is
documented in docs/reference/workflows.md and users are told to write these
files. `overlay add`, `enable`, `disable` and `set-priority` all round-trip
the file through `safe_dump`, so merely toggling an overlay mangled a UTF-8
file the user wrote by hand:

    message: "Revisar el plan — \xBFaprobar? 日本語"

The value still parses back identically, so this is not corruption -- it is
the loss of a documented, hand-edited file's legibility.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jawwad-ali
jawwad-ali requested a review from mnriem as a code owner August 15, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant