diff --git a/.github/justfile b/.github/justfile index 12f79a8..2ea79ef 100644 --- a/.github/justfile +++ b/.github/justfile @@ -1,7 +1,4 @@ -# CI job bodies: each workflow job runs exactly one `gha::` recipe, so -# `just ci` is exactly CI and the workflow files carry no logic beyond -# environment setup (checkout, toolchains, caches). Step-level rationale -# lives on the root-justfile recipes these compose. +# CI job bodies; recipe-specific rationale lives in the root justfile. set working-directory := '..' @@ -24,9 +21,7 @@ _step recipe: just {{recipe}} fi -# Like _step, but failures are tolerated: the CI-side replacement for the -# workflow-level `continue-on-error` the collapsed steps used to carry. A -# failure surfaces as a ::warning annotation, never as a job failure. +# Like _step, but report failures as warnings rather than failing the job. [private] _step-tolerated recipe: #!/usr/bin/env bash @@ -43,16 +38,9 @@ _step-tolerated recipe: just {{recipe}} || echo "just {{recipe}}: FAILED (tolerated in CI; see the recipe's comment)" fi -# The required per-push/PR matrix job (ci.yml `core`). -# `version-guard-local` runs first, unconditionally — label-free/event-free -# tree checks (lockstep agreement, monotonicity, the protocol byte-identity -# tear check) that need no PR context, so a direct push to main is covered -# too (the #232 gap: `version-guard-pr` no-ops without PR_NUMBER, and #232 -# only got caught because it happened to go through a `pull_request` run). -# `version-guard-pr` runs right after and costs seconds: a versioning -# mistake is cheap to hear about before a 40-minute matrix, and the recipe -# no-ops outside pull_request runs (no PR_NUMBER), so pushes and `just ci` -# are unaffected. ci.yml supplies PR_NUMBER / PR_BASE_SHA / GH_TOKEN. +# Context-free checks precede PR checks (no-op without PR_NUMBER). +# ci.yml supplies PR_NUMBER / PR_BASE_SHA / GH_TOKEN. +# Required core matrix. core: @just gha::_step version-guard-local @just gha::_step version-guard-pr @@ -76,9 +64,8 @@ core: @just gha::_step conformance @just gha::_step sched-seeds -# The post-merge browser job (ci.yml `browser`; gates the prerelease). -# The worker / shared-worker realm rows (issue #129) reuse each engine's -# expectation verbatim — a delta is a realm leak and fails naming the realm. +# Page and worker realms share per-engine expectations; WebKit is best-effort. +# Post-merge browser job; gates prerelease artifacts, not PR mergeability. browser: @just gha::_step shim @just gha::_step corpus @@ -99,9 +86,7 @@ browser: @just gha::_step-tolerated "browsers-install-webkit --with-deps" @just gha::_step-tolerated "browser-lane webkit" -# Never gates: the drivers themselves exit 0 on deviations, 2 on -# infrastructure failure. -# The findings-only canary lanes (canary.yml, x64 leg). +# Findings-only x64 canaries: deviations exit 0; infrastructure failures exit 2. canary: @just gha::_step shim @just gha::_step corpus @@ -109,8 +94,7 @@ canary: @just gha::_step "shell-lane jsc-trunk --json jsc-results.json" @just gha::_step "deno-canary --json deno-canary-results.json" -# JSC trunk has no arm64 channel — the two lanes with arm64 coverage. -# The findings-only canary lanes (canary.yml, arm64 leg). +# Findings-only arm64 canaries; JSC has no arm64 channel. canary-arm: @just gha::_step shim @just gha::_step corpus diff --git a/AGENTS.md b/AGENTS.md index 1082264..17b861c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,198 +1,129 @@ -# polyengine — development protocol - -Instructions for agents (and context for humans) working in this repo. The -repo was built by a multi-agent workflow and its discipline is part of the -project: unusually dense objective gates are what make delegated -implementation safe. +# polyengine development protocol ## Authorities -- Semantic tie-breaker for runtime behavior: the Component Model spec + - `design/mvp/canonical-abi/definitions.py` (in the - `third_party/component-model` submodule), with wasmtime as corroborating - evidence — never the other way around. See - [docs/architecture.md](docs/architecture.md) §1 for the parity policy. -- Interface contracts between workstreams live in `contracts/` (plan format, - descriptor IR, intrinsics, digest, embedder API). **Contract changes are - versioned events made only by the orchestrator**; implementation tracks - report contract friction, they never edit around it. -- Design and decisions: [docs/architecture.md](docs/architecture.md). - Consumer track: [docs/consumers.md](docs/consumers.md). Upstream links: - [docs/references.md](docs/references.md). - -## Gates (exact commands) - -The justfile is the command surface: recipe bodies are the exact commands, -and each CI job runs exactly one `gha::` recipe (`just ci` = exactly CI; -`.github/justfile` holds the job bodies). Run the recipes your change can -affect; the full pass before commit is: +- Runtime semantics: the pinned Component Model spec and + `third_party/component-model/design/mvp/canonical-abi/definitions.py`. + Wasmtime is corroborating evidence, not the tie-breaker. The single named + corpus exception is defined in + [architecture §1](docs/architecture.md#1-goals). +- Interfaces: `contracts/`. Semantic contract changes are versioned events owned + by the orchestrator; implementation tracks report conflicts rather than + changing the contract to fit their code. +- Design: [architecture](docs/architecture.md). Consumer requirements: + [consumers](docs/consumers.md). Upstream sources: + [references](docs/references.md). + +## Gates + +The justfile is the command surface. `just ci` runs the CI recipes in +`.github/justfile`; each CI job invokes one `gha::` recipe. `just gates` also +includes local consumer smokes and is the full pre-commit gate: ```sh -just gates # everything below, in this order; see the justfile (or - # `just --list`) for the recipe list and per-gate one-liners +just gates ``` -Conformance discipline: the harness fails loudly on unexpected failures *and* -on stale xfails; per-browser deltas live in `harness/browser/expectations/` -with stale-delta detection. Never absorb a regression into an xfail/overlay -without a named class and a tracking issue. +Use `just --list` for focused gates. Runtime formatting is checked by +`just fmt-check`. Run gates with non-interactive stdin, as in CI; a terminal +session may need `just gates < /dev/null` for WASI terminal-detection tests. + +Conformance gates reject unexpected failures and stale expected failures. +Browser deltas live in `harness/browser/expectations/`. Never absorb a +regression into an xfail or overlay without a named class and tracking issue. +Passing with exclusions is not full spec conformance. ## Multi-agent protocol -Work is parallelized across model-pinned subagents defined in the operator's -**global** opencode config — deliberately not vendored into this repo, so all -repo-specific context (contracts, spec authorities, gates) travels in each -dispatch prompt. - -| Agent | Model | Role | -|---|---|---| -| orchestrator (primary session) | fable | planning, contracts, dispatch, integration, review, **all commits** | -| `coder` | sonnet | implementation tracks against pinned contracts | -| `coder-hard` | opus | subtle tracks: shim internals, CABI edge cases, scheduler periphery | -| `reviewer` | fable | parallel code review when the orchestrator is the bottleneck | -| `explore` | haiku | fast read-only codebase search | - -Dispatch rules: - -- Every track prompt names: **territory** (paths owned), **governing - contracts** (`contracts/*.md` + design-doc sections), and **gates** (exact - commands). Territories are disjoint across concurrent tracks. -- Subagents never commit (permission-enforced); the orchestrator commits - after review. -- The task-scheduler **core** is single-owner (coherence risk): - `coder-hard` at most, under close orchestrator review; parallelism stays at - the periphery. - -Review protocol: every track is reviewed against its contracts before commit -— by the orchestrator inline, or by `reviewer` subagents in parallel. A -review dispatch **must** name the diff scope, the governing `contracts/*.md`, -and — for anything touching CABI/async semantics — -[docs/architecture.md](docs/architecture.md) §5–§7 plus the spec sources -(`definitions.py` as tie-breaker): the reviewer judges only against named -authorities and flags unnamed ones rather than filling gaps from memory. -Revision rounds go back to the *same* coder session via `task_id` (context -intact), not a fresh agent. - -Failure recovery (content-filter false positives, driver interrupts): an -aborted `task` call kills neither the child session (context persists in the -opencode db) nor its effects (files/commands persist on disk). Ladder: - -1. Locate the orphan (`opencode-agent-sessions `, on - PATH); resume via `task_id` — "summarize status, then continue". -2. Two failed resumes → assume poisoned context: fresh agent, handoff prompt - = original track + "partial work exists, audit state first" + artifact - pointers. Gates arbitrate what's already done. -3. Repeated failures across fresh contexts → escalate to the human; the - trigger may live in the artifacts themselves. - -Standing rules: - -- After any fan-out, reconcile launched-vs-completed before proceeding — a - missing result is not missing work. -- Never run one-off `npm:` specifiers (e.g. `deno run npm:yaml`) from the - workspace root: Deno records them into the root `deno.lock`, silently - dirtying the tree. Use python3 or - run from `/tmp`; check `git diff deno.lock` before staging. -- `main` is branch-protected: required checks = the `core` CI matrix, - force-pushes and deletions blocked, auto-merge enabled. Admin direct - pushes still work (`enforce_admins: false`), but PR + auto-merge is the - preferred delivery: it gets the required checks for free. The `browser` - job is deliberately NOT a required PR check (it runs post-merge only, - gating the prerelease) — do not add it to the protection contexts or - PRs will never merge. -- Versioning (README §Consuming): `@polyengine/{runtime,translator,wasi, - ct-runner}` version in **lockstep**, and the manifests always carry the - NEXT release. Still 0.x/unstable but caret-honest: a PR that breaks the - published surface bumps the lockstep minor in the same PR; compatible - work leaves the version alone. Releases are cut from a green `main` - commit via release.yml `workflow_dispatch` with `release=true` (guards: - lockstep, tag-exists, green `pre-` present), followed - immediately by a manifest-bump PR to the next patch — the four manifests - plus runtime's copy-identity constant `RUNTIME_VERSION` - (runtime/src/embedder/copy.ts; `just test-runtime` pins the sync). - `@polyengine/protocol` - versions independently; bumping its manifest publishes it for real at the - next cut. Prereleases (`pre-`, every green `main`) are GitHub - releases carrying artifacts only — JSR and npm are published by cut - releases exclusively. -- Breaking changes are declared by PR **label**, one per package: - `breaking/{runtime,translator,wasi,ct-runner,protocol}`. A label asserts - that the PR breaks that package's published surface (caret-incompatible); - no label means compatible. The labels are read LIVE from the API wherever - they are consulted — never from an event payload — because retroactive - edits are expected and load-bearing: noticing at cut time that a merged PR - was mislabelled and fixing the label there is a supported workflow, and - the cut re-reads the whole window. `tools/version-guard/check.ts` - enforces them in four places (`just test-version-guard` covers its - logic): `local` mode, first in `just gates` and an unconditional - `gha::core` step (label-free tree checks — lockstep agreement, - monotonicity, the protocol byte-identity tear check — so pre-push runs - and direct pushes are covered without PR context); - `pr` mode in `gha::core` (lockstep agreement, monotonicity, - label ↔ minor-bump agreement both ways, protocol-tear warning — an early - warning only, since label edits deliberately do not re-trigger CI); - `publish` mode in release.yml's publish step, both modes (in-tree - protocol must be byte-identical to the published version its manifest - names — the authoritative tear guard, which PR-time - checks cannot own because they miss post-run label edits, direct pushes - to main, and their own staleness at cut time; on the prerelease path - nothing publishes, so it is early detection of a tear the next cut would - hit); and `cut` mode on - `release=true`, which turns the window's labels into the minor-bump - requirement and renders the release notes. -- **The host ABI is versioned by `@polyengine/protocol`, gated by goldens** - (contracts/embedder-api.md §"The host-ABI surface and its version"). The - conventions suite - (`runtime/tests/conventions/`, rides `just test-runtime`; focused run: - `just test-conventions`) pins the host-facing lift/lower behavior as - committed transcripts under `runtime/tests/conventions/golden/`. - Modifying or deleting a golden asserts a host-ABI behavior change and - requires `breaking/protocol` in the same PR (the reviewed - behavior-neutral escape is the `conventions-fix` label); adding goldens - is free. version-guard enforces this in `pr` mode (advisory, live - labels) and authoritatively in `cut` mode (window-wide diff of the - goldens dir; M/D requires protocol on a later minor line than the last - cut, or a `conventions-fix` window PR). Host modules import - `@polyengine/protocol` at most — the runtime's exported surface is - application-only — so lockstep releases that leave the goldens - byte-identical cannot touch a host-provider package. -- **Cutting a release.** (1) Sanity pass, the step no machine can do: - enumerate the window — `gh pr list --search "base:main merged:>="` (or `gh api repos/$R/compare/v...main --jq - '.commits[].sha'`) plus `git log v..origin/main --first-parent - --oneline` for direct pushes — and read titles and diffs against the - labels. Fix labels retroactively NOW; a MISSING breaking label is the one - failure mode every mechanical check here is blind to. (2) Verify the - manifests against the final label set (breaking ⇒ the lockstep minor must - already be ahead of the last cut) and that `RUNTIME_VERSION` - (runtime/src/embedder/copy.ts) matches; check protocol's manifest too if - protocol moved. (3) Confirm the sha you are cutting has its green - `pre-` release — release.yml refuses otherwise, and that - refusal is the green-pipeline proof. (4) Dispatch release.yml with - `release=true` (`gh workflow run release.yml -f release=true --ref main`). - (5) Land the post-cut manifest-bump PR to the next patch immediately: the - four lockstep manifests + `RUNTIME_VERSION`. (6) Confirm the - npm-publish.yml run release.yml dispatched, and spot-check the dist-tags - (`npm view @polyengine/runtime dist-tags`) — `latest` must name the cut. - (The `pre` dist-tag is retired with the prerelease-publishing flow and - stays frozen wherever it last pointed.) -- Two registries, one version (protocol rides its own manifest version on - both). JSR is published inline by release.yml; npm - is published by npm-publish.yml, triggered by the GitHub release, from - packages built by `tools/npm-build/build.ts` (dnt). The npm side reads - name/version/exports out of the same `deno.json` manifests, so adding an - entry point or bumping a version needs no second edit — but `just - test-npm` is the gate that proves it, and the property it exists to pin - is that cross-package imports stay npm **dependencies** rather than - inlined source (duplicate copies are the multi-runtime-copy failure - mode the protocol brands exist to diagnose). npm auth is - OIDC trusted publishing keyed on the `npm-publish.yml` filename; there - is no npm token in the repository or its secrets. -- Consumer checkouts (the polymorph family, under `~/p/polymorph/`) are - **strictly read-only**: verify `git status` in any consumer tree you ran - commands near, before and after. Build artifacts go to `/tmp` or a - redirected `CARGO_TARGET_DIR`, never into consumer trees. -- Findings against foreign repos go in the tracker files - (`upstream-component-model-repo-findings.md`, - `upstream-consumer-findings.md`), not inline notes; filing them is the - operator's call. +Agent definitions and model choices live in the operator's global config, not +this repository. Honor session-specific model instructions. + +- Dispatches name owned paths, governing contracts/spec sections, and exact gate + commands. Concurrent implementation territories must be disjoint. +- The scheduler core has one implementation owner. Parallelize peripheral work + rather than independently changing shared scheduling rules. +- Subagents do not commit. The primary agent integrates, reviews, and commits. +- Review every track against its named authorities. CABI/async reviews include + architecture §5-§7 and the pinned spec/reference. Flag missing authorities + rather than supplying rules from memory. +- Resume revision rounds in the same agent session. After fan-out, reconcile + every launched track with its result; an absent response is not absent work. +- Interrupted sessions retain context and filesystem effects. Resume first; + after two failed resumes, hand off to a new agent with the partial artifacts. + Repeated failure requires escalation, not repeated blind relaunches. The + operator's subagent-recovery instructions describe the tooling. + +## Repository and consumers + +- `main` is protected. Deliver through a PR and auto-merge after required `core` + checks. The `browser` job runs post-merge and gates prerelease artifacts; it + must not become a required PR check. +- Do not run one-off `npm:` imports from the workspace root: Deno may write them + into `deno.lock`. Use an existing dependency or scratch work under `/tmp`; + inspect lockfile changes before staging. +- Consumer checkouts under `~/p/polymorph/` are read-only. Check their git + status before and after verification. Put new build artifacts in `/tmp` or a + redirected `CARGO_TARGET_DIR`, never in consumer trees. +- Findings against foreign repositories belong in + `upstream-component-model-repo-findings.md` or + `upstream-consumer-findings.md`. Public filing requires the operator's + authorization. + +## Versioning and publishing + +`@polyengine/{runtime,translator,wasi,ct-runner}` version in lockstep. Their +manifests carry the **next** release. Compatible changes leave versions alone; +breaking changes move the lockstep minor. `@polyengine/protocol` versions +independently, and changing its manifest publishes that version at the next cut. +Runtime's `RUNTIME_VERSION` in `runtime/src/embedder/copy.ts` must match its +manifest. + +Declare breaking surfaces with `breaking/runtime`, `breaking/translator`, +`breaking/wasi`, `breaking/ct-runner`, or `breaking/protocol` PR labels. Labels +are read live: correcting a merged PR's label before a release is supported. +Missing labels are not detectable by the mechanical guard; review the release +window's diffs. + +`tools/version-guard/check.ts` has four modes (`just test-version-guard`): + +| Mode | Checks | +| --------- | ----------------------------------------------------------------------------------------------------------------- | +| `local` | Label-free lockstep, monotonicity, and protocol byte identity; first in `just gates` and unconditional in core CI | +| `pr` | Live label/minor-bump agreement and golden-change labels; early warning because label edits do not rerun CI | +| `publish` | In-tree protocol is byte-identical to its named published version; runs for both prereleases and cuts | +| `cut` | Release-window labels, required minor advances, golden changes, and release notes | + +The host ABI belongs to protocol, not the runtime package. Its committed +transcripts are in `runtime/tests/conventions/golden/` +(`just test-conventions`): modifying/deleting one requires `breaking/protocol` +and the protocol minor bump, unless reviewed as a suite correction under +`conventions-fix`. Adding goldens is free. The cut guard checks the whole +release window. Host modules import protocol at most; runtime exports are +application machinery. + +Both JSR and npm use the same package manifests. JSR publishes in `release.yml`; +npm packages are built by `tools/npm-build/build.ts` using dnt and published by +`npm-publish.yml`. `just test-npm` verifies packaged exports, declarations, and +cross-package dependencies: dependencies must not be inlined into duplicate +runtime/protocol copies. npm uses OIDC trusted publishing keyed to the workflow +filename, not a repository token. + +Every green main commit produces a GitHub `pre-` release containing +artifacts only. Registry publishing happens only on cut releases; the old npm +`pre` tag is frozen. + +## Cutting a release + +1. Enumerate changes since the last cut, including direct main commits. Use + `gh pr list --search "base:main merged:>="` and + `git log v..origin/main --first-parent --oneline`. Read diffs against + labels and correct missing breaking labels before proceeding. +2. Check final lockstep versions, `RUNTIME_VERSION`, and protocol's version + against the release window. A breaking surface requires the appropriate minor + advance beyond the last cut. +3. Confirm the target SHA has its green `pre-` release. +4. Dispatch `gh workflow run release.yml -f release=true --ref main`. +5. Immediately land a manifest-bump PR to the next patch for the four lockstep + manifests and `RUNTIME_VERSION`. +6. Confirm the dispatched `npm-publish.yml` run and check + `npm view @polyengine/runtime dist-tags`: `latest` must name the cut. diff --git a/README.md b/README.md index a3b819c..d2b82e6 100644 --- a/README.md +++ b/README.md @@ -1,103 +1,55 @@ # polyengine -A WebAssembly **Component Model host** for JavaScript engines — async-native -(Component Model 0.3 concurrency), built on wasmtime's translation frontend -compiled to wasm, running on Deno and in browsers. - -Instead of ahead-of-time transpilation, polyengine is a **runtime linker**: it -takes a `.wasm` component binary, translates it in-process (wasmtime-environ + -FACT fused adapters, running as a wasm32 module), and executes the -instantiation plan on the stock `WebAssembly` JS API. Cross-component calls -stay pure wasm; the 0.3 task model (tasks, streams, futures, backpressure, -cancellation) is the runtime's core structure, mapped onto the JS event loop — -the callback ABI needs no JSPI at all, and the stackful/blocking forms light -up via JSPI where the engine provides it. +A WebAssembly **Component Model host for JavaScript**. It loads component +binaries at runtime, using wasmtime's translation frontend compiled to wasm +and the stock `WebAssembly` JS API to execute them. -## Status - -Pre-1.0, but densely gated: - -- **Official Component Model test suite**: 1285 passing / 0 failing commands - across all directories (remaining: named xfail classes — deferred threads - ([#12](https://github.com/polymorph-components/polyengine/issues/12)), - cm705-sync-sched ([#249](https://github.com/polymorph-components/polyengine/issues/249)), - cm707-cancel ([#250](https://github.com/polymorph-components/polyengine/issues/250)), - upstream-nyi ([#248](https://github.com/polymorph-components/polyengine/issues/248))), - identical on Deno, Chromium, and Firefox - (behind its JSPI pref); WebKit reaches the same totals on trunk builds - (the pinned build lacks JSC multi-memory, since implemented upstream). -- **Real-workload proof points** (the [polymorph] component family): - the iroh endpoint component — detached pump tasks, multi-export - concurrency, cross-task wakeups; the workload that deadlocks jco's - scheduler — runs its relay-echo and WebRTC-upgrade paths end-to-end; - polymorph-websocket's conformance suite passes 55/55 under this host; - an 8 MB componentize-go engine instantiates and runs. -- Guest toolchains exercised: wit-bindgen (Rust) and componentize-go, - sync and async. - -## Layout - -| Path | What | -|---|---| -| `crates/translator-shim` | wasmtime-environ + FACT → versioned plan format (wasm32, runs everywhere) | -| `runtime/` | TS core: plan executor, canonical ABI, 0.3 task scheduler, JSPI bridge, embedder API (`runtime/src/embedder`) | -| `crates/bindgen` | WIT → TypeScript types for the embedder conventions | -| `examples/` | **start here to embed**: hello-world + kitchen-sink (WIT + Rust guest + TS host, self-checking), plus the guest fixture corpus | -| `translator/` | `@polyengine/translator`: the packaged translator asset + `defaultTranslator()` per-platform loader (build-time alternative: `tools/translate`) | -| `wasi/` | minimal WASI providers (p2 baseline + p3 clocks), one per semver track | -| `ct-runner/` | conformance-suite runner for the polymorph-test L1 contract | -| `harness/` + `tools/browser` | official-suite harness; Deno lane + Chromium/Firefox/WebKit lanes | -| `contracts/` | the versioned interface contracts (plan format, embedder API, intrinsics, digest) | - -## Quick start +The runtime implements Component Model 0.3 concurrency: tasks, streams, +futures, backpressure, and cancellation. Callback-ABI async calls do not +require JSPI; stackful async calls and blocking sync imports do. FACT fused +adapters handle cross-component ABI conversion in wasm; TypeScript handles +the host boundary and scheduler. -```sh -git clone --recursive https://github.com/polymorph-components/polyengine -cd polyengine -just test-runtime # runtime suite (builds the shim + fixtures + corpus first) -just conformance # official CM suite on Deno -just browsers-install && just browser-lane chromium # same corpus, real browser -``` - -Deno workspace (TS) + cargo workspace (Rust); [`just`](https://github.com/casey/just) -is the command surface (`just --list`; recipe bodies are the exact commands). - -## Translating components - -Running a component takes a translation (an execution plan + FACT adapter -modules). Three ways to get one: - -| method | production ships | choose when | -|---|---|---| -| **build-time** — [`tools/translate`](tools/translate/) emits a single-file *envelope*; the host reconstitutes it with `artifactsFromEnvelope(envelope, componentBytes)` | component + envelope + runtime — **no translator** | you know your components at build time (most apps; the browser default — saves ~0.5 MB gzip and a compile per visitor). The envelope embeds the component's sha-256, so a stale pair fails loudly at instantiation | -| **runtime, packaged** — `defaultTranslator()` from [`@polyengine/translator`](translator/), passed to `instantiate({ componentBytes, translator })` | your host + the translator asset (~1.85 MB raw, 520 KB gzip) | components arrive dynamically (plugin systems), or dev/server contexts where the asset size is irrelevant. Pair with the artifact cache (`@polyengine/runtime/cache`) so each component translates once per client, not once per load | -| **runtime, explicit** — `Translator.create(bytes)` / `Translator.fromExports(ns)` from `@polyengine/runtime/shim` | same, minus the packaged loader | you source the translator wasm yourself: custom delivery, one shared instance across many components, or cache keying via `buildHash` | +## Status -Translation itself is sub-millisecond warm in all three; the methods differ -only in *when* it runs and *what you deploy*. Worked code: the -[examples](examples/) use the packaged form, [`tools/translate`'s -README](tools/translate/README.md) shows the build-time deploy recipe, and -the full decision record is the design note on -[#16](https://github.com/polymorph-components/polyengine/issues/16). +Pre-1.0, with conformance and integration gates rather than a claim of full +spec coverage. The official Component Model corpus runs on Deno, browser, +and engine-shell lanes. Expected failures and engine-specific differences +are tracked explicitly; unexpected failures and stale expectations fail +the applicable gate. + +- Supported workloads include Rust/wit-bindgen and componentize-go guests, + sync and async, including composed components and overlapping exports. +- Runtime coverage includes canonical ABI values, resources, async host + imports, streams/futures, and background progress between export calls. +- Known gaps include deferred thread features, upstream-unimplemented + features, and sync scheduling gaps. See + [architecture §11](docs/architecture.md#11-conformance-and-testing) and the + [issue tracker](https://github.com/polymorph-components/polyengine/issues). +- Guest-initiated resource destructors cannot suspend through JSPI: their + current dispatch path contains a JS frame. See + [architecture §7](docs/architecture.md#7-canonical-abi-decisions). +- Engine support depends on the component's core-wasm features as well as + its use of JSPI. See the + [compatibility table](docs/architecture.md#3-compatibility-targets). + +Host filesystem and network access is opt-in. **WASI path and request +checks are not a sandbox for hostile guests**; read [security.md](docs/security.md) +before granting host access. ## Consuming -Everything here is **unstable** (0.x, [#16](https://github.com/polymorph-components/polyengine/issues/16)): -no compatibility promise across minor lines. But releases are -**caret-honest**: within a minor line they stay backward-compatible, and -anything breaking bumps the minor — so caret constraints are the intended -way to consume: +Packages are available on JSR and npm. Releases are compatible within a +minor line; breaking changes bump the minor. Use a caret range for a +released version, not the next-release version in the checkout's manifests: ```ts -import { instantiate } from "jsr:@polyengine/runtime@^0.4.0/embedder"; -import { defaultTranslator } from "jsr:@polyengine/translator@^0.4.0"; +import { instantiate } from "jsr:@polyengine/runtime@^0.6.7/embedder"; +import { defaultTranslator } from "jsr:@polyengine/translator@^0.6.7"; ``` -The same five packages ship to **npm** under the same names, built from the -same sources at the same version by the same release: - ```sh -npm install @polyengine/runtime @polyengine/translator +npm install @polyengine/runtime@^0.6.7 @polyengine/translator@^0.6.7 ``` ```js @@ -105,65 +57,96 @@ import { instantiate } from "@polyengine/runtime/embedder"; import { defaultTranslator } from "@polyengine/translator"; ``` -The npm distribution is ESM-only (Node >= 22.14) and carries `.d.ts`; entry -points match the JSR subpaths one for one, so the import specifier is the only -line that differs between registries. Two things are JSR-only, both by -necessity rather than policy: `dirCache()` (the `Deno.*` filesystem cache -backend — use `webCache()` or your own `ArtifactCache`), and the translator's -permission-free Deno wasm-module load, which the npm build replaces with a -`node:fs` read of the same packaged asset. - -`@polyengine/{runtime,translator,wasi,ct-runner}` release in lockstep — one -version, cut from one green commit, matching the `vX.Y.Z` -[GitHub release](https://github.com/polymorph-components/polyengine/releases) that carries the -same commit's artifacts. (`@polyengine/protocol` versions independently; the -others depend on it by caret.) - -This project was previously named **deltic** and published under the -`@deltic` JSR scope, which stops at `0.2.1`. The rename is a clean break, -not an alias: `@polyengine/*` starts a fresh version line, the -`Symbol.for("polyengine.*/1")` cross-copy brands do not match the old -`deltic.*` ones, and the `POLYENGINE_*` environment variables replace their -`DELTIC_*` spellings. Nothing bridges the two — port in one step. - -Between releases, every green `main` commit still gets a -`pre-` [GitHub release](https://github.com/polymorph-components/polyengine/releases) -carrying the same artifacts a cut carries — the translator shim wasm, the -embedder bundle, `SHA256SUMS` — but **nothing is published to JSR or npm -between cuts**: the registries carry `vX.Y.Z` releases only. To track -`main`, use those release assets or a git reference and upgrade -deliberately. (The `-pre.g` versions on JSR and the npm -`pre` dist-tag are frozen leftovers of the earlier flow, which published -every green commit; nothing new lands there.) +Start with [hello-world](examples/hello-world/) for a complete embedding or +[kitchen-sink](examples/kitchen-sink/) for suspending imports, resources, +and host value shapes. Both examples build a Rust guest and run a +self-checking TypeScript host. + +The npm distribution is ESM-only, includes `.d.ts`, and declares Node +>= 22.14. That package floor does not imply JSPI support; blocking forms +need a suitable engine. JSR and npm expose matching subpaths, but +`dirCache()` requires Deno and the packaged translator uses platform-specific +asset loading. npm replaces Deno's permission-free wasm-module import with +a `node:fs` read. + +`@polyengine/{runtime,translator,wasi,ct-runner}` release in lockstep. +`@polyengine/protocol`, the shared host-ABI vocabulary, versions +independently. JSR and npm receive cut releases only. Green `main` commits +produce `pre-` +[GitHub releases](https://github.com/polymorph-components/polyengine/releases) +with artifacts, not registry publications; use those assets or a git +reference to track unreleased work. Deno's [minimum-dependency-age](https://docs.deno.com/runtime/packages/supply_chain/#minimum-dependency-age) -gate (24 h by default) applies to every publish, so a freshly cut release -won't resolve on day zero. To consume same-day releases while keeping the -gate for the rest of your graph, exempt the scope -(wildcard excludes work as of Deno 2.9): +may delay resolution of a fresh cut. If same-day releases are needed, Deno +2.9+ supports a scope-specific exception without disabling the check for +the rest of the dependency graph: ```jsonc // deno.json { "minimumDependencyAge": { "age": "P1D", "exclude": ["jsr:@polyengine/*"] } } ``` -(or `--minimum-dependency-age=0` for a one-off run). +## Translating components -## Documentation +Execution needs the original component bytes, an instantiation plan, and +FACT adapter modules. Choose when to produce the plan and adapters: -| Where | What | -|---|---| -| [`examples/`](examples/) | runnable embedder examples: [hello-world](examples/hello-world/) (smallest complete embedding) and [kitchen-sink](examples/kitchen-sink/) (imports incl. suspending, resources both directions, value-shape tour) | -| [`docs/architecture.md`](docs/architecture.md) | the system design and decisions, with rationale (§-numbered; cited from code comments) | -| [`docs/security.md`](docs/security.md) | what the WASI filesystem/network confinement does and does not guarantee — **read before granting a guest host access** | -| [`docs/consumers.md`](docs/consumers.md) | the polymorph adoption track: standing conventions, pins, in-repo consumer artifacts | -| [`docs/references.md`](docs/references.md) | canonical upstream links (spec, JSPI, wasmtime internals, toolchain pins) | -| [`contracts/`](contracts/) | versioned interface contracts — [plan format](contracts/plan-format.md), [descriptor IR](contracts/descriptor-ir.md), [intrinsics](contracts/intrinsics.md), [digest](contracts/digest.md), [embedder API](contracts/embedder-api.md) | -| [`AGENTS.md`](AGENTS.md) | development protocol and the full gate list | -| [issue tracker](https://github.com/polymorph-components/polyengine/issues) | open and deferred work | +| Method | Deployment | Use when | +|---|---|---| +| **Build-time**: [`tools/translate`](tools/translate/) emits an envelope; load it with `artifactsFromEnvelope(envelope, componentBytes)` | Component + envelope + runtime, no translator | Components are known at build time; avoids shipping and compiling the translator on clients | +| **Runtime, packaged**: `defaultTranslator()` from [`@polyengine/translator`](translator/), passed to `instantiate({ componentBytes, translator })` | Component + runtime + translator asset | Components arrive dynamically or translation belongs in the host process | +| **Runtime, explicit**: `Translator.create(bytes)` / `Translator.fromExports(ns)` from `@polyengine/runtime/shim` | Same, with a caller-managed translator | Custom asset delivery or translator-instance management | + +These use the same translation pipeline, not different execution engines. +The host boundary currently interprets CABI descriptors; build-time +translation does **not** emit specialized JavaScript. The envelope records +the component's SHA-256 to reject mismatched pairs, but is itself a trusted +input, not a proof of translation correctness. + +For runtime translation, [`@polyengine/runtime/cache`](runtime/src/cache/) +can reuse artifacts across loads when the translator has a `buildHash`. +The packaged Deno wasm-module loader currently lacks that hash; caching +requires a translator constructed from bytes or supplied with a known +asset hash. See [caching](docs/architecture.md#10-caching), the +[build-time recipe](tools/translate/README.md) and +[cache trust boundary](docs/security.md#the-artifact-cache-is-a-trust-input). + +## Quick start + +Development uses Deno, Rust, and [`just`](https://github.com/casey/just). +`just --list` lists the supported commands; recipe bodies are the command +source of truth. + +```sh +git clone --recursive https://github.com/polymorph-components/polyengine +cd polyengine +just test-runtime # builds the shim, guest fixtures, and corpus first +just conformance # official Component Model corpus on Deno +just browsers-install && just browser-lane chromium +``` -[polymorph]: https://github.com/polymorph-components +## Layout ---- +| Path | Purpose | +|---|---| +| `crates/translator-shim` | wasmtime-environ + FACT to versioned plan format; compiled to wasm32 | +| `runtime/` | Plan executor, canonical ABI, scheduler, JSPI bridge, embedder API | +| `crates/bindgen` | WIT to typed TypeScript facades | +| `protocol/` | Shared host-ABI brands, errors, and provider conventions | +| `translator/` | Packaged translator asset and loader | +| `wasi/` | WASI providers, with filesystem, sockets, HTTP, and host stdio as opt-in fragments | +| `ct-runner/` | Runner for polymorph-test L1 conformance suites | +| `examples/` | Runnable embeddings and guest fixtures | +| `harness/`, `tools/browser/`, `tools/shell/` | Conformance corpus and cross-engine lanes | + +## Documentation -125% more engine! +| Document | Purpose | +|---|---| +| [Architecture](docs/architecture.md) | Current implementation, semantic policy, and known limitations | +| [Security](docs/security.md) | Host authority, confinement limits, and artifact trust | +| [Consumers](docs/consumers.md) | Polymorph integration conventions and consumer gates | +| [References](docs/references.md) | Spec, JSPI, dependency pins, and platform references | +| [Contracts](contracts/) | Versioned [plan format](contracts/plan-format.md), [descriptor IR](contracts/descriptor-ir.md), [intrinsics](contracts/intrinsics.md), [digest](contracts/digest.md), and [embedder API](contracts/embedder-api.md) | +| [Development protocol](AGENTS.md) | Change discipline, gates, and release process | diff --git a/bench/boundary/README.md b/bench/boundary/README.md index 9f6b533..90c4643 100644 --- a/bench/boundary/README.md +++ b/bench/boundary/README.md @@ -1,27 +1,27 @@ # bench/boundary — the host-boundary microbench -Calls-per-second across the host import boundary, per ABI shape — the -instrument behind [#17](https://github.com/polymorph-components/polyengine/issues/17)'s -jco-vs-polyengine baseline, [#54](https://github.com/polymorph-components/polyengine/issues/54)'s -lift-throughput finding, and [#8](https://github.com/polymorph-components/polyengine/issues/8)'s -cost ledger. The design goal is attribution, not realism: the guest is a -tight loop over echo-shaped imports whose host bodies are trivial, so -what the clock sees is lift/lower + dispatch + (for async shapes) the -suspension machinery — and both stacks run on **the same engine** (plain -`node` runs the polyengine callback ABI with no flag; the jco lane and -polyengine's jspi mode share `--experimental-wasm-jspi`), so V8/GC/JIT -variables cancel. +A manual instrument for host calls, stream transfer, and compound-value +lift/lower. The guest uses synthetic loops and trivial host bodies, not a real +application workload. The Node lanes share one Node executable, reducing engine +version differences, but GC, JIT, process scheduling, and thermal variation do +not cancel. Deno is a separate engine build. ```sh just bench-boundary # polyengine lanes: node callback+jspi, deno just bench-boundary with-jco # + the incumbent jco lane (npm ci + transpile on first use) ``` -The polyengine lanes measure the CURRENT TREE: the recipe builds the local -embedder bundle (`tools/release-bundle/build.ts`) and the local -translator shim. Numbers are box-relative — compare lanes within one -run, or the same lane across commits on one box, never absolute values -across machines. +The recipe builds the local guest, embedder bundle, and translator shim. It +requires Rust with `wasm32-wasip2` and `wasm32-unknown-unknown`, Node, Deno, and +optionally npm for the jco lane. `sweep.mjs` currently passes +`--experimental-wasm-jspi` to the Node JSPI lanes; callback lanes explicitly +disable JSPI. Consult [architecture](../../docs/architecture.md) for engine +support, rather than inferring requirements from this historical runner flag. + +The tables below are **historical measurements, not current-tree performance +claims**. Compare fresh interleaved runs on the same machine and record the +engine versions, component/toolchain pins, and source revision. These +microbenchmarks are not gates or evidence of application throughput. ## Shapes @@ -31,13 +31,18 @@ across machines. | `recv` | `fetch: async func(u32) -> list` | UDP-receive-shaped: payload host→guest | | `send-sync` | `ping-sync: func(list) -> u32` | the sync-lowered control | -Host settlement `mode`: `immediate` (a plain return value — the fast -path) and `microtask` (an async host fn, i.e. an already-resolved -promise — the wakeup-shaped path a real receive takes). Payload sizes 0 +Host settlement `mode`: `immediate` (a plain return value) and `microtask` +(an async host function returning an already-resolved Promise). The latter +does not simulate network delay. Payload sizes 0 (pure call overhead) and 1200 B (QUIC-ish MTU). Medians of 5 timed export calls after a warmup call; each export call runs `iters` boundary crossings. +`send-sync` uses a plain synchronous host function in both mode rows; the +`microtask` label does not make that control suspend. Timing also includes +guest work: `send` clones its payload on each iteration, while `send-sync` +borrows its buffer. + ## Stream shapes ([#68](https://github.com/polymorph-components/polyengine/issues/68)) | export | what it measures | @@ -46,17 +51,17 @@ boundary crossings. | `stream-source: async func(n: u32) -> stream` (guest pumps n bytes) | guest→host payload | | `stream-pass: async func(s: stream) -> stream` (guest returns its input unchanged, never reads) | host↔host rendezvous after identity transfer (contracts/embedder-api.md §"Streams and futures") | -Unlike the calls-per-second shapes above, none of these involve a host -import: the host drives the stream endpoint directly via the embedder's -`Stream` API (`contracts/embedder-api.md` §"Streams and futures"), so -what's measured is the rendezvous/copy cost in isolation. Reported as -MB/s (bytes moved ÷ elapsed), medians of 5 timed runs after a warmup, -same convention as the calls-per-second table. Chunk sizes 1200 B, -16 KiB, 256 KiB; the chunk COUNT per size is chosen (`sweep.mjs`, -`STREAM_CONFIGS`) so one timed run lands in the tens-of-ms range — -smaller chunks need more of them to reach a measurable duration, larger -chunks need fewer. **polyengine drivers only**: jco's p3 stream support is -not under test here, so the jco lane is skipped for these rows. +These shapes do not involve a host import: the host creates stream pairs with +`createStream()` and reads or writes handles directly. See the +[embedder contract](../../contracts/embedder-api.md) for stream semantics. +Timing includes export dispatch, rendezvous, and any payload allocation/copy +inside the timed region, not an isolated copy primitive. + +The driver labels throughput `MB/s` but computes **MiB/s** (bytes divided by +`1024 * 1024`, then by elapsed seconds). The historical tables retain their +original labels. Results are medians of five timed runs after a warmup. +`STREAM_CONFIGS` in `sweep.mjs` sets chunk sizes and counts. Only polyengine +drivers run these shapes; jco stream support is not measured. ## Compound element shapes ([#261](https://github.com/polymorph-components/polyengine/issues/261)) @@ -65,38 +70,28 @@ not under test here, so the jco lane is skipped for these rows. | `lift-ops: async func(n: u32) -> list` (guest returns `n` elements) | compound-element LIFT | | `lower-ops: async func(ops: list) -> u64` (guest folds, returns a checksum) | compound-element LOWER | -`op` is a 16-case variant over records — the width and payload mix -mirror #261's reported 17-case consumer schema (a DOM-mutation op -stream, records carrying `string`, `option`, `list`, and a -nested payload-free variant `node-update-kind`), because two of the -costs #261 identifies — `maxCaseAlignment` (`runtime/src/cabi/layout.ts`) -and the embedder facade's variant case resolution -(`runtime/src/embedder/values.ts` `toHost`) — are both O(case count) per -element. Every other shape in this instrument — `list`, `u32`, -`stream` — is a flat scalar and takes a bulk copy path (issues -#63/#67); before this lane, NOTHING in `bench/boundary` exercised the -per-element interpreted lift/lower loop that compound types (records, -variants, options, strings) actually walk. Both directions are measured -separately because `load.ts`'s per-element path and `store.ts`'s -per-element path are separate code with the same defect. - -Reported as ns/element (`iters` reused as the element count `n`; `size` -is unused, passed through as "n/a" like `mode` is for the stream shapes) -— the unit that makes these numbers comparable to a whole variant-over- -records lift/lower rather than a byte or a call. Medians of 5 timed runs -after a warmup, same convention as the other tables. Element count -10000, calibrated (`sweep.mjs`, `ELEMENT_N`) so a timed run lands in the -tens-of-ms range. **polyengine drivers only**, same reason as the stream -shapes: jco is not under test here. - -Methodology footnote: `lift-ops`'s guest caches its `Vec` in a -`thread_local!` keyed on `n` — the warmup call builds it, every timed -call clones the cached vector, measured in isolation at ~15 ns/element -on this box (a temporary export cloned without crossing the boundary). -`lower-ops`'s host array is built ONCE outside the timed loop, since -that lane measures lowering, not host array construction. - -## Baseline (2026-08-11, linux-arm64 dev box, Node 24.18 / Deno 2.9.5, guest wit-bindgen 0.60; post-#63/#67 bulk list copies) +`op` is a 16-case variant over records, based on #261's DOM-mutation workload. +It contains strings, options, byte lists, and a nested payload-free variant. +Unlike the bulk byte-copy shapes, these rows exercise per-element compound +conversion. Lift and lower are measured separately; the old per-element +layout/case scans that motivated these rows have since been optimized. + +Reported as ns/element: `iters` is the element count `n`, and the input `size` +argument is unused (the JSON result records `size: n`). `ELEMENT_N` in +`sweep.mjs` is 10000. Each result is the median of five timed runs after a +warmup. Only polyengine drivers run these shapes; jco is not measured. + +`lift-ops`'s guest caches its `Vec` by `n`; warmup builds it and timed calls +clone it, so cloning remains in the measurement. The original investigation +measured that clone at about 15 ns/element on its dev box using a temporary +export, not a standing benchmark. `lower-ops` builds its host array outside +the timed loop. + +The sink also collects received bytes into a guest vector, and `lower-ops` +folds the received values into a checksum. These costs are part of the timed +work, not subtracted overhead estimates. + +## Historical baseline (2026-08-11, linux-arm64 dev box, Node 24.18 / Deno 2.9.5, guest wit-bindgen 0.60; post-#63/#67 bulk list copies) ``` shape mode size polyengine-node-callback polyengine-node-jspi polyengine-deno-callback jco-node-jspi @@ -131,14 +126,12 @@ Two methodology footnotes for the stream rows: - `stream-source` allocates and fills its whole payload inside the guest within the timed region (one `vec![0x5a; n]` per run), where `stream-sink`'s host payload is preallocated outside it — the source - lane over-measures by one guest alloc+fill per run. Consistent across - runs, so trend-tracking is unaffected; just don't read sink-vs-source - deltas as pure copy-direction cost. -- "size" means write granularity for `stream-sink` but host *read* - granularity for `stream-source`/`stream-pass` (their producers offer - everything at once; `wit_stream`'s writer does its own internal - chunking). One dimension, two meanings — split it if a finding ever - hinges on the distinction. + lane includes one guest alloc+fill per run. Do not read sink-vs-source + deltas as pure copy-direction cost, or assume allocation cost is stable. +- In the current driver, "size" means host write granularity for `stream-sink`, + host read granularity for `stream-source`, and both host write and read + granularity for `stream-pass`. The guest source offers its payload to the + stream writer, whose internal chunking is separate. ### Compound element shapes baseline (2026-09-03, linux-arm64 dev box, Node 24.18 / Deno 2.9.5, guest wit-bindgen 0.60) — before #261's optimization PRs (#263/#264/#265 landed after this was recorded) @@ -149,13 +142,12 @@ lift-ops 3,804.4 3,909.3 lower-ops 3,482.3 3,646.8 3,285.3 ``` -This is the "before" baseline for #261, recorded before any optimization -of the per-element interpreted path lands. ~3.2-3.9 µs/element here vs. -#261's reported ~5 µs/element on a similar box — same order of -magnitude, within ~1.6x; the residual gap reads as box/config drift. -See the 2026-09-04 block below for the "after" numbers. +This is the pre-optimization record for #261. The 2026-09-04 block below records +the post-optimization run; neither table predicts the current tree's cost. +The original #261 consumer report was about 5 microseconds/element on a similar +machine, but that was a different workload/configuration, not a matched lane. -## Baseline (2026-09-04 — post-#261, linux-arm64 dev box, Node 24.18 / Deno 2.9.5, guest wit-bindgen 0.60; #263 layout-node cache + #264 adapter tables + #265 flatten-count memoization) +## Historical baseline (2026-09-04 — post-#261, linux-arm64 dev box, Node 24.18 / Deno 2.9.5, guest wit-bindgen 0.60; #263 layout-node cache + #264 adapter tables + #265 flatten-count memoization) ``` compound-element lanes (ns/element; n=10000; jco lane skipped — see README.md): @@ -164,58 +156,33 @@ lift-ops 677.8 704.2 lower-ops 630.4 661.4 694.6 ``` -The compound-element drop against the 2026-09-03 "before" table is -3.7x-5.6x fewer ns/element depending on lane (#263's layout-node cache -plus #264's adapter tables). The calls-per-second table is NOT refreshed -here: this box cannot currently reproduce it — `send immediate 0` / -`polyengine-node-jspi` alone read 780,785/s (2026-08-11), 1,023,625/s -(an interleaved run today), and 521,044/s (this sweep), a 2x spread on -identical code, so a fresh table would be noise with a date on it. What -is known instead, from interleaved before/after pairs (medians of paired -differences, `immediate`, size 0, attributable to #265's per-call -flatten-count memoization, reproduced across two passes): `send-sync` -+27%/+32%, `send` +22%/+28%, `recv` +34%/+31% calls/sec — a delta, not a -new absolute baseline. The 2026-08-11 table remains the recorded -calls-per-second baseline, known to understate the current tree, until a -quiet box allows a real re-measurement. Stream rows are also NOT -re-measured: `stream-sink` at 256 KiB spans 2,900-10,800 MB/s across four -interleaved runs with no consistent before/after sign, and none of -#263/#264/#265 touch the `stream` bulk-copy path, so the 2026-08-11 -stream rows above still stand as the current record. - -## What the baselines say - -- **Async import round-trips**: polyengine's callback ABI sustains 0.3–1.1 M - crossings/s; jco's async path costs ~3 ms per call flat - (timer-quantized — its sync path is healthy at ~300 k/s, so the cost - is the async task loop, the same machinery behind lann/jco#11 and - polymorph-iroh's 5 ms polling workaround). For the UDP direct path - (#4) this is the difference between "boundary is free" and "boundary - is the bottleneck". -- **#54 (fixed in #63; sentinel rows)**: `recv @ 1200` once ran ~18 k/s - (~22 MB/s, a per-element interpreted store); it now tracks the empty - call within ~1 % — the payload copy is bulk in both directions, and - these rows are the regression sentinel. #67 extended the bulk copies - to the remaining flat element types (not separately represented here; - the shapes are `list`). -- **jspi vs callback** (same runtime, same engine): parity on - immediate-settled paths, ~2–4× behind on deferred (microtask) paths — - the suspend/resume cost, recorded for #8. -- **#68 stream shapes**: `stream-pass` (host↔host rendezvous, no guest - memory touched — contracts/embedder-api.md §"Streams and futures") - consistently beats `stream-sink` and - `stream-source` (host↔guest, which pay a real memory copy through the - guest's linear memory) at every chunk size, confirming the identity - transfer is doing what it claims. All three scale up sharply with - chunk size — per-rendezvous overhead amortizes over more bytes. -- **#261 compound elements**: the first instrument for the interpreted - per-element lift/lower path — every prior shape here is flat and - bulk-copies. #263 (layout-node cache) + #264 (adapter tables) moved - `lift-ops`/`lower-ops` from ~3.2-3.9 µs/element to ~0.6-0.9 µs/element - — same sentinel role #54/#67 played for flat types, now proven out. - -The jco lane pins the family's own toolchain (the lann/jco all-fixes -transpile + preview2-shim release tarballs, the vendored -`jco-transpile.mjs` wrapper, and polymorph-test's `bindImports` for the -WASI spellings — the exact stack the consumer repos' jco legs run). It -exists as the incumbent baseline and retires with the jco era. +The recorded compound-element time fell by 3.7x-5.6x relative to 2026-09-03. +Calls/sec and stream tables were not refreshed on 2026-09-04 because repeated +measurements varied substantially. For `send immediate 0` in the Node JSPI lane, +the recorded values were 780,785/s on 2026-08-11, then 1,023,625/s in an +interleaved run and 521,044/s in the September sweep. Paired September passes +reported `send-sync` +27%/+32%, `send` +22%/+28%, and `recv` +34%/+31% calls/sec +for #265 (`immediate`, size 0). Those are historical relative observations, not +a replacement absolute baseline. `stream-sink` at 256 KiB ranged from +2,900 to 10,800 in the driver's `MB/s` units across four interleaved runs, with +no consistent before/after sign. + +## Interpreting comparisons + +The August table captured a large async-call difference against the pinned jco +toolchain and a callback/JSPI difference on microtask-settled imports. It does +not establish current jco performance or isolate a single scheduling cost. +Likewise, stream throughput does not prove ownership-transfer correctness; +functional tests establish that. The byte-copy rows are useful regression +probes for #54/#63/#67, and the compound rows for #261/#263/#264/#265, but each +new claim needs a fresh controlled comparison. + +For historical context, #54 reported about 18,000 calls/sec (about 22 MB/s as +originally reported) for `recv` at 1200 bytes before #63's bulk-copy change. +That earlier finding is not a current measurement or a row from the dated +tables above. + +The optional jco lane uses the toolchain pinned in `package.json` and +`package-lock.json`, the local `jco-transpile.mjs` wrapper, and `bindImports` +for WASI spellings. It is a comparison with that pinned stack, not necessarily +the current stack in any consumer repository. diff --git a/contracts/descriptor-ir.md b/contracts/descriptor-ir.md index 83ce2f4..7f14e63 100644 --- a/contracts/descriptor-ir.md +++ b/contracts/descriptor-ir.md @@ -1,14 +1,14 @@ # Contract: CABI Descriptor IR -The **descriptor IR** is the type/options information that drives host-boundary -lift/lower — docs/architecture.md §8's "one IR, two executors." Producers: the -translator shim (inside `plan.json` `types` / `canonicalOptions` tables) and -tests. Consumers: the v1 interpreter (`runtime/src/cabi/`), the future -generated-JS executor, and the world-digest computation. +The **descriptor IR** supplies types and canonical options for host-boundary +lift/lower. The translator emits it in the plan's `types` and `canonicalOptions` +tables. The runtime interpreter (`runtime/src/cabi/`) and world-digest +computation consume it. A generated-JS executor is planned, not implemented; see +[architecture §8](../docs/architecture.md#8-performance-strategy). The normative in-memory model is `runtime/src/cabi/types.ts`; this document -defines its meaning and its JSON wire form inside the plan. Known -wire↔memory divergences are pinned below; any other divergence is a bug. +defines its meaning and its JSON wire form inside the plan. Known wire↔memory +divergences are pinned below; any other divergence is a bug. ## Value type model @@ -19,102 +19,88 @@ structurally). Kinds: `f32`, `f64`, `char`, `string` - `list` (`element`, optional fixed `length`), `record` (`fields`: `{label, type}[]`), `tuple` (`elements`), `variant` (`cases`: - `{label, type|null}[]`), `enum` (`labels`), `option` (`type`), - `result` (`ok|null`, `err|null`), `flags` (`labels`), `map` (`key`, - `value` — despecializes to `list` per the reference) -- Handles: `own` / `borrow` (`resource`: index into the plan's - `resourceTables`) + `{label, type|null}[]`), `enum` (`labels`), `option` (`type`), `result` + (`ok|null`, `err|null`), `flags` (`labels`), `map` (`key`, `value` — + despecializes to `list` per the reference) +- Handles: `own` / `borrow` (`resource`: index into the plan's `resourceTables`) - Async: `stream` / `future` (`element|null`), `error-context` Specialized forms are preserved (tuple/enum/option/result/flags/map are not pre-despecialized in the IR); `despecialize` is defined once, in the runtime, -mirroring `definitions.py`. Labels remain strings (interning is a -measured-need optimization). +mirroring `definitions.py`. Labels remain strings. -`FuncType` is `{ params: {label, type}[], results: ValType[], async?: bool }`. +Wire function declarations are +`{ kind: "func", params: {label, type}[], results: ValType[], async: bool }`. Pinned wire↔memory divergences (the plan loader maps): wire `result.err` ↔ -types.ts `result.error`; wire `FuncType.params` are labeled -`{label, type}[]` while types.ts drops names — names live on the wire and in -bindgen, not in the interpreter's hot path. +types.ts `result.error`; wire `FuncType.params` are labeled `{label, type}[]` +while types.ts drops names — names live on the wire and in bindgen, not in the +interpreter's hot path. ## Canonical options Per lifted/lowered function, referencing plan tables by index (see -plan-format.md): `stringEncoding` (`utf8` | `utf16` | `latin1+utf16`), -`memory?`, `realloc?`, `postReturn?`, `callback?`, `async`, `cancellable`, -and the expected flat `coreType` (`{params, results}` of `i32|i64|f32|f64`). -This mirrors `wasmtime_environ::component::CanonicalOptions` minus -runtime-irrelevant fields; `data_model` is fixed to linear memory (the GC -data model is rejected by the shim). +plan-format.md): `stringEncoding` (`utf8` | `utf16` | `latin1+utf16`), nullable +`memory`, `realloc`, `postReturn`, `callback`, plus `async`, `cancellable`, and +the expected flat `coreType` (`{params, results}` of `i32|i64|f32|f64`). This +mirrors `wasmtime_environ::component::CanonicalOptions` minus runtime-irrelevant +fields; `data_model` is fixed to linear memory (the GC data model is rejected by +the shim). ## Flattening The plan does **not** precompute flat lane lists. Executors compute flattening from `ValType` via the shared rules in `runtime/src/cabi/flatten.ts`, which is tested against fixtures generated from `definitions.py` (`flatten_functype`, -MAX_FLAT_PARAMS=16, MAX_FLAT_RESULTS=1, async variants with their own -limits, spill-to-memory rules). Rationale: one implementation of the trickiest -rules, differentially anchored to the executable spec; smaller plans; less -shim logic. The consistency check between computed flattening and the -options' `coreType` is an instantiate-time assertion — validated across the -whole fixture corpus. (Precomputed lanes can be added later as a pure -optimization without changing this contract's semantics.) +MAX_FLAT_PARAMS=16, MAX_FLAT_RESULTS=1, async variants with their own limits, +spill-to-memory rules). Rationale: one implementation of the trickiest rules, +differentially anchored to the executable spec; smaller plans; less shim logic. +The consistency check between computed flattening and the options' `coreType` is +an instantiate-time assertion — validated across the fixture corpus. ## Host value shapes -Host-facing value conventions are `contracts/embedder-api.md`'s territory -(implemented by the bindgen-generated layer). The raw executor boundary -mirrors `definitions.py`'s value *semantics*; its *representation* is this -implementation's to choose, and diverges deliberately where measurement -justifies it (docs/architecture.md §1 sanctions exactly this — "parity -means functional parity, not behavioral identity", and lists JS-native -host value shapes among the divergences). The shapes are: variant as -`{kind: label, value: payload}`, enum as `{kind: label, value: null}`, -option as `{kind: "none", value: null} / {kind: "some", value: v}`, result -error kind `"error"`, tuple as despecialized record. It is an internal -surface with no stability promise. - -Integer lanes wrap mod 2⁶⁴ at the raw boundary, matching -definitions.py's `% 2**64`; host-side range *asserts* (host-precondition -errors, not traps) exist only on the scalar `storeInt` path. NaN handling, -lane widening/padding (i64 lanes as `bigint`, `0n` padding), and -latin1(windows-1252) details follow the decisions recorded in -`runtime/README.md` and docs/architecture.md §7. - -The variant family carries its case in a `kind` property rather than as the -object's sole key — the one place the representation departs from the -reference's dicts. `definitions.py` uses a single-key mapping; that form -cost a computed-key literal here (a distinct hidden class per case label, -so every variant-reading site went megamorphic) plus an `Object.keys()` -allocation at each end to read one key. `value` is always present, `null` -for a payload-free case: omitting it to match the host layer exactly was -measured slower, because the producer site then emits two shapes instead -of one. Both findings are measured on `bench/boundary`'s compound-element -lanes; see issue #261 and the PR that landed this for the numbers and -method. The property names deliberately match `contracts/embedder-api.md`'s -host variant shape, but **the two are not interchangeable** — that -document's "Implementation strategy" enumerates every way they still -differ. +The runtime facade implements [embedder-api.md](embedder-api.md); bindgen emits +types and typed wrappers for that facade. The raw executor preserves the +reference's value semantics in these internal shapes: variant as +`{kind: label, value: payload}`, enum as `{kind: label, value: null}`, option as +`{kind: "none", value: null} / {kind: "some", value: v}`, result error kind +`"error"`, tuple as despecialized record. It is an internal surface with no +stability promise. + +Core integer lanes are normalized to unsigned values of their width before +lifting; narrower component integers wrap according to `definitions.py`. i64 +values use `bigint`, including `0n` padding. The public facade validates host +values before lowering. At the raw internal boundary, scalar `storeInt` asserts +range while bulk numeric stores wrap; neither is a substitute for facade +validation. NaNs are canonicalized. Latin-1 decoding uses the ISO-8859-1 byte +mapping, not WHATWG `TextDecoder`'s Windows-1252 alias. See +[architecture §7](../docs/architecture.md#7-canonical-abi-decisions). + +Fixed `kind`/`value` properties avoid per-case object shapes and key +enumeration. `value` is always present, with `null` for a payloadless case. +Despite shared property names, raw and facade values are not interchangeable; +see [the adaptation table](embedder-api.md#implementation-strategy). ## Trap discipline -Lift/lower failures raise the runtime's `ComponentTrap` (not arbitrary -`Error`s), with the trap conditions of `definitions.py` (`trap_if`) as the -authority. Executors must produce the same trap/no-trap verdict for the same -inputs — this is part of the differential-testing contract. +Guest lift/lower violations raise `Trap`, following the reference's `trap_if` +conditions. Internal/precondition assertions use `AssertionError`; invalid +public host values are rejected by facade validation. These failure classes must +not be conflated with unsupported capabilities or translator errors. ## Executor contract -Interpreter (v1) and generated-JS executors consume this IR unchanged; -the differential test harness runs both over the same fixture corpus -(`runtime/tests/fixtures/`, regenerable from the Python reference). Any IR -extension must land with fixtures. +The interpreter is checked against fixtures generated from the Python reference +(`runtime/tests/fixtures/`). A future generated-JS executor must consume the +same IR and agree on values and trap conditions. Any IR extension must land with +fixtures. ## Resource-type identity -The shim emits `resource` indices into the plan's `resourceTables`; the -runtime builds identity tokens (`ResourceTypeInfo`) at plan-load time. -**Tokens must be fresh per instantiation** (the executor re-runs plan -loading per instantiate), so resource-type identity never leaks across -instances. +The shim emits `resource` indices into the plan's `resourceTables`; the runtime +builds identity tokens (`ResourceTypeInfo`) at plan-load time. **Tokens must be +fresh per instantiation** (the executor re-runs plan loading per instantiate), +so resource-type identity never leaks across instances. Within one +instantiation, concrete tables naming the same resource share a token; table +indices are aliases, not distinct types. diff --git a/contracts/digest.md b/contracts/digest.md index d30cb60..7f7bc88 100644 --- a/contracts/digest.md +++ b/contracts/digest.md @@ -1,20 +1,17 @@ # Contract: Canonical World Digest (`cewd:1`) -The digest is the skew-protection handshake of docs/architecture.md §9: bindings generated -from WIT embed an expected digest; the generated typed `instantiate` recomputes -it from the loaded plan and fails fast on mismatch, -before any guest code runs (contracts/embedder-api.md §"Module wiring and -instantiation" — the runtime's untyped `instantiate` names no world, so it -does not verify). A digest match must imply ABI-shape compatibility for -positional calling. +Bindings embed a digest of their expected WIT world. Their typed `instantiate` +wrapper recomputes it from the plan and rejects mismatches before running guest +code. Untyped runtime instantiation does not check an expected world. A digest +match must imply ABI-shape compatibility for positional calling. It does not +authenticate a component or prove its behavior. -Normalization version: **`cewd:1`**. The version tag is folded into the -hashed document, so any future incompatible renormalization cannot collide. +Normalization version: **`cewd:1`**. The version tag is folded into the hashed +document, so any future incompatible renormalization cannot collide. ## Reference implementations (normative, kept in lockstep) - Rust, from `wit_parser::Resolve` + world: `crates/bindgen/src/digest.rs` - (module docs carry the full rule-by-rule spec) - TypeScript, from a loaded plan's types/imports/exports: `runtime/src/digest/digest.ts` @@ -25,35 +22,36 @@ implementations plus fixtures in the same commit. ## Normalization rules (summary) -- **Order-independent**: import/export lists and nested interface-instance - item lists are sorted by name — the only order normalized away. +- **Order-independent**: import/export lists and nested interface-instance item + lists are sorted by name — the only order normalized away. - **Order-preserved (ABI-relevant)**: record fields, tuple elements, - variant/enum case order (discriminants), flags label order (bit - positions), function parameter order, fixed-list lengths. + variant/enum case order (discriminants), flags label order (bit positions), + function parameter order, fixed-list lengths. - **Resource identity** by qualified name, not table index. -- **Excluded**: function parameter *labels* (calling is positional; renames - are not ABI changes — two worlds differing only in param names digest - equal, by design), docs/stability gates/spans, plan `features`, - `importedResources`, table ordering, `producer` metadata. +- **Excluded**: function parameter _labels_ (calling is positional; renames are + not ABI changes — two worlds differing only in param names digest equal, by + design), docs/stability gates/spans, plan `features`, `importedResources`, + table ordering, `producer` metadata. - **Included**: package `@version` in qualified interface names. -- Only functions and resources contribute as export/import *items*; named +- Only functions and resources contribute as export/import _items_; named non-resource types are structural and appear where referenced. -- Hash: sha256 over a canonical JSON document prefixed `{"cewd":1,…}`; - name sorting is byte-wise/UTF-16-code-unit (equivalent for ASCII WIT - identifiers — comment pinned on both sides). +- Hash: sha256 over a canonical JSON document prefixed `{"cewd":1,…}`; name + sorting is byte-wise/UTF-16-code-unit (equivalent for ASCII WIT identifiers — + comment pinned on both sides). ## Guards (fail loudly, never guess) -`DigestError` is thrown — rather than risking a wrong-but-matching digest — -when the plan carries **imported resources**, or **≥2 named resources** -whose own/borrow table indices cannot be soundly attributed, or own/borrow -references with **zero** named resources. All three paths are test-pinned. -Lifting these requires a plan-format extension carrying an explicit -resource-alias map (open item, tracked in plan-format.md). +The current plan-side implementation throws `DigestError` for imported +resources, unresolved own/borrow references, and ambiguous multi-resource table +attribution. It maps all table references to the sole name in a single-resource +world. Multi-resource worlds use direct table-to-name matches and reject +unresolved aliases. These are limits of the digest implementation, not of +resource execution; the executor separately resolves concrete resource-table +aliases. ## Known divergence (loud, not silent) -Interface *imports* are digested nested on the Rust side and flattened on -the TS side — guaranteed handshake mismatch (safe failure direction) for -worlds with interface imports; no current fixture has any. Resolve both -sides together when the imports corpus lands. +Interface imports are nested on the Rust side and flattened on the TS side. Such +worlds can fail the digest handshake despite being executable by the untyped +runtime. Correcting this requires coordinated Rust/TS normalization and +cross-language fixtures, not bypassing the mismatch check. diff --git a/contracts/embedder-api.md b/contracts/embedder-api.md index f68f85e..540e55a 100644 --- a/contracts/embedder-api.md +++ b/contracts/embedder-api.md @@ -1,140 +1,123 @@ # Embedder API conventions (host-facing) -The normative contract for host-facing value shapes and behavior. The -runtime's *raw* boundary (`instance.exports`, `HostImports`) keeps the -`definitions.py` interpreter shapes as an internal surface with no -stability promise; this document specifies the bindgen-generated ergonomic -layer over it. - -* [Principles](#principles) -* [Naming and casing](#naming-and-casing) -* [Version canonicalization](#version-canonicalization) -* [Value mapping](#value-mapping-normative) -* [Error model](#error-model) -* [Functions and async](#functions-and-async) -* [Resources](#resources) -* [Streams and futures](#streams-and-futures) -* [Module wiring and instantiation](#module-wiring-and-instantiation) -* [Module identity and @polyengine/protocol](#module-identity-and-polyengineprotocol) -* [Realm boundaries and structured-clone-safe forms](#realm-boundaries-and-structured-clone-safe-forms) -* [The host-ABI surface and its version](#the-host-abi-surface-and-its-version) -* [Implementation strategy](#implementation-strategy) -* [The WASI parking kernel](#the-wasi-parking-kernel) - -Authorities: the Component Model [Explainer] and `definitions.py` -(tie-breaker), the draft JS-API (component-model PR #686), and wasmtime as -corroborating evidence. Issue numbers (`#N`) name the polyengine tracker -entry where a rule was decided. +The normative contract for host-facing value shapes and behavior. The runtime's +raw executor boundary (`instance.handle.exports`, `HostImports`) keeps the +`definitions.py` interpreter shapes as an internal surface with no stability +promise. This document specifies the runtime facade; bindgen emits types and +typed wrappers for it. Public `instance.exports` is facade-shaped. + +- [Principles](#principles) +- [Naming and casing](#naming-and-casing) +- [Version canonicalization](#version-canonicalization) +- [Value mapping](#value-mapping-normative) +- [Error model](#error-model) +- [Functions and async](#functions-and-async) +- [Resources](#resources) +- [Streams and futures](#streams-and-futures) +- [Module wiring and instantiation](#module-wiring-and-instantiation) +- [Module identity and @polyengine/protocol](#module-identity-and-polyengineprotocol) +- [Realm boundaries and structured-clone-safe forms](#realm-boundaries-and-structured-clone-safe-forms) +- [The host-ABI surface and its version](#the-host-abi-surface-and-its-version) +- [Implementation strategy](#implementation-strategy) +- [The WASI parking kernel](#the-wasi-parking-kernel) + +Authorities: the Component Model [Explainer] and `definitions.py` (tie-breaker), +the draft JS-API (component-model PR #686), and wasmtime as corroborating +evidence. Issue numbers (`#N`) name the polyengine tracker entry where a rule +was decided. [Explainer]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md - ## Principles -1. **Fresh design; jco compatibility is a non-goal** (docs/architecture.md - §2). Convergence with jco is by merit (camelCase, enum strings); - where the draft JS-API points elsewhere we follow it (`{kind, value}` - variants, not `{tag, val}`). -2. **Footguns are design defects.** Each convention is judged against the - defensive code host modules had to write under jco. -3. **One way to do each thing.** No dual error channels or alternative - spellings; liberal *acceptance* only where the TS type still names one - canonical form. -4. **TS-first.** Every shape is a precise TypeScript type bindgen can emit. -5. **WASI must come out natural.** The `wasi/` package is the executable - check against p2/p3 idioms. -6. **Async is the point.** Exports are Promise-shaped; async imports are - plain async functions. - +1. **Explicit TypeScript shapes.** Data has one canonical representation; + invalid host values are rejected before crossing the boundary. +2. **Separate results from faults.** WIT error results use `ComponentException`; + traps and producer failures are not error values. +3. **Explicit ownership.** Handle transfers, borrows, cancellation, and disposal + have defined lifetimes and failure behavior. +4. **Async-first exports.** Exports are Promise-shaped, with the exceptions and + synchronous adapter described below. Imports follow their WIT type. +5. **No jco compatibility layer.** Conventions serve the Component Model and JS + hosts; WASI providers and consumer suites exercise them. ## Naming and casing -| WIT construct | JS/TS | -|---|---| -| function, method, static, record field, flag, parameter (docs only; calls are positional) | camelCase (`get-resolution` → `getResolution`) | -| resource | PascalCase class (`tcp-socket` → `TcpSocket`) | -| enum value, variant/result case name (the `kind` value) | kebab-case **verbatim** string literal — data, not an identifier | -| interface key in the imports/exports record | fully-qualified WIT id verbatim, version included: `wasi:clocks/monotonic-clock@0.3.0` | -| world-level (bare) import/export | camelCase at the record's top level | - +| WIT construct | JS/TS | +| ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| function, method, static, record field, flag, parameter (docs only; calls are positional) | camelCase (`get-resolution` → `getResolution`) | +| resource | PascalCase class (`tcp-socket` → `TcpSocket`) | +| enum value, variant/result case name (the `kind` value) | kebab-case **verbatim** string literal — data, not an identifier | +| interface key in the imports/exports record | fully-qualified WIT id verbatim, version included: `wasi:clocks/monotonic-clock@0.3.0` | +| world-level (bare) import/export | camelCase at the record's top level | ## Version canonicalization -A version's **track key** is the spec's `canonversion` (Explainer -§"Canonical interface names", 🔗; wasmtime `alternate_lookup_key`): `@1` -for `1.2.3`, `@0.2` for `0.2.6`, none for `0.0.x` and prereleases. +A version's **track key** follows `canonversion` (Explainer §"Canonical +interface names"; wasmtime `alternate_lookup_key`): `@1` for `1.2.3`, `@0.2` for +`0.2.6`, none for `0.0.x` and prereleases. -**Resolution** (normative for `instantiate` and the wasi package): an -import name matches (1) an exact provided key, else (2) the provider -holding its track, where a track is held by the **highest-versioned** key -registered on it (wasmtime's linker rule). Structural type-checking of -the resolved instance supplies the safety. +**Resolution** (normative for `instantiate` and the wasi package): an import +name matches (1) an exact provided key, else (2) the provider holding its track, +where a track is held by the **highest-versioned** key registered on it +(wasmtime's linker rule). This selects the provider; the component's declared +types drive leaf lookup and value conversion. It does not inspect a JS +implementation's TypeScript signature. -**Registration**: providers register full-versioned keys (the track -alternate is derived) or the **track key itself** (`…@0.2`) as an -explicitly canonical provider; registering both on one track is refused. -Unversioned ids are legal exact-match keys but never serve a versioned -import, nor vice versa. *Folding* — an unversioned key as a cross-track -wildcard — is banned; helpers may expand a wildcard over interface -*names* within one track only. - -Divergent drafts sharing a track (two `@0.3.0` snapshots with different -function sets) are served by one **union** provider; per-leaf structural -resolution selects what each component imports. - -*Forward note.* When the wasmtime pin passes 🔗, the plan gains an -optional `versionSuffix` on import/export entries and resolution becomes -string equality; `semver::parse("0.2")` fails, so canonical names never -generate alternates and the mechanisms compose. +**Registration**: providers register full-versioned keys (the track alternate is +derived) or the **track key itself** (`…@0.2`) as an explicitly canonical +provider; registering both on one track is refused. Unversioned ids are legal +exact-match keys but never serve a versioned import, nor vice versa. _Folding_ — +an unversioned key as a cross-track wildcard — is banned; helpers may expand a +wildcard over interface _names_ within one track only. +Divergent drafts sharing a track (two `@0.3.0` snapshots with different function +sets) are served by one **union** provider; per-leaf structural resolution +selects what each component imports. ## Value mapping (normative) -| Component type | TS | | -|---|---|---| -| `bool` | `boolean` | | -| `u8 s8 u16 s16 u32 s32 f32 f64` | `number` | range-checked at lower | -| `u64 s64` | `bigint` | range-checked at lower | -| `char` | `string` (one code point) | validated at lower | -| `string` | `string` | lower applies USVString replacement (docs/architecture.md §7) | -| `list` | `Uint8Array` | always a copy, never a view of guest memory | -| `list`, T ≠ u8 | `T[]` | no typed-array widening | -| `tuple` | `[A, B, …]` | | -| `record` | plain object, camelCase fields | option-typed fields are optional properties: lift emits **absent** for none; lower accepts absent or `undefined` | -| `enum` | union of kebab-case string literals | | -| `variant` | `{ kind: "case" } \| { kind: "case", value: T }` | `value` **absent** for payloadless cases | -| `option` | `T \| undefined`; nested options box | see below | -| `result` as a value (nested, or parameter position) | `{ kind: "ok", value: T } \| { kind: "err", value: E }` | `value` absent for empty sides | -| `result` as a function result | return `T` / throw `ComponentException` | empty sides: `undefined` / `payload === undefined` | -| `map` | `[K, V][]` (its despecialization) | | -| `flags` | object of camelCase booleans | lift: every flag present; lower: absent = `false` | -| `own`, `borrow` | the resource class instance | [Resources](#resources) | -| `stream`, `future`, `error-context` | `Stream`, `Future`, `ErrorContext` | [Streams and futures](#streams-and-futures) | - -The discriminant is `kind`, the payload `value`, matching the draft -JS-API's variant dictionary (`case` is a reserved word). A discriminant -property, rather than a single-key `{ [case]: value }` object, gives -compiler-checked exhaustiveness (`switch (v.kind)`), one uniform -payloadless shape, monomorphic access, and keeps case names data. Bindgen -may emit per-case constructor helpers (`Message.binary(bytes)`) as a -nicety. +| Component type | TS | | +| --------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| `bool` | `boolean` | | +| `u8 s8 u16 s16 u32 s32 f32 f64` | `number` | range-checked at lower | +| `u64 s64` | `bigint` | range-checked at lower | +| `char` | `string` (one code point) | validated at lower | +| `string` | `string` | lower applies USVString replacement (docs/architecture.md §7) | +| `list` | `Uint8Array` | always a copy, never a view of guest memory | +| `list`, T ≠ u8 | `T[]` | no typed-array widening | +| `tuple` | `[A, B, …]` | | +| `record` | plain object, camelCase fields | option-typed fields are optional properties: lift emits **absent** for none; lower accepts absent or `undefined` | +| `enum` | union of kebab-case string literals | | +| `variant` | `{ kind: "case" } \| { kind: "case", value: T }` | `value` **absent** for payloadless cases | +| `option` | `T \| undefined`; nested options box | see below | +| `result` as a value (nested, or parameter position) | `{ kind: "ok", value: T } \| { kind: "err", value: E }` | `value` absent for empty sides | +| `result` as a function result | return `T` / throw `ComponentException` | empty sides: `undefined` / `payload === undefined` | +| `map` | `[K, V][]` (its despecialization) | | +| `flags` | object of camelCase booleans | lift: every flag present; lower: absent = `false` | +| `own`, `borrow` | the resource class instance | [Resources](#resources) | +| `stream`, `future`, `error-context` | `Stream`, `Future`, `ErrorContext` | [Streams and futures](#streams-and-futures) | + +The discriminant is `kind` and the payload is `value`. This supports exhaustive +`switch (v.kind)` checks and keeps case names as data rather than property +names. **Option rule.** The outermost option maps to `T | undefined`; an option -*directly inside* another option uses the variant family, so boxing is -exactly as deep as the ambiguity. `option>`: +_directly inside_ another option uses the variant family, so boxing is exactly +as deep as the ambiguity. `option>`: + ```ts undefined // none { kind: "none" } // some(none) { kind: "some", value: 7 } // some(some(7)) ``` -**Example.** `result, own>, error>` as a -function result resolves to `[Counter, Counter]` (ownership transferred) -or rejects with a `ComponentException` whose `payload` is the `error` -value (`{ kind: "timed-out" }`); nested in a `list`, each element is +**Example.** `result, own>, error>` as a function +result resolves to `[Counter, Counter]` (ownership transferred) or rejects with +a `ComponentException` whose `payload` is the `error` value +(`{ kind: "timed-out" }`); nested in a `list`, each element is `{ kind: "ok", value: [Counter, Counter] } | { kind: "err", value: … }`. - ## Error model ```ts @@ -149,246 +132,223 @@ class PeerTrappedError extends Error { // a stream/future op whose peer instanc } ``` -- A guest export with `result` resolves `T` and rejects (throws, - on sync paths) `ComponentException`. `Trap` rejections are - distinguishable by class. -- A host import with `result` returns `T` and `throw`s - `new ComponentException(payload)` for err. The draft JS-API's - `WebAssembly.ComponentError` has this shape; renaming to follow it is a - deferred breaking event (#115). -- **An unbranded throw from a host import is a host bug and traps**, with - a message naming the import — never a guest-visible err. Only +- A guest export with `result` resolves `T` and rejects (throws, on sync + paths) `ComponentException`. `Trap` rejections are distinguishable by + class. +- A host import with `result` returns `T` and throws + `new ComponentException(payload)` for err. +- **An unbranded throw from a host import is a host bug and traps**, with a + message naming the import — never a guest-visible err. Only `ComponentException` crosses as an err value, so host modules need no - defensive wrappers. (Deliberate divergence from the draft, which - converts any thrown value to `E`.) -- Host code never swallows `Trap`; traps poison the instance - (docs/architecture.md §7). -- **`Trap.message` is diagnostic, not API.** Match the brand, never text. - Raw core traps carry the engine's own wording behind a `guest trapped:` - prefix, unnormalized across engines (the conformance harness reconciles - wording: `TRAP_MESSAGE_EQUIVALENTS`, harness/src/runner.ts). - Runtime-authored traps have stable wording; the same rule applies. + defensive wrappers. (Deliberate divergence from the draft, which converts any + thrown value to `E`.) +- A trap escaping a guest activation poisons its instance under the runtime's + policy (docs/architecture.md §6). Catching its host-side rejection does not + restore that instance. +- **`Trap.message` is diagnostic, not API.** Match the brand, never text. Raw + core traps carry the engine's own wording behind a `guest trapped:` prefix, + unnormalized across engines (the conformance harness reconciles wording: + `TRAP_MESSAGE_EQUIVALENTS`, harness/src/runner.ts). Runtime-authored traps + have stable wording; the same rule applies. - Results nested in values never throw; they are `{ kind, value }` data. -- **Recognition is by brand, not class.** Each class carries a - process-global brand symbol read by the runtime's checks. Same-copy - `instanceof` works; `@polyengine/protocol` exports the predicates - (`isComponentException`, `isTrap`, `isPeerTrappedError`, …) as the - multi-copy-robust form ([Module identity](#module-identity-and-polyengineprotocol)). - +- **Recognition is by brand, not class.** Each class carries a process-global + brand symbol read by the runtime's checks. Same-copy `instanceof` works; + `@polyengine/protocol` exports the predicates (`isComponentException`, + `isTrap`, `isPeerTrappedError`, …) as the multi-copy-robust form + ([Module identity](#module-identity-and-polyengineprotocol)). ## Functions and async **Exports are uniformly Promise-shaped**: bindgen types every export as -returning `Promise`, sync-typed or not (docs/architecture.md §1). -Exactly two exceptions: resource constructors (synchronous) and -`future`-typed results (eager handles). A synchronous *view* exists as -the per-use adapter `sync()`; WIT getters/setters are pre-ruled to ride -it as accessors. - -**Imports match their WIT type**: an `async func` import may be an async -JS function or return synchronously; a sync `func` import returns `T` -synchronously. Params are positional (names are excluded from the world -digest, contracts/digest.md). Interface members are invoked with their -containing object as receiver, so a class instance is a valid interface -provider; world-level bare imports are called unbound. - -**Between-calls liveness**: guest progress does not require an in-flight -export call. A host import that settles while nothing is being driven is -serviced then (the JS analogue of dwelling in wasmtime's -`run_concurrent`), so guest keep-alive tickers over `wasi:clocks` are -self-driving. Bounds: an operation waiting on the *embedder's* half of a -host stream/future hangs until the embedder acts (never a trap), and a -settlement-time failure surfaces on the next call into the instance. - -**An async-typed export's Promise may stay pending indefinitely** (#292). -When its task parks with no ready thread and no host call outstanding — -the long-poll shape: `next: async func() -> event` woken by a later -`push` — the Promise stays pending; it settles when a later call (any -export, a host stream/future operation) runs the task to completion, and -rejects with the poisoning cause if the instance is poisoned first. This -is definitions.py `canon_lift`, whose trapping driving loop runs only for -sync-typed exports (line 2189), and wasmtime `call_concurrent` under -`run_concurrent`; the embedder's event loop is always dwelling, so the -blocking `call_async` shape (trap on idle) has no JS analogue and is not -offered. An async guest that genuinely can never progress therefore -hangs rather than traps, as it does under `run_concurrent`. **Sync-typed -exports keep the spec's deadlock trap** in every mode. +returning `Promise`, sync-typed or not (docs/architecture.md §1). Exactly two +exceptions: resource constructors (synchronous) and `future`-typed results +(eager handles). A synchronous view exists as the per-use adapter `sync()`. WIT +getter/setter support is not implemented. + +**Imports match their WIT type**: an `async func` import may be an async JS +function or return synchronously; a sync `func` import returns `T` +synchronously. Params are positional (names are excluded from the world digest, +contracts/digest.md). Interface members are invoked with their containing object +as receiver, so a class instance is a valid interface provider; world-level bare +imports are called unbound. + +**Between-calls liveness:** a settlement pump services host imports even without +an in-flight export call. Work waiting on the embedder's half of a stream/future +remains pending until the embedder acts. Background faults are recorded for +pending operations and subsequent entry. + +**Idle async calls remain pending.** A long-poll export may need a later export +call or host operation to make progress. Its Promise resolves when the task +returns its value, not when all producer threads exit; poisoning first rejects +it with the recorded cause. An async task that can never progress therefore +remains pending rather than trapping. Sync-typed exports retain the reference's +deadlock trap. The conformance harness separately opts into trap-on-idle +behavior for its blocking `invoke` directive. ### Import marks Three marks, each a `Symbol.for` brand defined in and imported from `@polyengine/protocol`, each with two spellings — the direct call -(`f: suspending(fn)`, the only form in record literals) and a stage-3 -method decorator (`@suspending` on instance or static methods, refusing -non-method positions and the legacy `experimentalDecorators` convention -at class-definition time). Constructors are never markable. On -host-resource classes the **prototype** is the per-declaration authority -for instance methods (read at wrap time; instance-level overrides change -the body, never the mark); statics carry the mark on the function. Marks -are independent and may combine. - -| mark | effect | -|---|---| -| `suspending(fn)` | a sync-typed import may return a Promise, parking the calling wasm frame | -| `deferCancel(fn)` | guest cancellation never discards the import's result | -| `abortable(fn)` | the import receives a per-call `AbortSignal`, aborted on discard | +(`f: suspending(fn)`, the only form in record literals) and a stage-3 method +decorator (`@suspending` on instance or static methods, refusing non-method +positions and the legacy `experimentalDecorators` convention at class-definition +time). Constructors are never markable. On host-resource classes the +**prototype** is the per-declaration authority for instance methods (read at +wrap time; instance-level overrides change the body, never the mark); statics +carry the mark on the function. Marks are independent and may combine. + +| mark | effect | +| ----------------- | ------------------------------------------------------------------------ | +| `suspending(fn)` | a sync-typed import may return a Promise, parking the calling wasm frame | +| `deferCancel(fn)` | guest cancellation never discards the import's result | +| `abortable(fn)` | the import receives a per-call `AbortSignal`, aborted on discard | **`suspending()`**: only marked imports are handed to wasm as -`WebAssembly.Suspending`, so unmarked imports keep the plain convention -and sync-only components their zero-cost pin. A marked import selects -jspi mode without `jspi: true` (`jspi: false` still forces plain, where a -returned Promise is refused). Costs are visible: every call through a -marked import pays the engine's continuation hop even when it returns -synchronously (contracts/intrinsics.md pin (j)); a marked import reached -from a `start` function traps, even on a synchronous return (pin (c)); -on a non-JSPI engine a returned Promise is refused at the call site -(`NeedsJspi`), never degraded. The park is a plain sync-lower wait -(`canon_lower`, `definitions.py`). - -**Cancellation and discard** (#241). A guest may cancel an in-flight -async-typed import. A JS function has no cancellation channel, so the -runtime answers on its behalf, choosing the prompt-cancel host response +`WebAssembly.Suspending`, so unmarked imports keep the plain convention and +sync-only components their zero-cost pin. A marked import selects jspi mode +without `jspi: true` (`jspi: false` still forces plain, where a returned Promise +is refused). Costs are visible: every call through a marked import pays the +engine's continuation hop even when it returns synchronously +(`runtime/tests/jspi/fastpath_hop_test.ts`); a marked import reached from a +`start` function traps, even on a synchronous return; on a non-JSPI engine a +returned Promise is refused at the call site (`NeedsJspi`), never degraded. The +park is a plain sync-lower wait (`canon_lower`, `definitions.py`). + +**Cancellation and discard** (#241). A guest may cancel an in-flight async-typed +import. A JS function has no cancellation channel, so the runtime answers on its +behalf, choosing the prompt-cancel host response (`on_cancel = () => on_resolve(None)`): the subtask resolves `CANCELLED_BEFORE_RETURNED` immediately, both cancel forms return without -blocking, and the settlement is **discarded** — never lowered, never -reported, no longer guest-wakeable for deadlock detection. -The host operation is not interrupted; discard is about delivery, not -execution. - -- **`deferCancel()`** opts out: cancellation is accepted and ignored, the - async cancel form answers `BLOCKED`, the sync form parks under jspi - (`NeedsJspi` without it), and the guest observes `RETURNED` with the - real result. For imports with a commit point (a flush, a write). Inert - on sync-typed imports, which mint no subtask handle. -- **`abortable()`** appends a fresh `AbortSignal` after the WIT - parameters (`dial: abortable((addr, signal) => fetch(url, { signal }))`); - the signature changes unconditionally, the abort fires only on - discard by guest cancellation — on a microtask after the cancel - built-in returns (host listeners never run inside a guest activation), - so the guest sees `CANCELLED_BEFORE_RETURNED` first. Settlements the - abort provokes are discarded like any late settlement. Inert wherever - discard cannot happen (sync-typed, `deferCancel`, eager resolution). - Instance teardown does not abort in-flight calls. +blocking, and the settlement is **discarded** — never lowered, never reported, +no longer guest-wakeable for deadlock detection. The host operation is not +interrupted; discard is about delivery, not execution. + +- **`deferCancel()`** opts out: cancellation is accepted and ignored, the async + cancel form answers `BLOCKED`, the sync form parks under jspi (`NeedsJspi` + without it), and the guest observes `RETURNED` with the real result. For + imports with a commit point (a flush, a write). Inert on sync-typed imports, + which mint no subtask handle. +- **`abortable()`** appends a fresh `AbortSignal` after the WIT parameters + (`dial: abortable((addr, signal) => fetch(url, { signal }))`); the signature + changes unconditionally, the abort fires only on discard by guest cancellation + — on a microtask after the cancel built-in returns (host listeners never run + inside a guest activation), so the guest sees `CANCELLED_BEFORE_RETURNED` + first. Settlements the abort provokes are discarded like any late settlement. + Inert wherever discard cannot happen (sync-typed, `deferCancel`, eager + resolution). Instance teardown does not abort in-flight calls. ### `sync()` -Some host contexts cannot receive a Promise however promptly it resolves -(event handlers deciding `preventDefault()`, comparators, `Proxy` traps, -accessors: even a resolved Promise defers by a microtask). For a WIT-sync -export whose guest completes synchronously, `sync()` asks for the result -synchronously. It is an adapter applied per use, never a mode. - -`sync()` and `Sync` are exported from `@polyengine/runtime/embedder` -— application machinery, like `createStream`: only an instantiating -application holds export functions. A host module wanting a synchronous -guest callback is *handed* `sync(exports.f)` by the application and MUST -NOT import the runtime ([host-ABI surface](#the-host-abi-surface-and-its-version)). -Recognition is by brand (`polyengine.syncCallable/1`), so views work -across runtime copies. Dispatch by target: - -- `sync(fn)` on a lifted export function (plain, interface member, - resource static) → `(...args) => T`. -- `sync(instance)` on a guest-resource wrapper → a view whose members - call the synchronous forms with `instance` as receiver. `sync(method)` - on a bare prototype method throws `TypeError` naming this spelling. -- `sync(cls)` on a guest-resource class → a view of synchronous statics - (`new` the class itself; constructors are already synchronous). -- `sync(record)` on an exports or interface record → a view mapping every - member recursively; unbranded members pass through. +Some host contexts cannot receive a Promise however promptly it resolves (event +handlers deciding `preventDefault()`, comparators, `Proxy` traps, accessors: +even a resolved Promise defers by a microtask). For a WIT-sync export whose +guest completes synchronously, `sync()` asks for the result synchronously. It is +an adapter applied per use, never a mode. + +`sync()` and `Sync` are exported from `@polyengine/runtime/embedder` — +application machinery, like `createStream`: only an instantiating application +holds export functions. A host module wanting a synchronous guest callback is +_handed_ `sync(exports.f)` by the application and MUST NOT import the runtime +([host-ABI surface](#the-host-abi-surface-and-its-version)). Recognition is by +brand (`polyengine.syncCallable/1`), so views work across runtime copies. +Dispatch by target: + +- `sync(fn)` on a lifted export function (plain, interface member, resource + static) → `(...args) => T`. +- `sync(instance)` on a guest-resource wrapper → a view whose members call the + synchronous forms with `instance` as receiver. `sync(method)` on a bare + prototype method throws `TypeError` naming this spelling. +- `sync(cls)` on a guest-resource class → a view of synchronous statics (`new` + the class itself; constructors are already synchronous). +- `sync(record)` on an exports or interface record → a view mapping every member + recursively; unbranded members pass through. - Views are stable (repeated `sync(x)` returns the same object). - `sync()` on an async-typed export, or on anything unbranded, throws `TypeError` at adapter time. -**Call semantics.** A synchronous `canon_lift` through a plain -(non-`promising`) entry. `result` throws `ComponentException` -synchronously; handle-valued results return handles. +**Call semantics.** A synchronous `canon_lift` through a plain (non-`promising`) +entry. `result` throws `ComponentException` synchronously; +handle-valued results return handles. **Failure ladder** (ordered; 1–3 are non-poisoning): -1. Entry refusals shared with the Promise surface (reentrance, poisoned - instance) are thrown before entering. -2. *Hop-window contention* (jspi mode only): a `promising` entry settles - through a microtask hop, and the hop-quiescence gate defers - Promise-surface calls that would race a pending lift. A synchronous - call cannot defer, so it refuses: `SyncEntryBusy` - (`e.name === "SyncEntryBusy"`), transient — retry or use the Promise - surface. The constructor entry shares this refusal. +1. Poisoned-instance refusal is shared with the Promise surface and occurs + before entering. Reentrance into a live instance is allowed. +2. _Hop-window contention_ (jspi mode only): a `promising` entry settles through + a microtask hop, and the hop-quiescence gate defers Promise-surface calls + that would race a pending lift. A synchronous call cannot defer, so it + refuses: `SyncEntryBusy` (`e.name === "SyncEntryBusy"`), transient — retry or + use the Promise surface. The constructor entry shares this refusal. 3. A blocking built-in through the plain entry: `NeedsJspi`. -4. Genuine suspension — a `Suspending`-wrapped import reached from the - unwrapped frame — fails as a trap and poisons the entered instances. - A component with no `suspending()` imports and no async built-ins - never reaches this arm. - -**Cost.** Plain mode: the lifted function already completes inside the -entered bracket; `sync()` skips the Promise wrapper. Jspi mode: every -sync-typed export carries a second plain-entered entry (`SYNC_ENTRY`; the -constructor entry is one instance), deliberately outside the bridge -invariant (entries wrapped iff imports wrapped) — safe because a -synchronously-completing activation never reaches the Suspending seam. -Unused entries cost nothing per call. +4. Genuine suspension — a `Suspending`-wrapped import reached from the unwrapped + frame — fails as a trap and poisons the entered instances. A component with + no `suspending()` imports and no async built-ins never reaches this arm. +**Cost.** In plain mode, `sync()` skips the Promise facade. In JSPI mode, +sync-typed exports have a plain entry (`SYNC_ENTRY`) as well as the normal +entry. That plain entry cannot cross a `Suspending` import, even if the import +would return immediately; the failure ladder above applies. ## Resources -**A resource is a class instance on both sides.** Identity mapping and -name mangling are runtime obligations: no bare reps, no identity tables, -no hand-transcribed `[method]…` keys. +**A resource is a class instance on both sides.** Identity mapping and name +mangling are runtime obligations: no bare reps, no identity tables, no +hand-transcribed `[method]…` keys. -**Guest-implemented** (host holds handles): bindgen emits a class per -resource — the constructor calls the guest constructor; camelCase methods -and statics; `[Symbol.dispose]()` and `drop()` drop the handle (TS -`using` works); a `FinalizationRegistry` backstop drops leaks -(docs/architecture.md §7). +**Guest-implemented** (host holds handles): the runtime builds a class per +resource, with declarations supplied by bindgen. Its constructor calls the guest +constructor; camelCase methods and statics; `[Symbol.dispose]()` and `drop()` +drop the handle (TS `using` works); a `FinalizationRegistry` backstop drops +leaks (docs/architecture.md §7). **Host-implemented** (guest holds handles): the host supplies a class implementing the bindgen interface (camelCase methods, statics as static members, the WIT constructor as the JS constructor). The runtime owns the -instance↔rep mapping; when the guest drops its last own handle the -runtime calls `instance[Symbol.dispose]?.()`. Method `self` is the -instance. - -Overlapping host-originated borrows retain the mapping until the last -borrowing call ends. A guest drop during that interval defers disposal -until the final borrow ends; the pending-drop instance cannot be passed -as own again. A deferred disposal error is reported by the last borrowing -call, after all its borrow mappings are released. An existing call failure -remains primary; results that cannot be delivered because cleanup failed -are released rather than abandoned. +instance↔rep mapping; when the guest drops its last own handle the runtime calls +`instance[Symbol.dispose]?.()`. Method `self` is the instance. + +Overlapping host-originated borrows retain the mapping until the last borrowing +call ends. A guest drop during that interval defers disposal until the final +borrow ends; the pending-drop instance cannot be passed as own again. A deferred +disposal error is reported by the last borrowing call, after all its borrow +mappings are released. An existing call failure remains primary; results that +cannot be delivered because cleanup failed are released rather than abandoned. **Constructors are synchronous** (a JS constructor cannot await). A guest -constructor that does not complete synchronously raises a named error -rather than half-constructing; its plain entry is one instance of -`SYNC_ENTRY` and shares its failure ladder. A generated async static -factory is the escape hatch, deferred until demanded. - -| WIT position | guest-implemented R | host-implemented R | -|---|---|---| -| host receives `own` | new instance; host owns it (drop/`using`) | the host's own instance; the guest's handle is gone; no dispose call | -| host receives `borrow` | valid only during the call (retention throws) | the host's own instance; scoping is guest-side bookkeeping | -| host passes `own` | wrapper invalidated (transferred) | instance registered; guest owns the handle | -| host passes `borrow` | wrapper stays valid | an unregistered instance gets a rep for the call's duration | +constructor that does not complete synchronously raises a named error rather +than half-constructing; its plain entry is one instance of `SYNC_ENTRY` and +shares its failure ladder. A generated async static factory is the escape hatch, +deferred until demanded. + +| WIT position | guest-implemented R | host-implemented R | +| ------------------------- | --------------------------------------------- | -------------------------------------------------------------------- | +| host receives `own` | new instance; host owns it (drop/`using`) | the host's own instance; the guest's handle is gone; no dispose call | +| host receives `borrow` | valid only during the call (retention throws) | the host's own instance; scoping is guest-side bookkeeping | +| host passes `own` | wrapper invalidated (transferred) | instance registered; guest owns the handle | +| host passes `borrow` | wrapper stays valid | an unregistered instance gets a rep for the call's duration | ### Pattern (non-normative): binding platform classes directly -When a WIT resource's shape matches a native class, pass the class itself -(the draft JS-API's import story, PR #686; #115): +When a WIT resource's shape matches a native class, pass the class itself (the +draft JS-API's import story, PR #686; #115): + ```ts const instance = await instantiate(artifacts, { "test:platform/web": { params: URLSearchParams, decoder: TextDecoder }, }); ``` -Dispatch is a per-call `self[camelCase(member)]` lookup, constructor args -flow to `new Class(...)`, and the value conventions are the natural JS -shapes (`Uint8Array` is a `BufferSource`; a record is an options bag). -Named types in the interface need no imports entry. Limits, each with a -one-line `class X extends Native` bridge (executable reference: + +Dispatch is a per-call `self[camelCase(member)]` lookup, constructor args flow +to `new Class(...)`, and the value conventions are the natural JS shapes +(`Uint8Array` is a `BufferSource`; a record is an options bag). Named types in +the interface need no imports entry. Limits, each with a one-line +`class X extends Native` bridge (executable reference: runtime/tests/embedder/platform_class_test.ts): -1. **Accessor-backed properties are not methods**: `size: func() -> u32` - against `URLSearchParams.prototype.size` traps per call ("no method - 'size'"). The wrap-time mark probe reads data properties only, so - binding is safe and a mark cannot ride an accessor. Bridge: a method - delegating to the property. +1. **Accessor-backed properties are not methods**: `size: func() -> u32` against + `URLSearchParams.prototype.size` traps per call ("no method 'size'"). The + wrap-time mark probe reads data properties only, so binding is safe and a + mark cannot ride an accessor. Bridge: a method delegating to the property. 2. **Platform absent is `null`; WIT none is `undefined`**: a `null` where `option` is expected takes the `some` branch and fails conversion (`TypeError` naming the import, not a trap). Bridge: `?? undefined`. @@ -398,32 +358,27 @@ runtime/tests/embedder/platform_class_test.ts): ### Getters and setters (pre-ruling — not yet implementable) -component-model#701 (approved, 📡) adds `[get]foo`/`[set]foo` at interface -level, `[method][get]r.foo`/`[method][set]r.foo` on instances, -`[static][get]r.foo`/`[static][set]r.foo` on types. Upstream validation: -getters take no parameters and return a value; setters take one and -return nothing or `result<_, error?>`; neither is `async`; every `[set]` -requires its `[get]`; type agreement is not required (WebIDL -`PutForwards`). Blocked on the toolchain chain to a pinned -`wasmtime-environ` carrying 📡 (#254). The ruling: - -- **Export side: real JS accessors, sync-required.** Bindgen emits - `get prop(): T` / `set prop(v)` on resource classes, as static - accessors, or on the exports record. They ride `SYNC_ENTRY` and its - failure ladder (a JS setter cannot express async completion; symmetric - semantics are ruled). A fallible setter throws `ComponentException` - synchronously. Divergent types map to asymmetric accessor types. Where - both accessor and `get-prop` method spellings collide, the accessor - wins and the method is dropped with a bindgen warning. -- **Import side: property get and assignment on the receiver.** - `[get]foo` reads `self[camelCase(foo)]` (or the interface object); - `[set]foo` assigns. This retires platform-class limit 1 for worlds that - declare accessors. Accessors are never `suspending()`-markable; a host - getter returning a Promise is refused as any unmarked sync import. -- **Until then**: the runtime refuses unknown bracket forms loudly at - instantiation, and the translator keeps the gate off. Digest impact: - none expected. - +Getter/setter forms from component-model#701 are not implemented in the facade +([#254](https://github.com/polymorph-components/polyengine/issues/254)). The +intended mapping below is a design decision, not a supported API. Forms include +`[get]foo`/`[set]foo`, instance members such as `[method][get]r.foo`, and static +members such as `[static][set]r.foo`. + +- **Planned exports: real JS accessors, sync-required.** Bindgen would emit + `get prop(): T` / `set prop(v)` on resource classes, as static accessors, or + on the exports record. They ride `SYNC_ENTRY` and its failure ladder (a JS + setter cannot express async completion; symmetric semantics are ruled). A + fallible setter throws `ComponentException` synchronously. Divergent types map + to asymmetric accessor types. Where both accessor and `get-prop` method + spellings collide, the accessor wins and the method is dropped with a bindgen + warning. +- **Planned imports: property get and assignment on the receiver.** `[get]foo` + reads `self[camelCase(foo)]` (or the interface object); `[set]foo` assigns. + This retires platform-class limit 1 for worlds that declare accessors. + Accessors are never `suspending()`-markable; a host getter returning a Promise + is refused as any unmarked sync import. +- **Current behavior:** unknown bracket forms are refused at facade + instantiation, rather than bound as ordinary function names. ## Streams and futures @@ -452,297 +407,289 @@ function createStream(): { stream: Stream, writer: StreamWriter }; // ``` The interfaces, `Chunk`, `StreamSource`/`FutureSource`, and the -predicates `isStream`/`isStreamWriter`/`isFuture`/`isErrorContext` are -exported from `@polyengine/protocol`; the runtime's concrete classes -implement them and are not exported. - -**Lifting.** `stream`/`future` arrive as `Stream`/`Future`. -An export whose result is `future` returns `Future` **directly**, -not `Promise>`: promise resolution adopts thenables, so a -Promise cannot resolve *to* a PromiseLike handle. `await exportFn()` -yields `T`; call without awaiting to hold the handle. Awaiting a future -whose write end dropped without a value rejects `DroppedError`. - -**Disposal is total and silent** (#182). `drop()`/`cancel()` never -throw, never return a promise, and never surface a failure of the -producing call. A future from an export call is *deferred* (its host end -materializes when the call completes), so a held handle may outlive a -failing producer; disposal discards the failure, which still surfaces to -anyone awaiting. The runtime attaches rejection handling at the handle so -no disposal or abandonment raises an unhandled rejection. - -**Lowering accepts natural JS producers**: for `stream` a -`ReadableStream`, `AsyncIterable`, finite array, or `Stream`; for -`future` a `Promise` or `Future`. A `Future` handle is -lowerable once its host end has materialized; a still-deferred handle is -refused loudly (a Promise has no such window). Bindgen owns the pumping: -arms auto-close on end/`DROPPED`; cross-store reuse is asserted. - -**An import whose result is `future` returns the future source.** A -thenable returned by the host (`Promise` or `Future`) IS the -future: the import completes immediately and the future settles on the -producer's schedule. It is not adopted as the call's completion — the -`wasi:sockets@0.3` `send: func(data: stream) -> future>` shape settles only after post-return guest writes, so -adoption would livelock. A rejected source promise is a producer failure -(site-named, on the consuming call), never a guest-visible err; fallible -payloads ride inside the future. Executable spec: +predicates `isStream`/`isStreamWriter`/`isFuture`/`isErrorContext` are exported +from `@polyengine/protocol`; the runtime's concrete classes implement them and +are not exported. + +**Lifting.** `stream`/`future` arrive as `Stream`/`Future`. An +export whose result is `future` returns `Future` **directly**, not +`Promise>`: promise resolution adopts thenables, so a Promise cannot +resolve _to_ a PromiseLike handle. `await exportFn()` yields `T`; call without +awaiting to hold the handle. Awaiting a future whose write end dropped without a +value rejects `DroppedError`. + +**Disposal is total and silent** (#182). `drop()`/`cancel()` never throw, never +return a promise, and never surface a failure of the producing call. A future +from an export call is _deferred_ (its host end materializes when the call +completes), so a held handle may outlive a failing producer; disposal discards +the failure, which still surfaces to anyone awaiting. The runtime attaches +rejection handling at the handle so no disposal or abandonment raises an +unhandled rejection. + +**Lowering accepts natural JS producers**: for `stream` a `ReadableStream`, +`AsyncIterable`, finite array, or `Stream`; for `future` a `Promise` or +`Future`. A `Future` handle is lowerable once its host end has +materialized; a still-deferred handle is refused loudly (a Promise has no such +window). The runtime facade owns pumping and closes activity retention on +end/drop. Cross-store reuse is refused. + +**An import whose result is `future` returns the future source.** A thenable +returned by the host (`Promise` or `Future`) IS the future: the import +completes immediately and the future settles on the producer's schedule. It is +not adopted as the call's completion — the `wasi:sockets@0.3` +`send: func(data: stream) -> future>` shape settles +only after post-return guest writes, so adoption would livelock. A rejected +source promise is a producer failure (site-named, on the consuming call), never +a guest-visible err; fallible payloads ride inside the future. Executable spec: examples/guests/future-import, runtime/tests/embedder/future_result_test.ts. -**Streams of resources** (`stream>`, the `listen` shape): a -producer yields class instances; each element lowers by the normal `own` -transfer. Obligations: -- **Un-taken elements are destroyed, never leaked** — reader dropped - mid-stream or peer trapped (`PeerTrappedError.progress` marks the - delivered prefix): destructors run at pump teardown. Top-level `own` - only; composite elements with nested owns are out of scope until a - consumer links one. -- **Producers are cancellable**: when the stream dies while the producer - is parked with no write in flight, the pump cancels it — a - `ReadableStream` via `reader.cancel()`, an (async-)iterable via its - optional `cancel(): void` (then drains the pending pull so a straggler - reaches the un-taken path). A source with no hook stays parked until - its next element (the embedder-negligence hang class). -- World-level host resources register under the resource's camelCase - name; their mangled leaves dispatch on that class. -Executable spec: examples/guests/resource-stream, -runtime/tests/embedder/resource_stream_test.ts. - -**Stream values survive round trips.** Lifting a stream/future the host -already handled — a host-created stream passed back, or a guest stream on -its second hop — is idempotent, yielding a handle over the same end. -Hence: host→guest→host pass-through works with the guest never reading -(payload moves host↔host without touching guest memory); a readable end -hops any number of times (each lower transfers it); host↔host rendezvous -is legal for every element type; a `createStream()` writer keeps feeding -the same stream across hops. - -**Deadlock-verdict suppression tracks host retention** (#162). While the -host retains a way to act (a retained end, a parked host operation, an -unfinished pump), a stalled guest is the embedder-may-act hang, never a -deadlock trap. Lowering a lifted handle back (the `identity` round trip) -ends retention and verdicts go live; a re-lift restores suppression; -every drop path releases it. `read` through a handle — or awaiting a -`Future` — already passed to a guest rejects `TypeError` naming the -transfer. `StreamWriter` is unaffected (it addresses the host-retained -writable end). Returning a *different* stream while keeping the original -is genuine retention. +**Streams of resources** (`stream>`, the `listen` shape): a producer +yields class instances; each element lowers by the normal `own` transfer. +Obligations: + +- **Untaken elements are destroyed.** Failed chunk conversion releases its + converted prefix; short writes, cancellation, and peer faults release the + undelivered tail. Cleanup attempts every element and preserves an existing + operation error; a standalone cleanup error is reported. This applies to pumps + and explicit writers for top-level `own` elements. Composite stream elements + containing nested owns are not covered by this cleanup policy. +- **Producers are cancellable**: when the stream dies while the producer is + parked with no write in flight, the pump cancels it — a `ReadableStream` via + `reader.cancel()`, an (async-)iterable via its optional `cancel(): void` (then + drains the pending pull so a straggler reaches the un-taken path). A source + with no hook stays parked until its next element. +- World-level host resources register under the resource's camelCase name; their + mangled leaves dispatch on that class. Executable spec: + examples/guests/resource-stream, + runtime/tests/embedder/resource_stream_test.ts. + +**Stream and future values survive round trips.** Lifting an end the host +already handled — a host-created stream passed back, or a guest stream on its +second hop — is idempotent, yielding a handle over the same end. Hence: +host→guest→host pass-through works with the guest never reading (payload moves +host↔host without touching guest memory); a readable end hops any number of +times (each lower transfers it); host↔host rendezvous is legal for every element +type; a `createStream()` writer keeps feeding the same stream across hops. + +**Deadlock-verdict suppression tracks host retention** (#162). While the host +retains a way to act (a retained end, a parked host operation, an unfinished +pump), a stalled guest is the embedder-may-act hang, never a deadlock trap. +Lowering a lifted handle back (the `identity` round trip) ends retention and +verdicts go live; a re-lift restores suppression; every drop path releases it. +`read` through a handle — or awaiting a `Future` — already passed to a guest +rejects `TypeError` naming the transfer. `StreamWriter` is unaffected (it +addresses the host-retained writable end). Returning a _different_ stream while +keeping the original is genuine retention. **u8 chunks are `Uint8Array` both ways**: a `Uint8Array` passed to -`write`/`writeAll` is already-lowered bytes, passed by reference -(borrowed until the promise settles) and copied once at the rendezvous; -reads hand back that copy. One copy host↔host, one per direction with a -guest peer. - -**Foreign-copy handles are refused, loudly.** A handle minted by another -runtime copy is recognized by brand at lowering and raises a named -cross-copy error listing both URLs — never pumped as a generic producer. -Remediation is by value: `.readable()`, `Promise.resolve(f)`. -Error-contexts are exempt: message-valued, a branded foreign one lowers -by minting a fresh local context. - -**Component faults are loud.** When the peer instance traps, its ends -retire: a parked host `read`/`write`/`writeAll`/await, and every later -operation, rejects `PeerTrappedError` (`cause` chains to the trap; -`progress` on writes). A fault is never a clean end-of-stream or bare -`DroppedError`; an operation that completed before the trap keeps its -result. A trapping host import drops the lifted arguments it abandoned, -so their peers settle truthfully. Only embedder negligence (lowering a -host end and never acting) hangs. - -**One in-flight operation per end, per direction**: a second parked -`write`, `read`, or future operation throws `TypeError` synchronously -(the host spelling of the `CopyEnd` busy trap). Reading while a write is -parked on the same stream is legal. +`write`/`writeAll` is already-lowered bytes, passed by reference (borrowed until +the promise settles) and copied once at the rendezvous; reads hand back that +copy. One copy host↔host, one per direction with a guest peer. + +**Foreign-copy handles are refused, loudly.** A handle minted by another runtime +copy is recognized by brand at lowering and raises a named cross-copy error +listing both URLs — never pumped as a generic producer. Remediation is by value: +`.readable()`, `Promise.resolve(f)`. Error-contexts are exempt: message-valued, +a branded foreign one lowers by minting a fresh local context. + +**Component faults are loud.** When the peer instance traps, its ends retire: a +parked host `read`/`write`/`writeAll`/await, and every later operation, rejects +`PeerTrappedError` (`cause` chains to the trap; `progress` on writes). A fault +is never a clean end-of-stream or bare `DroppedError`; an operation that +completed before the trap keeps its result. A trapping host import drops +abandoned top-level stream/future arguments; this cleanup does not traverse +compound arguments. Their peers can then settle rather than waiting on abandoned +arguments. Work still waiting on a live host producer remains pending until it +acts. + +**One in-flight operation per end, per direction**: a second parked `write`, +`read`, or future operation throws `TypeError` synchronously (the host spelling +of the `CopyEnd` busy trap). Reading while a write is parked on the same stream +is legal. **Dropping an unwritten future is abandonment, not DROPPED** (#90). `Future.drop()` on a lowered, never-written future never throws and is -idempotent; the guest's readable end observes a **trap at its rendezvous** -("the host dropped the writable end without writing a value") — the -host-side spelling of the guest's own drop-before-write trap — never a -DROPPED event or a hang. An unlowered future just releases state. -Producer rejections keep the loud site-named fault path. +idempotent; the guest's readable end observes a **trap at its rendezvous** ("the +host dropped the writable end without writing a value") — the host-side spelling +of the guest's own drop-before-write trap — never a DROPPED event or a hang. An +unlowered future just releases state. Producer rejections keep the loud +site-named fault path. -**`cancelRead` is indistinguishable from end-of-stream** (#97): it -settles the in-flight `read` with an empty chunk, presented as clean EOS -by `readable()` and the iterator. The canceller is the observer, so no -signal is warranted. +**`cancelRead` is indistinguishable from end-of-stream** (#97): it settles the +in-flight `read` with an empty chunk, presented as clean EOS by `readable()` and +the iterator. The canceller is the observer, so no signal is warranted. ### Direct-access byte edges For `stream` only (#128; wasmtime `DirectSource`/`DirectDestination` -shaped), both host ends gain a form whose last hop *is* the canonical -ABI copy, so external buffer movers pay no second copy. +shaped), both host ends gain a form whose last hop _is_ the canonical ABI copy, +so external buffer movers pay no second copy. - `writeDirect(produce)` / `readDirect(consume)` (also on the low-level `HostWritableEnd`/`HostReadableEnd`) park a **direct session**. At each rendezvous with a peer operation of nonzero capacity, the callback runs - exactly once, synchronously, inside the rendezvous. `remaining()` is - the reader's unfilled landing zone (`produce`) or the writer's unread - bytes (`consume`); with a guest peer it aliases guest linear memory, so - the embedder's `set()`/`subarray` copy is the ABI copy. `"more"` keeps - the session parked; `"done"` ends it, resolving with the total byte - count. -- **Scope is the validity window.** The object dies when the callback - returns (later calls throw `TypeError`). Views are re-derived per - `remaining()` call (`memory.grow` never yields a stale view). Inside - the callback, running guest code or operating this stream is - forbidden; direct forms participate in the one-in-flight rule. + exactly once, synchronously, inside the rendezvous. `remaining()` is the + reader's unfilled landing zone (`produce`) or the writer's unread bytes + (`consume`); with a guest peer it aliases guest linear memory, so the + embedder's `set()`/`subarray` copy is the ABI copy. `"more"` keeps the session + parked; `"done"` ends it, resolving with the total byte count. +- **Scope is the validity window.** The object dies when the callback returns + (later calls throw `TypeError`). Views are re-derived per `remaining()` call + (`memory.grow` never yields a stale view). Inside the callback, running guest + code or operating this stream is forbidden; direct forms participate in the + one-in-flight rule. - **Marks acknowledge on clean return only.** `markWritten`/`markRead` - accumulate per invocation (over-marking throws). Returning with ≥ 1 - marked completes the peer's copy with that count. `"done"` with zero - marked is *retraction*: the session ends with its running total, the - peer stays parked, no event is delivered (the speculative-park - pattern). `"more"` with zero marked rejects `TypeError`. A throwing - callback rejects the session and discards its marks. In every outcome - the peer's operation survives and the stream stays alive; a - zero-progress COMPLETED copy is never emitted (a guest may read it as - EOS). -- **Zero-length-read readiness**: a parked session is the readiness - claim — a zero-length probe completes immediately without invoking the - callback; retraction corrects a speculative claim. -- **Host↔host**: a direct session against a peer *chunk* end costs one - copy (`produce` fills a fresh scratch that becomes the chunk; `consume` - gets a scoped view of the offered chunk). Two direct sessions cannot - rendezvous (neither owns memory): the arriving side throws `TypeError`. -- **Inherited rules**: peer trap rejects `PeerTrappedError` with the byte - count; reader/writer drop resolves with the total (a resolution the - producer's `"done"` did not cause is the peer-gone signal); - `cancelWrite`/`cancelRead` retract; the post-transfer refusal applies; - a parked session is retention. `writeDirect` on an unbound - `createStream()` writer parks until the element type binds, then - requires u8; `readDirect` on a non-u8 stream throws. -- **Deliberately absent**: an ownership-transfer chunk variant - (`write`'s borrowed-until-settled already meets the one-copy floor), a - `list` intake form, and any conduit/credit/realm machinery - (polyengine provides the byte edge, not the mover). SAB-backed - `Uint8Array`s are legal on the embedder's side of every copy. The - `HostBuffer` length bound (#97) applies to the buffered path only. - + accumulate per invocation (over-marking throws). Returning with ≥ 1 marked + completes the peer's copy with that count. `"done"` with zero marked is + _retraction_: the session ends with its running total, the peer stays parked, + no event is delivered (the speculative-park pattern). `"more"` with zero + marked rejects `TypeError`. A throwing callback rejects the session and + discards its marks. In every outcome the peer's operation survives and the + stream stays alive; a zero-progress COMPLETED copy is never emitted (a guest + may read it as EOS). +- **Zero-length-read readiness**: a parked session is the readiness claim — a + zero-length probe completes immediately without invoking the callback; + retraction corrects a speculative claim. +- **Host↔host**: a direct session against a peer _chunk_ end costs one copy + (`produce` fills a fresh scratch that becomes the chunk; `consume` gets a + scoped view of the offered chunk). Two direct sessions cannot rendezvous + (neither owns memory): the arriving side throws `TypeError`. +- **Inherited rules**: peer trap rejects `PeerTrappedError` with the byte count; + reader/writer drop resolves with the total (a resolution the producer's + `"done"` did not cause is the peer-gone signal); `cancelWrite`/`cancelRead` + retract; the post-transfer refusal applies; a parked session is retention. + `writeDirect` on an unbound `createStream()` writer parks until the element + type binds, then requires u8; `readDirect` on a non-u8 stream throws. +- **Deliberately absent**: an ownership-transfer chunk variant (`write`'s + borrowed-until-settled already meets the one-copy floor), a `list` intake + form, and any conduit/credit/realm machinery (polyengine provides the byte + edge, not the mover). SAB-backed `Uint8Array`s are legal on the embedder's + side of every copy. The `HostBuffer` length bound (#97) applies to the + buffered path only. ## Module wiring and instantiation One nested record keyed by verbatim interface id: + ```ts const instance = await instantiate(artifacts, { - "wasi:clocks/monotonic-clock@0.3.0": { now, getResolution, waitFor, waitUntil }, - "polymorph:websocket/connections@0.1.0": { Websocket }, // resource class + "wasi:clocks/monotonic-clock@0.3.0": { + now, + getResolution, + waitFor, + waitUntil, + }, + "polymorph:websocket/connections@0.1.0": { Websocket }, // resource class // world-level bare imports at the top level, camelCase }); ``` - Bindgen emits the world's `Imports` and `Exports` types and a typed - `instantiate` wrapper that **verifies the world digest** - (contracts/digest.md) before instantiating: it resolves artifacts - (translating if given the untranslated form; translation never runs - guest code), compares against the embedded `WORLD_DIGEST`, and throws a - named mismatch error carrying the divergence. The untyped runtime - `instantiate` names no world and checks nothing. `verify(plan)` and - `WORLD_DIGEST` stay exported; `bind()` is an explicitly unchecked cast. + `instantiate` wrapper that **verifies the world digest** (contracts/digest.md) + before instantiating: it resolves artifacts (translating if given the + untranslated form; translation never runs guest code), compares against the + embedded `WORLD_DIGEST`, and throws a named mismatch error carrying the + divergence. The untyped runtime `instantiate` does not check an expected world + digest; it still validates the plan, component hash, and required imports. + `verify(plan)` and `WORLD_DIGEST` stay exported; `bind()` is an explicitly + unchecked cast. - **Untranslated artifacts**: `instantiate({ componentBytes, translator })` - where `translator` is the shim wasm bytes or a shared `Translator` - (prefer sharing: the wasm compile is the cost). `requiredImports` still - takes a plan. -- **Build-time translation**: the translation envelope (single-file JSON - from `Translator.translateRaw` / tools/translate, plan + FACT adapters) - is the deploy artifact — `component.wasm` + envelope + runtime, no - translator. `artifactsFromEnvelope(envelopeJson, componentBytes)` - reconstitutes `ComponentArtifacts`; the envelope's component sha-256 is - verified at instantiation. Fetch-agnostic. + where `translator` is the shim wasm bytes or a shared `Translator` (prefer + sharing: the wasm compile is the cost). `requiredImports` still takes a plan. +- **Build-time translation**: the translation envelope (single-file JSON from + `Translator.translateRaw` / tools/translate, plan + FACT adapters) is the + deploy artifact — `component.wasm` + envelope + runtime, no translator. + `artifactsFromEnvelope(envelopeJson, componentBytes)` reconstitutes + `ComponentArtifacts`; the envelope's component sha-256 is verified at + instantiation. Fetch-agnostic. - **Per-interface module authoring** is a helper over the same record: a - module's named export, camelCase of the interface short-name, provides - that interface. `"ns:pkg/*@0.2": mod` wildcards expand over interface - *names* at one version or track key. -- `requiredImports(artifacts)` enumerates the component's linkable import - leaves with kinds and types — the replacement for hand-rolled walks of - `plan.imports`. - + module's named export, camelCase of the interface short-name, provides that + interface. `"ns:pkg/*@0.2": mod` wildcards expand over interface _names_ at + one version or track key. +- `requiredImports(artifacts)` enumerates the component's linkable import leaves + with kinds and types. ## Module identity and @polyengine/protocol -Nothing in a source-distributed ecosystem guarantees one runtime copy per -module graph, and class-identity checks then fail latently. Class identity -is therefore not part of this contract. +Multiple runtime copies can coexist in a module graph. Cross-copy recognition +uses protocol brands rather than `instanceof`; stateful handles still belong to +the runtime copy that created them. -**The protocol package.** `@polyengine/protocol` is dependency-free and -carries the contract's vocabulary: brand symbols, the error classes +**The protocol package.** `@polyengine/protocol` is dependency-free and carries +the contract's vocabulary: brand symbols, the error classes (`ComponentException`, `Trap`, `DroppedError`, `PeerTrappedError`, `InvalidHandleError`, `StreamProducerError`), the import marks and their -predicates, the recognition predicates, the handle interfaces, realm -crossing, the copy registry, and `PROTOCOL_GENERATION`. Host modules -import it at most; with hand-rolled brands even that is optional. Copies -of it are harmless: identity rests on registry symbols only. - -**Brands** are `Symbol.for` registry symbols, so N copies agree without -sharing modules. Keys are generation-suffixed; bumping a generation is a -breaking vocabulary change (a semver major in effect). - -| brand key | carried by | marks | -|---|---|---| -| `polyengine.componentException/1` | `ComponentException.prototype` | err-result values | -| `polyengine.trap/1` | `Trap.prototype` | component-fatal errors | -| `polyengine.dropped/1` | `DroppedError.prototype` | dropped-future rejections | -| `polyengine.peerTrapped/1` | `PeerTrappedError.prototype` | peer-fault rejections | -| `polyengine.invalidHandle/1` | `InvalidHandleError.prototype` | handle misuse | -| `polyengine.streamProducer/1` | `StreamProducerError.prototype` | producer-side failures | -| `polyengine.suspending/1` | marked function / class prototype | suspendable sync imports | -| `polyengine.deferCancel/1` | marked function | cancel-discard exempt imports | -| `polyengine.abortable/1` | marked function | imports receiving an `AbortSignal` | -| `polyengine.syncCallable/1` | lifted exports and guest-resource members (runtime-defined; application-tier) | sync-callable exports | -| `polyengine.stream/1` | `Stream.prototype` | stream handles | -| `polyengine.streamWriter/1` | `StreamWriter.prototype` | writer handles | -| `polyengine.future/1` | `Future.prototype` | future handles | -| `polyengine.errorContext/1` | `ErrorContext.prototype` | error-contexts (message-valued at lowering) | -| `polyengine.resourceState/1` | guest-resource wrappers (internal state key) | resource wrappers | -| `polyengine.pollable/1` | `Pollable.prototype` (wasi package) | pollables | -| `polyengine.wasiExit/1` | `ExitError.prototype` (wasi package) | wasi exit unwinds | -| `polyengine.runtimeCopies/1` | `globalThis` | the copy registry | - -Keys are the class name minus `Error`, camelCased, or the concept -(`wasiExit` is prefixed because bare `exit` is too generic). The digest's -`cewd` constant is hashed wire content, frozen independently. +predicates, the recognition predicates, the handle interfaces, realm crossing, +the copy registry, and `PROTOCOL_GENERATION`. Host modules import it at most; +with hand-rolled brands even that is optional. Copies of it are harmless: +identity rests on registry symbols only. + +**Brands** are `Symbol.for` registry symbols, so N copies agree without sharing +modules. Keys are generation-suffixed; bumping a generation is a breaking +vocabulary change (a semver major in effect). + +| brand key | carried by | marks | +| --------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------- | +| `polyengine.componentException/1` | `ComponentException.prototype` | err-result values | +| `polyengine.trap/1` | `Trap.prototype` | component-fatal errors | +| `polyengine.dropped/1` | `DroppedError.prototype` | dropped-future rejections | +| `polyengine.peerTrapped/1` | `PeerTrappedError.prototype` | peer-fault rejections | +| `polyengine.invalidHandle/1` | `InvalidHandleError.prototype` | handle misuse | +| `polyengine.streamProducer/1` | `StreamProducerError.prototype` | producer-side failures | +| `polyengine.suspending/1` | marked function / class prototype | suspendable sync imports | +| `polyengine.deferCancel/1` | marked function | cancel-discard exempt imports | +| `polyengine.abortable/1` | marked function | imports receiving an `AbortSignal` | +| `polyengine.syncCallable/1` | lifted exports and guest-resource members (runtime-defined; application-tier) | sync-callable exports | +| `polyengine.stream/1` | `Stream.prototype` | stream handles | +| `polyengine.streamWriter/1` | `StreamWriter.prototype` | writer handles | +| `polyengine.future/1` | `Future.prototype` | future handles | +| `polyengine.errorContext/1` | `ErrorContext.prototype` | error-contexts (message-valued at lowering) | +| `polyengine.resourceState/1` | guest-resource wrappers (internal state key) | resource wrappers | +| `polyengine.pollable/1` | `Pollable.prototype` (wasi package) | pollables | +| `polyengine.wasiExit/1` | `ExitError.prototype` (wasi package) | wasi exit unwinds | +| `polyengine.runtimeCopies/1` | `globalThis` | the copy registry | + +Keys are the class name minus `Error`, camelCased, or the concept (`wasiExit` is +prefixed because bare `exit` is too generic). The digest's `cewd` constant is +hashed wire content, frozen independently. **Brands are contract markers, not a security boundary.** An Error with -`[Symbol.for("polyengine.componentException/1")]: true` and a `payload` -IS a ComponentException to every copy; a function with the suspending -brand IS marked. The classes are conveniences. - -**Stateless vs stateful.** For error classes and marks, brand agreement -is the whole story: a copy-B `ComponentException` is honored at a copy-A -boundary. Stateful values (handles, resource wrappers) live in the copy -that minted them, so the brand converts "misclassified" into -"recognized-but-foreign": a named error listing both copies' URLs (the -cross-store assert family distinguishes cross-copy from cross-store). -Error-contexts sit between: any branded carrier of a string `message` -lowers by minting a fresh local context. +`[Symbol.for("polyengine.componentException/1")]: true` and a `payload` IS a +ComponentException to every copy; a function with the suspending brand IS +marked. The classes are conveniences. + +**Stateless vs stateful.** For error classes and marks, brand agreement is the +whole story: a copy-B `ComponentException` is honored at a copy-A boundary. +Stateful values (handles, resource wrappers) live in the copy that minted them, +so the brand converts "misclassified" into "recognized-but-foreign": a named +error listing both copies' URLs (the cross-store assert family distinguishes +cross-copy from cross-store). Error-contexts sit between: any branded carrier of +a string `message` lowers by minting a fresh local context. **The copy registry.** Each embedder module instance appends `{ url, runtimeVersion, protocolGeneration }` to -`globalThis[Symbol.for("polyengine.runtimeCopies/1")]` on evaluation. -Copies are diagnosed, never refused (isolated bundles exchanging no -values are legal): cross-copy errors name both URLs, and the -unbranded-throw trap appends a copy census when more than one copy is -registered. Resolution discipline stays necessary for cost (N compiles, N -payloads): host-module packages carry no `@polyengine/*` import-map -entries (docs/consumers.md). - -**Identity is realm-local** (#129). Two realms (window and worker, two -workers) are two runtimes by construction — placement, not a defect. -`Symbol.for` does not span agents, and structured clone strips prototypes -and refuses functions and symbol keys, so a handle, wrapper, branded -error, or marked function crossing `postMessage` arrives as an inert -object recognized by nothing. The next section defines the sanctioned -representations. The runtime, translator, and embedder paths carry no -main-thread-only dependencies; the conformance realm rows (Deno worker, -browser dedicated/shared worker, OPFS worker) gate this in CI, and where -a platform API differs by realm the runtime uses the intersection. - +`globalThis[Symbol.for("polyengine.runtimeCopies/1")]` on evaluation. Copies are +diagnosed, never refused (isolated bundles exchanging no values are legal): +cross-copy errors name both URLs, and the unbranded-throw trap appends a copy +census when more than one copy is registered. Resolution discipline stays +necessary for cost (N compiles, N payloads): host-module packages carry no +`@polyengine/*` import-map entries (docs/consumers.md). + +**Identity is realm-local** (#129). Two realms (window and worker, two workers) +are two runtimes by construction — placement, not a defect. `Symbol.for` does +not span agents, and structured clone strips prototypes and refuses functions +and symbol keys, so a handle, wrapper, branded error, or marked function +crossing `postMessage` arrives as an inert object recognized by nothing. The +next section defines the sanctioned representations. The runtime, translator, +and embedder paths carry no main-thread-only dependencies; the conformance realm +rows (Deno worker, browser dedicated/shared worker, OPFS worker) gate this in +CI, and where a platform API differs by realm the runtime uses the intersection. ## Realm boundaries and structured-clone-safe forms -Proxies carrying embedder-typed values over `postMessage` need a defined -form, or every author invents a subtly wrong one. `@polyengine/protocol` -exports: +Proxies carrying embedder-typed values over `postMessage` need a defined form, +or every author invents a subtly wrong one. `@polyengine/protocol` exports: ```ts function toCloneable(v: unknown, opts?: { @@ -753,188 +700,178 @@ function toCloneable(v: unknown, opts?: { function fromCloneable(data: unknown): unknown; ``` -`toCloneable` returns plain data safe for `structuredClone`/`postMessage` -(no transfer list; hence also `BroadcastChannel`, IndexedDB — a property -of plain data, not a compatibility promise). `fromCloneable` rehydrates -every envelope into a value **branded by the local copy** — a new local -value, never "the same" one. No RPC, no proxying, no cross-realm identity. +`toCloneable` returns plain data safe for `structuredClone`/`postMessage` (no +transfer list; hence also `BroadcastChannel`, IndexedDB — a property of plain +data, not a compatibility promise). `fromCloneable` rehydrates every envelope +into a value **branded by the local copy** — a new local value, never "the same" +one. No RPC, no proxying, no cross-realm identity. **Round-trip law** (tested per taxonomy member): -`fromCloneable(structuredClone(toCloneable(v)))` is indistinguishable -from `v` under every matcher this contract offers — predicates, -`payload`/`kind`/`value`, `message`, `cause` chains, `progress`. Cause -chains are walked to full depth through branded and unbranded links (a -`PeerTrappedError.cause` is an unbranded poisoning record whose `cause` -is the `Trap`, which must still satisfy `isTrap`). `stack` is carried -verbatim. - -**The envelope**: a plain object whose tag property -`"polyengine.cloneable/1"` holds the brand key string. No WIT-mapped value -collides (WIT identifiers contain neither `.` nor `/`; `map` is a list), -and an input already carrying the tag is refused, so no escaping is -needed. Detection is by brand, so hand-rolled branded values encode -identically. - -| tag value | encodes | fields besides the tag | -|---|---|---| -| `polyengine.componentException/1` | `ComponentException` | `message`, `stack?`, `cause?` (walked), `payload` (walked) | -| `polyengine.trap/1` | `Trap` | `message`, `stack?`, `cause?` (walked) | -| `polyengine.dropped/1` | `DroppedError` | `message`, `stack?`, `cause?` (walked) | -| `polyengine.invalidHandle/1` | `InvalidHandleError` | `message`, `stack?`, `cause?` (walked) | -| `polyengine.peerTrapped/1` | `PeerTrappedError` | `message`, `stack?`, `progress?`, `cause` (walked) | -| `polyengine.streamProducer/1` | `StreamProducerError` | `message`, `stack?`, `cause` (walked) | -| `polyengine.errorContext/1` | error-context (its message) | `message` | -| `polyengine.wasiExit/1` | the wasi package's `ExitError` | `message`, `stack?`, `ok`, `code?` | -| `error` | an unbranded `Error` | `name`, `message`, `stack?`, `cause?` (walked) | - -`fromCloneable` rehydrates the six error tags as protocol class -instances; `error` as a plain `Error` with `name` restored; `wasiExit` as -a hand-rolled branded `Error` with `ok`/`code` (protocol does not import -wasi); `errorContext` as a branded `{ message }`, which lowering accepts. -An unknown tag throws `TypeError`: mixed engine versions are outside the -matrix. +`fromCloneable(structuredClone(toCloneable(v)))` is indistinguishable from `v` +under every matcher this contract offers — predicates, `payload`/`kind`/`value`, +`message`, `cause` chains, `progress`. Cause chains are walked to full depth +through branded and unbranded links (a `PeerTrappedError.cause` is an unbranded +poisoning record whose `cause` is the `Trap`, which must still satisfy +`isTrap`). `stack` is carried verbatim. + +**The envelope**: a plain object whose tag property `"polyengine.cloneable/1"` +holds the brand key string. No WIT-mapped value collides (WIT identifiers +contain neither `.` nor `/`; `map` is a list), and an input already carrying the +tag is refused, so no escaping is needed. Detection is by brand, so hand-rolled +branded values encode identically. + +| tag value | encodes | fields besides the tag | +| --------------------------------- | ------------------------------ | ---------------------------------------------------------- | +| `polyengine.componentException/1` | `ComponentException` | `message`, `stack?`, `cause?` (walked), `payload` (walked) | +| `polyengine.trap/1` | `Trap` | `message`, `stack?`, `cause?` (walked) | +| `polyengine.dropped/1` | `DroppedError` | `message`, `stack?`, `cause?` (walked) | +| `polyengine.invalidHandle/1` | `InvalidHandleError` | `message`, `stack?`, `cause?` (walked) | +| `polyengine.peerTrapped/1` | `PeerTrappedError` | `message`, `stack?`, `progress?`, `cause` (walked) | +| `polyengine.streamProducer/1` | `StreamProducerError` | `message`, `stack?`, `cause` (walked) | +| `polyengine.errorContext/1` | error-context (its message) | `message` | +| `polyengine.wasiExit/1` | the wasi package's `ExitError` | `message`, `stack?`, `ok`, `code?` | +| `error` | an unbranded `Error` | `name`, `message`, `stack?`, `cause?` (walked) | + +`fromCloneable` rehydrates the six error tags as protocol class instances; +`error` as a plain `Error` with `name` restored; `wasiExit` as a hand-rolled +branded `Error` with `ok`/`code` (protocol does not import wasi); `errorContext` +as a branded `{ message }`, which lowering accepts. An unknown tag throws +`TypeError`: mixed engine versions are outside the matrix. **Walk semantics** (`fromCloneable` mirrors): -- Pass through: primitives, `null`, `undefined`; `ArrayBuffer`, typed - arrays, `DataView` by reference. -- Walk into fresh containers: arrays; plain objects (`Object.prototype` - or `null`), own enumerable string keys. + +- Pass through: primitives, `null`, `undefined`; `ArrayBuffer`, typed arrays, + `DataView` by reference. +- Walk into fresh containers: arrays; plain objects (`Object.prototype` or + `null`), own enumerable string keys. - Encode branded values per the table and unbranded `Error`s as `error`; encodable brands take precedence over the realm-local pill. -- Refuse with `InvalidHandleError`: realm-local leaves — `isRealmLocal`, - the `STREAM`/`FUTURE`/`POLLABLE` brands, resource wrappers — unless - `replace` substitutes. **Proxy the interface, not the handle.** +- Refuse with `InvalidHandleError`: realm-local leaves — `isRealmLocal`, the + `STREAM`/`FUTURE`/`POLLABLE` brands, resource wrappers — unless `replace` + substitutes. **Proxy the interface, not the handle.** - Refuse with `TypeError`: functions, symbols, cycles, other prototypes. - `Map`/`Set`/`Date`/`RegExp` cannot occur in WIT data and pass through - unwalked (a hidden handle still trips its pill at clone time). + `Map`/`Set`/`Date`/`RegExp` cannot occur in WIT data and pass through unwalked + (a hidden handle still trips its pill at clone time). - Every refusal names the path to the leaf (`payload.attempts[2].handle`). -Aliasing is not preserved; cycles are refused. **Version-internal, not a -wire format**: both realms run the same engine version -(docs/consumers.md); the envelope may change in any release. +Aliasing is not preserved; cycles are refused. **Version-internal, not a wire +format**: both realms run the same engine version (docs/consumers.md); the +envelope may change in any release. **The realm-local pill.** Every `Stream`, `StreamWriter`, `Future`, -`ErrorContext`, guest-resource wrapper, and wasi `Pollable` carries an -own, enumerable, string-keyed property `"polyengine.realmLocal/1"` whose -value is the named function `polyengineRealmLocalValue`. Structured -serialization visits exactly such properties and refuses functions, so a -raw `postMessage` of such a value — even buried in a record — throws -`DataCloneError` in the **sender**. (It must be a string key and own; the -brand mechanism cannot serve.) `JSON.stringify` and spread are -unaffected. Vocabulary: `REALM_LOCAL`, `defineRealmLocal(target)`, -`isRealmLocal(v)`. - -**Errors cannot be pilled**: the serializer's `[[ErrorData]]` branch -keeps `name`/`message`/`stack` only, so a raw-cloned branded error husks -silently. `toCloneable` fills that gap; the pill covers the stateful -handles; together they partition the vocabulary. - -**Error-context is message-valued** (definitions.py): lowering accepts -any branded carrier of a string `message` — `fromCloneable` output or a -hand-rolled `{ [Symbol.for("polyengine.errorContext/1")]: true, message }` -— minting a fresh local context. A branded error-context without a -string `message` keeps the cross-copy refusal. - +`ErrorContext`, guest-resource wrapper, and wasi `Pollable` carries an own, +enumerable, string-keyed property `"polyengine.realmLocal/1"` whose value is the +named function `polyengineRealmLocalValue`. Structured serialization visits +exactly such properties and refuses functions, so a raw `postMessage` of such a +value — even buried in a record — throws `DataCloneError` in the **sender**. (It +must be a string key and own; the brand mechanism cannot serve.) +`JSON.stringify` and spread are unaffected. Vocabulary: `REALM_LOCAL`, +`defineRealmLocal(target)`, `isRealmLocal(v)`. + +**Errors cannot be pilled**: the serializer's `[[ErrorData]]` branch keeps +`name`/`message`/`stack` only, so a raw-cloned branded error husks silently. +`toCloneable` fills that gap; the pill covers the stateful handles; together +they partition the vocabulary. + +**Error-context is message-valued** (definitions.py): lowering accepts any +branded carrier of a string `message` — `fromCloneable` output or a hand-rolled +`{ [Symbol.for("polyengine.errorContext/1")]: true, message }` — minting a fresh +local context. A branded error-context without a string `message` keeps the +cross-copy refusal. ## The host-ABI surface and its version A host module consumes protocol vocabulary only. Naming -`jsr:@polyengine/runtime` would let every lockstep engine release (plan -format, translator) invalidate its range, so the surfaces split: +`jsr:@polyengine/runtime` would let every lockstep engine release (plan format, +translator) invalidate its range, so the surfaces split: -**Protocol carries the whole host-boundary vocabulary**: brands, error -classes, marks, predicates, and the handle interfaces of +**Protocol carries the whole host-boundary vocabulary**: brands, error classes, +marks, predicates, and the handle interfaces of [Streams and futures](#streams-and-futures) as structural TypeScript (`Stream`, `StreamWriter`, `Future`, `ErrorContext`, `Chunk`, -`DirectSource`, `DirectDestination`, `StreamSource`, -`FutureSource`) with brand predicates. The runtime's classes declare -`implements` against them (`just test-runtime`). +`DirectSource`, `DirectDestination`, `StreamSource`, `FutureSource`) with +brand predicates. The runtime's classes declare `implements` against them +(`just test-runtime`). **The runtime's exported surface is application-only.** -`@polyengine/runtime/embedder` exports `instantiate`/ -`instantiateEmbedder`, artifact resolution, `requiredImports`, the import -resolver and version canonicalization, `NameCollisionError`, the -value-bridge/casing utilities, `sync()`, and `createStream()`. It -exports no error classes, predicates, brands, marks, realm crossing, copy -registry, or concrete handle classes. Host modules produce streams and -futures as natural JS producers; one wanting writer-driven push -(`writeDirect`) or a synchronous guest callback (`sync(exports.f)`) is -handed it by the application, keeping placement with the deployer. - -**Host modules MUST NOT import `@polyengine/runtime`**; a one-line -no-specifier check gates it. The wasi package is protocol-only -(`isStream`, not `instanceof Stream`). +`@polyengine/runtime/embedder` exports `instantiate`/ `instantiateEmbedder`, +artifact resolution, `requiredImports`, the import resolver and version +canonicalization, `NameCollisionError`, the value-bridge/casing utilities, +`sync()`, and `createStream()`. It exports no error classes, predicates, +brands, marks, realm crossing, copy registry, or concrete handle classes. Host +modules produce streams and futures as natural JS producers; one wanting +writer-driven push (`writeDirect`) or a synchronous guest callback +(`sync(exports.f)`) is handed it by the application, keeping placement with the +deployer. + +**Host modules MUST NOT import `@polyengine/runtime`**; a one-line no-specifier +check gates it. The wasi package is protocol-only (`isStream`, not +`instanceof Stream`). **The conventions suite is the executable definition of the host ABI.** -`runtime/tests/conventions/` exercises this contract against a probe host -module written as consumers write theirs (protocol imports only) and -commits normalized transcripts under `runtime/tests/conventions/golden/`. -- Modifying or deleting a golden asserts a host-ABI behavior change and - requires `breaking/protocol` (with its protocol minor bump) in the same - PR; the reviewed behavior-neutral escape is the `conventions-fix` label. -- Adding goldens is free. -- version-guard enforces this at PR time (labels, advisory) and - authoritatively at cut time (any M/D under the goldens dir in the - release window requires protocol past the last cut). +`runtime/tests/conventions/` exercises this contract against a probe host module +written as consumers write theirs (protocol imports only) and commits normalized +transcripts under `runtime/tests/conventions/golden/`. -**Consequence: protocol's version is the host-ABI version.** A host -module pins `jsr:@polyengine/protocol@^0.x`; an engine change that leaves -the goldens byte-identical is host-ABI-neutral by definition. +- Modifying or deleting a golden asserts a host-ABI behavior change and requires + `breaking/protocol` (with its protocol minor bump) in the same PR; the + reviewed behavior-neutral escape is the `conventions-fix` label. +- Adding goldens is free. +- version-guard enforces this at PR time (labels, advisory) and authoritatively + at cut time (any M/D under the goldens dir in the release window requires + protocol past the last cut). +**Consequence: protocol's version is the host-ABI version.** A host module pins +`jsr:@polyengine/protocol@^0.x`. Golden changes are a mechanical versioning +signal, not exhaustive coverage: review behavior changes against this contract +even when existing transcripts are unchanged. ## Implementation strategy -Bindgen emits, per world: `Imports`/`Exports` types; resource classes -(both directions); `ComponentException` payload types per fallible -function; value types per the mapping table; mangled-name assembly -(`[method]r.f` ↔ class methods) both ways; stream/future adapters -including pumping; the digest handshake. The generated layer adapts the -raw boundary, whose shapes are pinned by the reference-test ports and the -conformance harness. - -The interpreter's variant family now also carries `kind`/`value` (#261), -so internal and host values can be structurally identical yet mean -different things. Four differences every adapting site translates -deliberately: +Bindgen emits world `Imports`/`Exports`, value and resource types, and a typed +instantiation wrapper with the digest handshake. The runtime facade builds +resource classes, assembles mangled names, adapts values and errors, and pumps +streams/futures from plan descriptors. These behaviors do not depend on +generated JS adapters. -| | internal | this layer | -|---|---|---| -| `result` error case | `"error"` (definitions.py) | `"err"` | -| `enum` | a variant value | bare kebab-case string | -| `option` | always `{kind: "none" \| "some", value}` | outermost `T \| undefined`; only directly-nested options box | -| payloadless case | `value: null` | property omitted | +Raw and facade values can share property names while having different +representations. Every adapting site preserves these distinctions: -Full convergence is a perf-track concern (descriptor-driven codegen can -emit convention shapes directly). `instance.exports` stays -internal-shaped. +| | internal | this layer | +| ------------------- | ---------------------------------------- | ------------------------------------------------------------ | +| `result` error case | `"error"` (definitions.py) | `"err"` | +| `enum` | a variant value | bare kebab-case string | +| `option` | always `{kind: "none" \| "some", value}` | outermost `T \| undefined`; only directly-nested options box | +| payloadless case | `value: null` | property omitted | +| tuple | record with numeric keys | array | +The raw executor's exports remain internal-shaped; the public facade applies +these conversions. ## The WASI parking kernel WASI stays out of the runtime core (docs/architecture.md §2); the `wasi/` package is the executable check that the conventions fit. Most of it maps directly (p3 clocks over `setTimeout` with no JSPI; p3 sockets and http as -resource classes with async methods and stream I/O). The idiom that fights -a JS host is p2's sync-blocking surface — `pollable.block()`, `poll()`, -`blocking-read`/`blocking-write-and-flush` are **sync** WIT functions that -must park. The wasi package ships the parking kernel, always on: +resource classes with async methods and stream I/O). The idiom that fights a JS +host is p2's sync-blocking surface — `pollable.block()`, `poll()`, +`blocking-read`/`blocking-write-and-flush` are **sync** WIT functions that must +park. The wasi package ships the parking kernel, always on: - `block`/`poll` are `suspending()`-marked with sync fast paths: a ready - pollable costs one engine hop; only a genuine wait parks. Timer - pollables are real (monotonic-clock `subscribe-*`). + pollable costs one engine hop; only a genuine wait parks. Timer pollables are + real (monotonic-clock `subscribe-*`). - Without JSPI, `chooseMode` degrades to plain and a genuine park raises - `NeedsJspi` at the park site; `jspi: false` is the per-instantiation - opt-out. -- Stream `read`/`check-write` are plain. The p2 `blocking-*` declarations - are marked park-capable: buffer-backed default streams take the sync - fast path; genuinely async impls (host stdin behind `blocking-read`) - return Promises and park. The prototype-declares/instances-behave mark - relay lets duck-typed impls park through the registered types. -- `Pollable` is publicly constructible (`new Pollable(ready, wait)`) as - the seam for external providers; `wait()` follows the promise-swap - shape (settle and re-arm per event; spurious wakes are fine). - -Consequence for the zero-cost pin: a component importing marked providers -auto-detects jspi mode on JSPI engines even if it never parks — "zero-cost -plain path" means sync-only plan **and** no marked imports -(contracts/intrinsics.md). + `NeedsJspi` at the park site; `jspi: false` is the per-instantiation opt-out. +- Stream `read`/`check-write` are plain. The p2 `blocking-*` declarations are + marked park-capable: buffer-backed default streams take the sync fast path; + genuinely async impls (host stdin behind `blocking-read`) return Promises and + park. The prototype-declares/instances-behave mark relay lets duck-typed impls + park through the registered types. +- `Pollable` is publicly constructible (`new Pollable(ready, wait)`) as the seam + for external providers; `wait()` follows the promise-swap shape (settle and + re-arm per event; spurious wakes are fine). + +A component importing marked providers selects JSPI on capable engines even if a +particular call never waits. The plain fast path requires a sync-only plan and +no marked imports; see [intrinsics.md](intrinsics.md). diff --git a/contracts/intrinsics.md b/contracts/intrinsics.md index ee6611f..683eedd 100644 --- a/contracts/intrinsics.md +++ b/contracts/intrinsics.md @@ -1,240 +1,149 @@ # Contract: Host Intrinsics -Everything the TS runtime must provide to wasm it did not author: (A) imports -of FACT-generated adapter modules, and (B) host trampolines referenced from -the plan (`CoreDef::Trampoline` / `lower-import`). Producers of the -requirement: the translator shim (per-plan manifest). Implementor: the runtime -(`runtime/src/intrinsics/`). - -Sources of truth (pinned `wasmtime-environ`, git rev in the root `Cargo.toml`): -- (A) `wasmtime_environ::fact::Import` — every import FACT can emit. -- (B) `wasmtime_environ::component::Trampoline` — every host trampoline the - plan can reference. -The shim must fail translation with a clear error if it encounters a variant -not representable in the plan (never silently drop). +The runtime supplies core-Wasm imports for FACT adapters and canonical builtins. +The translator records these requirements in the plan; the implementation lives +in `runtime/src/intrinsics/`. + +The pinned `wasmtime-environ` enums define the calling convention: +`fact::Import` for adapter imports and `component::Trampoline` for host +trampolines. The shim rejects variants it cannot represent. Component Model +semantics remain governed by the pinned spec and `definitions.py`, subject to +[architecture §1](../docs/architecture.md#1-goals)'s named exception. ## Universal semantics -1. **Synchronous, non-suspending.** Every intrinsic and trampoline body runs - to completion in JS and returns. JS frames here are compatible with JSPI - because they complete before any suspension occurs - (docs/architecture.md §5). An intrinsic that needs to wait is a design - error — waiting belongs to the task core. -2. **Traps** are thrown as the runtime's `ComponentTrap` and must not be - catchable by guest code (they propagate through wasm as JS exceptions). -3. **Instance-state rules** (`may_leave` bookkeeping) implement the - Component Model invariants — the engine (JSPI) will not - enforce them for us. The spec has no reentrance gate (CM#705 removed - `may_enter`/`entering_set`): reentrance into a live instance - (host-mediated, dtor, `*-start-call`, `enter-sync-call`) is valid. What - entry sites enforce instead is **per-instance poisoning refusal** — a - docs/architecture.md §6 named divergence, not a reference rule: a - trapped instance's corpse refuses entry permanently with the recorded - cause, with the same-instance exemption preserved for dtor self-drops. +1. **Suspension belongs to the task machinery.** A synchronous intrinsic returns + directly. A blocking intrinsic returns a scheduler-controlled Promise through + `WebAssembly.Suspending`; its JS body returns before Wasm suspends. No + ordinary JS frame may span the suspended Wasm stack. See + [architecture §5](../docs/architecture.md#5-the-jspi-frame-rule-load-bearing-constraint). +2. **Trap and capability failures differ.** Guest violations raise `Trap`. + Unsupported operations raise capability errors and must not satisfy + conformance trap assertions. Component Model traps must be uncatchable, but + this runtime's JS exceptions do not fully provide that guarantee: a guest + `try_table catch_all` can catch a host trap. Adapter exception barriers + preserve the original diagnostic through `HostTrapState`; they do not + eliminate the limitation. +3. **Instance invariants are runtime obligations.** JSPI does not enforce + `may_leave`, borrow scopes, or task exclusivity. Reentrance into a live + instance is valid. Entry refusal is the runtime's per-instance poisoning + policy, not a spec reentrance rule; it preserves the same-instance exemption + for destructor self-drops. See + [architecture §6](../docs/architecture.md#6-concurrency-the-core-deliverable). ## A. FACT adapter imports -Import-module namespaces observed in generated adapters: -`sync`, `async`, `transfer`, `transcode`, `callee`, `post_return`, `m` -(memories), `f` (funcs), `flags`, `runtime`, `instance`, `callback`. - -**Everything folds into CoreDef wiring**: `translate/adapt.rs` -(`fact_import_to_core_def`) folds *every* `fact::Import` into -instantiation-argument `CoreDef`s — the runtime never sees `fact::Import` -directly. Intrinsic-like imports arrive as `CoreDef::Trampoline` entries -(Trap, Enter/ExitSyncCall, Transcoder, ResourceTransfer*, PrepareCall, -*StartCall, *Transfer), `CoreDef::UnsafeIntrinsic` (the `context.{get,set}` -slot save/restore FACT wraps around `realloc` and `post-return`), or plain -wiring (callee funcs, memories, flags globals). The per-adapter manifest is import-names × -resolved args, categorized — which is exactly what the shim emits. - -Pinned decisions: - -- **Instance flags**: one `WebAssembly.Global(i32, mutable, initial 1)` per - component instance serves as both the FACT-visible flags global and - host-side `may_leave`; FACT 47 reads/writes it as a plain 0/1 boolean (no - bitmask). -- **`Trap` is one nullary import per trap code** (`runtime.trap`); the - code is a plan-visible field of the `trap` trampoline (plan-format.md), - and every code maps to `ComponentTrap` with wasmtime's message text. -- **No eager sync-blocking check in adapters.** wasmtime #14146 removed the - `task_may_block` global and the static same-instance/ancestor - `cannot enter component` stub: a sync-typed function may call an - async-typed function or blocking built-in, and the trap fires only if it - actually has to block with no runnable thread left (the scheduler's - deadlock trap, definitions.py `Thread.wait_until`/`switch`). Reentrance is - allowed except into a trapped instance. -- **ResourceTransfer semantics**: `resource-transfer-borrow` registers the - source handle as a lender on the current sync-call scope and increments - `num_lends` **unconditionally — borrow handles may be re-lent onward** - (`definitions.py lift_borrow`/`Subtask.add_lender`; a lent handle blocks - `resource.drop` until the call returns). Same-instance transfers take the - rep fast path but still register the lender. -- **Trap-unwind obligations**: when a trap escapes a FACT sync-call - bracket, the host must unwind sync-call scopes (releasing lenders) AND - restore `may_leave` on all component instances — FACT clears it around - lift/lower and a trap skips its restore; without both unwinds the - instance is unusable for post-trap re-entry, which this runtime - deliberately supports. The obligation covers every window that registers - lenders, including the prepare/start protocol — `sync-start-call`'s - inline lender scope and `async-start-call`'s subtask-attached lenders - release on every non-success exit that does not poison the caller (trap - rethrow AND capability signals: `NeedsJspi` is expressly non-poisoning - and must not strand lenders). -- **Host-trap preservation across nested barriers**: the trap trampoline - must (re)record the pending trap before every throw, so the specific - message survives arbitrarily nested adapter exception barriers. Residual, - documented limitation: our traps are JS exceptions, so a guest - `try_table catch_all` can observe them mid-flight (wasmtime's are - unforgeable); full unforgeability would need an out-of-band poison flag. -- **`Transcoder` trampoline parameters are plan-visible**: `op` (one of the - 12 `Transcode` ops), `from`/`to` runtime-memory indices, `from64`/`to64`. - Semantics authority is wasmtime's libcalls (partial-progress primitives - driving FACT's realloc/retry protocol), NOT definitions.py's whole-string - transcoding model. All 12 ops implemented and reference-tested. -- **Trap messages align to wasmtime's `Display for Trap` texts** (with the - `wasm trap: ` prefix where wasmtime uses it) — the official suite asserts - these strings, and wasmtime-compat is a plan goal. +Wasmtime's `fact_import_to_core_def` maps every adapter import to a `CoreDef`. +The runtime receives trampolines, context intrinsics, or ordinary core +functions, memories, and flags globals; it never consumes `fact::Import` +directly. `modules[].intrinsics` records import names and resolved categories. + +- **Instance flags:** a mutable i32 `WebAssembly.Global`, initially 1, + represents `may_leave` as a 0/1 boolean, not a bitmask. +- **Traps:** each `runtime.trap` is a nullary import. Its plan declaration + carries the pinned wasmtime trap discriminant. Known trap codes use + wasmtime-compatible diagnostic text for the conformance corpus; unknown codes + retain their numeric identity. +- **Blocking:** a sync-typed function may call an async function or blocking + builtin. The scheduler applies the reference's actual-blocking and deadlock + rules; adapters must not reject the call eagerly merely because the callee + could block. +- **Resource borrows:** transfer registers a lender on the current call scope, + including re-lending an already borrowed handle and same-instance rep fast + paths. A lend prevents own transfer or drop until its scope ends. +- **Unwind:** a failed call releases the lenders it registered, including + non-poisoning capability failures. The host boundary restores `may_leave` + according to entry identity: it excludes the host entry's own instance and + skips global restoration during a nested guest destructor call. This is not a + general filter on sibling poison state. FACT call scopes are activation-local + so interleaved tasks cannot release each other's lends. +- **Destructive removal:** validation follows handle removal, as in the + reference. If a removed stream/future end fails validation, local unwind + retires it; a later table walk cannot find it. Successful transfers do not + retire their shared state. Peer notification failure must neither replace the + original trap nor skip drop observers. +- **Nested exception barriers:** trap trampolines re-record the pending host + trap before throwing, preserving its cause through outer barriers. +- **Transcoding:** the plan records `op`, source/destination memory indices, and + `from64`/`to64`. Wasmtime's libcalls define the partial-progress protocol; the + composed lift/lower must still follow the Component Model string rules. All + twelve operations are implemented for memory32. The shim's + `fact_string_limits` correction enforces the pinned reference's source-byte + limit before guest realloc; its dependency-pattern guards and encoding-matrix + tests must be reviewed when upgrading environ. ## B. Host trampolines (`Trampoline` enum) -The authoritative variant list is the enum itself; the shim emits the full -typed list into `plan.json` `trampolines`, and the runtime's coverage -assertion at instantiate time is what keeps the implemented set honest. -Implemented: `LowerImport` (host function call through descriptor-IR -lift/lower), `ResourceDrop` (incl. the dtor call rules of -docs/architecture.md §7), `ResourceNew`, `ResourceRep`, `Transcoder`, and -the task-core set — `BackpressureInc/Dec`, `TaskReturn`, `TaskCancel`, -`WaitableSetNew/Wait/Poll/Drop`, `WaitableJoin`, `ThreadYield` (the other -`Thread*` built-ins are plan-representable but unimplemented, refused at -instantiate time), -`SubtaskDrop/Cancel`, `Stream*`, `Future*`, `ErrorContext*`, -`ContextGet/Set`. - -**Lazy materialization is the general rule**: trampolines/intrinsics are -materialized at first *reference during instantiation* — unreferenced -unsupported kinds never fail, referenced unsupported kinds fail at -instantiate time with a message naming the missing capability -("instantiate-time, never call-time", with the `PendingCapability` -carve-out of plan-format.md's executor obligations). - -Open discussion item (from `values/variants.wast:83`): one async-lifted -export makes a component's sync exports unreachable (instantiate-time -refusal of `task-return`). The rule is correct per the loud-failure -policy; a future change could permit lazily-trapping trampolines for -exports the embedder never calls — a deliberate silent-acceptance -tradeoff, not adopted without discussion. +Implemented groups are host import lowering; resource new/rep/drop and transfer; +transcoding; backpressure; task return/cancel; waitable sets and join; subtask +drop/cancel; stream/future operations; error contexts; context get/set; and +thread yield. Other explicit thread builtins are representable in the plan but +unsupported by the runtime. + +Trampolines are materialized on first reference during instantiation. +Unsupported referenced kinds fail then with a capability diagnostic; unused +entries do not prevent instantiation. A supported blocking operation may still +require JSPI at call time. The runtime's `createTrampoline` switch is the +current implementation inventory; [plan-format.md](plan-format.md) defines the +wire representation. ## Manifest -`plan.json` carries, per adapter module, its full import list categorized by -the table in §A (`modules[].intrinsics`), and the full `trampolines` table. -The runtime asserts coverage at instantiate time and reports which -capability is missing — "this component needs the task core" is a feature, -not a crash. +The plan carries the full `trampolines` table and each adapter's categorized +import list in `modules[].intrinsics`. Import resolution must either supply the +declared operation or report its missing capability; silently omitting an import +is not permitted. ## JSPI integration constraints (empirically derived) -Every one of these is pinned by a test under `runtime/tests/jspi/`; they -generalize to any JS host of the 0.3 task model. - -1. **The `current_thread()` ambient has no free implementation in JS.** - definitions.py resolves the running task via a thread-local, exact - because its threads are OS threads. A JSPI host gets resumed in a - microtask outside every frame it controls. Two working mechanisms, both - pinned: (h) `AsyncLocalStorage` propagates across a resumption (the - engine registers its continuation at suspension time, inside the host's - frame); (i) a resumed activation runs strictly before the host regains - control, so a single "resuming activation" slot is unambiguous between - settling a suspension and the host's next turn. The slot mechanism - needs no async-context support — the safer floor for a browser matrix. - (Probe discipline: a suspension probe must call the import at least - TWICE; a single-call fixture has no post-resumption observation point - and reads indistinguishably from "context lost".) -2. **"Task resolved" and "activation finished" are different events.** - The reference collapses them (threads run to completion synchronously - once resolved); a suspending host must model both. A guest may - `task.return` and keep executing — the wit-bindgen producer pattern. - Abandoning the activation on resolution leaks task state (exclusive - thread, table slots); waiting for it deadlocks producers. Required: a - detached-but-live activation the scheduler keeps servicing after the - export call returns. Corollary: **audit every piece of host state - assumed to nest within one export call** — the FACT sync-call bracket - stack is per-task because activations interleave. -3. **Resolve at most one suspension per scheduler turn.** Settling a - Suspending import's Promise hands control to wasm in a microtask: the - settling call returns with the activation not yet run. A tight - `while (tick())` drain — natural for a purely cooperative scheduler — - overwrites the ambient claim and mis-attributes the first activation's - built-ins. `tick()` must refuse progress while a claim is live; drains - must yield to the microtask queue. (The special case of constraint 5.) -4. **The Suspending "fast path" still suspends** (pin (j), - `fastpath_hop_test.ts`): a plain-value return from a Suspending import - does NOT continue the wasm synchronously — the continuation is deferred - to a microtask. Consequence: a promising-wrapped callee can never - complete inside the call that entered it, so any caller that must - observe an eager callee's completion (FACT `async-start-call` reporting - subtask state) must park until the callee is **determinate** - (resolved / finished / genuinely scheduler-parked) — never until - *resolution* (that parks an async caller on its callee, which is - forbidden by what async lowering means). -5. **Settled-tail atomicity**: between a suspension's settlement and the - servicing of that activation's continuation ("tail"), no other - scheduling decision may be made — the reference's synchronous - run-to-completion resume, reconstructed. Implemented as eager - settle-tagging (`Store.settled`), tick-refusal while a finished - activation's bookkeeping is unserviced, and tails-first ordering in - every driver. +The tests in `runtime/tests/jspi/` pin the engine behavior these rules rely on. + +1. **Explicit activation attribution.** Wasm resumes in a microtask outside the + JS frame that initiated suspension. The runtime retains an explicit + activation claim across that interval so context access and builtins refer to + the correct task/thread. This works without platform async-local storage. + Attribution probes need a second import call after resumption. +2. **Resolution is not thread exit.** `task.return` delivers a value while a + producer may remain alive. Return the value once the required activation + bookkeeping has settled; keep background threads available for later + scheduling. Waiting for every producer thread to exit can deadlock the + consumer of the returned stream or future. Call scopes therefore cannot be + assumed to nest inside one host export invocation. +3. **One resumption decision at a time.** Resolving a suspension schedules Wasm + on a later microtask. A synchronous drain must not schedule another + activation while the first activation's claim is outstanding. +4. **Plain-value returns still incur a JSPI hop.** A `Suspending` import that + returns a value does not resume Wasm synchronously (`fastpath_hop_test.ts`). + FACT async-call startup waits for the callee to become determinate: resolved, + finished, or parked at a scheduler condition. Cancellation's determinacy + predicate is stricter: resolution alone does not end its JSPI-hop wait. Async + lowering must not become a synchronous wait for its result. +5. **Settled tails precede new scheduling.** Promise settlement is tagged + eagerly. Drivers service completed activation bookkeeping before making + another scheduling decision, and must not deliver the same settlement twice. + A driver's completion verdict is latched rather than recomputed after another + driver has changed store state. ## Suspendability classification (current state) -Auto-detection is **ON by default**: all blocking sites lit, -per-declaration suspendability classification (async-form copy built-ins -never block; sync forms do; cancel forms per their own flag), -`KNOWN_DIVERGENT` empty, the plain path pinned zero-cost for sync-only -components. - -- One spot where wasmtime supersedes definitions.py is implemented and - tracked upstream: cancel-copy completion superseding - (upstream-component-model-repo-findings.md CM-3). -- Task-exclusivity gating implements the hold rule (gate lifetime = the - core invocation, the pristine definitions.py shape) plus a deferred - entry *decision*: an async-lowered call reports STARTING only if the - callee is still unstarted after the callee instance's runnable work is - drained to quiescence (`Store.hasRunnableWork`, consumed by - `createAsyncStartCall`'s determinacy park). `async-start-call` (and - `subtask.cancel`, for its own post-`on_cancel` determinacy park) is - `Suspending`-wrapped for the park, and **wrapped implies marked**: pin (c) - traps a Suspending import reached from a non-promising activation even - when it returns a plain value, so a guest importing one gets - promising-wrapped entries — as a separate evidence tier that neither - selects jspi mode nor promotes a FACT adapter's pass-through exports - (which would promising-wrap eagerly-completing lift callees). Plain mode - provably never - needs the drain (without JSPI a frame cannot park mid-invocation, so a - held gate always belongs to the currently-running activation — the one - obstacle a drain cannot remove), so the plain path stays zero-cost for - sync-only components. -- Host-import lowers participate in the classification - (contracts/embedder-api.md §"Functions and async"): a - `suspending()`-marked import is a genuine blocker — Suspending-wrapped, - importer-contaminating (transitive suspendability, so entries get - promising-wrapped per pin (c)), and evidence for auto-detection — with - the park implemented as `blockCurrentActivation` on the recorded - settlement (`readyFunc`-driven; result lowering deferred to `produce` so - realloc re-entry runs under the resume-time attribution claim). The park - is the reference's plain non-cancellable - `thread.wait_until(subtask.resolved)` (canon_lower line 2286); the gate - stays held across it. UNMARKED sync-lowered Promise-returning host - functions degrade to the clean `NeedsJspi` capability signal in every - mode — marking is the embedder's explicit, per-declaration opt-in, never - inferred. -- The wasi package's parking kernel narrows the zero-cost pin: its marked - `block`/`poll` are auto-detection evidence, so any wasi-consuming - component runs jspi mode on JSPI engines — the plain path stays - zero-cost for components that are sync-only AND import no marked - providers (or instantiate with `jspi: false`). - -Pinned by `runtime/tests/embedder/suspending_imports_test.ts` (park round -trip, resume-time realloc, pin-(c) start trap, refusal messages) and the -plain-mode guard in `runtime/tests/async_lower_test.ts`. +Auto-detection selects JSPI when supported and required by the plan or a marked +host import. Explicit `jspi: false` forces the plain path. Copy builtins are +classified by their sync/async form; cancellation forms follow their own flags. +Tests exercise the same corpus under both modes where applicable. + +- Async-call startup and cancellation may need determinacy parks across the JSPI + hop. Wrapping an import as `Suspending` also marks its importer suspendable: + even a plain return requires a `promising` entry. This propagation must not + eagerly wrap FACT pass-through callees that can otherwise finish + synchronously. +- Marked `suspending()` host imports park using recorded settlement and a + scheduler readiness predicate. Result lowering runs at resume time under the + activation claim. Unmarked sync imports returning Promises raise `NeedsJspi`; + the runtime never infers the mark from a returned value. +- WASI's marked blocking declarations select JSPI even when a particular call + completes immediately. The plain fast path applies to sync-only plans with no + marked imports, not to every call that happens not to wait. +- The async `subtask.cancel` determinacy park is a named non-atomicity + divergence. The synchronous form still waits for **resolution**, never merely + a STARTED event. Copy-cancel completion superseding is the separate CM-3 + exception. Both are documented in architecture §6. diff --git a/contracts/plan-format.md b/contracts/plan-format.md index 2a85e5e..c8e13ec 100644 --- a/contracts/plan-format.md +++ b/contracts/plan-format.md @@ -1,27 +1,23 @@ # Contract: Plan Format -The **plan** is the translator shim's output: everything the TS runtime needs -to instantiate and link one component, derived deterministically from the -component binary. This document is the interface between `crates/translator-shim` +The **plan** is the translator shim's output: everything the TS runtime needs to +instantiate and link one component, derived deterministically from the component +binary. This document is the interface between `crates/translator-shim` (producer) and `runtime/` (consumer); see also [descriptor-ir.md](descriptor-ir.md) and [intrinsics.md](intrinsics.md). -Current `formatVersion`: **5**. The compat rule is strict equality; any -change bumps `formatVersion` and updates producer and consumer in the same -commit. A stale cached artifact fails loudly rather than executing subtly -differently. +Current `formatVersion`: **5**. Loaders require strict equality. Schema changes +bump the version and update producer and consumer together; editorial changes to +this document do not change the wire format. ## Decisions (with rationale) -1. **Own schema, not wasmtime's.** `wasmtime_environ::component::Component` - derives `Serialize`, which makes the *shim's mapping code* cheap — but its - shape is an unstable internal API and is never exposed in the plan. The - plan schema is defined here and owned by us (docs/architecture.md §4.2). - The shim is the only code that sees both shapes. -2. **JSON encoding.** Debuggable, diffable, good enough. Revisit - (postcard / custom section) only on measured need. +1. **Own schema, not wasmtime's.** The shim maps wasmtime's unstable internal + API to this versioned format. Only the shim depends on both shapes. +2. **JSON encoding.** Inspectable and deterministic, without a separate + binary-format implementation. 3. **No duplicate bytes.** Embedded core modules are referenced as - `[offset, len)` byte ranges into the original component binary — the + `[offset, offset + len)` byte ranges into the original component binary — the executor slices them itself. Only FACT adapter modules (bytes that don't exist in the input) ship as separate artifacts. 4. **Types, not precomputed lanes.** The plan carries component-level types @@ -30,16 +26,19 @@ differently. ## Artifact set -A translation produces, content-addressed by -`sha256(component) x shim version x feature flags`: +A translation produces: ``` plan.json this document's schema adapters/.wasm FACT-generated core modules (kilobytes each) ``` -The original component binary is the third input at instantiation time; the -plan never embeds it. +The original component binary is the third input at instantiation time; the plan +never embeds it. + +The artifact cache keys by component hash, **shim binary hash**, and feature +flags. `producer.shimVersion` alone is not a build identity: two shim builds +with that version can produce different adapters. ## plan.json schema @@ -47,12 +46,12 @@ plan never embeds it. { "formatVersion": 5, "producer": { - "shimVersion": "…", // crates/translator-shim crate version + "shimVersion": "…", // crates/translator-shim crate version "wasmtimeEnviron": "49.0.0-dev+4675ee1", // crate version + pinned git rev - "features": ["cm-async", "…"] // wasmparser feature set used - // (incl. cm-fixed-length-lists, cm-map, - // cm-implements, cm-threading) — - // artifact-cache key input + "features": ["cm-async", "…"] // wasmparser feature set used + // (incl. cm-fixed-length-lists, cm-map, + // cm-implements, cm-threading) — + // artifact-cache key input }, "component": { "sha256": "…", "len": 123 }, @@ -60,8 +59,11 @@ plan never embeds it. // exactly as wasmtime-environ returns them (PrimaryMap). "modules": [ { "kind": "embedded", "offset": 10, "len": 52 }, - { "kind": "adapter", "file": "adapters/2.wasm", "len": 290, - "intrinsics": [ /* see intrinsics.md: required imports, categorized */ ] + { + "kind": "adapter", + "file": "adapters/2.wasm", + "len": 290, + "intrinsics": [/* see intrinsics.md: required imports, categorized */] } ], @@ -70,17 +72,25 @@ plan never embeds it. // instantiate-module | lower-import | extract-memory | extract-realloc | // extract-callback | extract-post-return | extract-table | resource "initializers": [ - { "op": "instantiate-module", "module": 0, - "instance": 0, // RuntimeComponentInstanceIndex; null = adapter - "args": [ /* CoreDef */ ] }, + { + "op": "instantiate-module", + "module": 0, + "instance": 0, // RuntimeComponentInstanceIndex; null = adapter + "args": [/* CoreDef */] + }, { "op": "lower-import", "index": 0, "import": 0 }, - { "op": "extract-memory", "index": 0, "export": { /* CoreExport */ } }, - { "op": "extract-realloc", "index": 0, "def": { /* CoreDef */ } }, - { "op": "extract-callback", "index": 0, "def": { /* CoreDef */ } }, - { "op": "extract-post-return", "index": 0, "def": { /* CoreDef */ } }, - { "op": "extract-table", "index": 0, "export": { /* CoreExport */ } }, - { "op": "resource", "index": 0, "rep": "i32", "dtor": { /* CoreDef? */ }, - "instance": 0 } + { "op": "extract-memory", "index": 0, "export": {/* CoreExport */} }, + { "op": "extract-realloc", "index": 0, "def": {/* CoreDef */} }, + { "op": "extract-callback", "index": 0, "def": {/* CoreDef */} }, + { "op": "extract-post-return", "index": 0, "def": {/* CoreDef */} }, + { "op": "extract-table", "index": 0, "export": {/* CoreExport */} }, + { + "op": "resource", + "index": 0, + "rep": "i32", + "dtor": {/* CoreDef? */}, + "instance": 0 + } ], // CoreDef encoding (wasmtime_environ::component::CoreDef, tag-for-tag): @@ -88,9 +98,6 @@ plan never embeds it. // { "kind": "instance-flags", "instance": n } i32 flags global // { "kind": "trampoline", "index": n } host trampoline // { "kind": "unsafe-intrinsic", "intrinsic": "" } - // (v5 removed `task-may-block`: wasmtime dropped the FACT-visible - // may-block global with #14146; sync-blocking is now enforced lazily by - // the scheduler, never by adapter code.) // The unsafe-intrinsic symbol is wasmtime's stable UnsafeIntrinsic::name() // ("context-get-i32-0", …), never the #[repr(u32)] ordinal (unstable // internal). All 21 variants are wire-representable. Executor obligation: @@ -110,39 +117,46 @@ plan never embeds it. // Host trampolines (ComponentTranslation::trampolines), one per // wasmtime_environ::component::Trampoline variant. Executors must fail - // loudly (at *instantiate* time, not call time) on unimplemented kinds — - // see the capability carve-out under "Executor obligations". Full kind - // list: see intrinsics.md §B. + // during instantiation on referenced unimplemented kinds. See "Executor + // obligations" for call-time capability errors. Full wire declarations: + // TrampolineDecl in the shim and WireTrampoline in runtime/src/plan/format.ts. "trampolines": [ - { "kind": "lower-import", "lowered": 0 /* LoweredIndex */, - "options": 0 /* -> canonicalOptions */, "type": 0 /* -> types */ }, - { "kind": "resource-drop", "instance": 0, "resource": 0 }, + { + "kind": "lower-import", + "index": 0, + "lowered": 0, /* LoweredIndex */ + "options": 0, /* -> canonicalOptions */ + "type": 0 /* -> types */ + }, + { "kind": "resource-drop", "index": 1, "instance": 0, "resource": 0 }, // FACT `runtime.trap` import, nullary: the trap code is static per // import site (wasmtime `Trampoline::Trap(Trap)`), so it rides in the // plan rather than as a call argument. `code` is wasmtime's `Trap` - // discriminant (`trap_encoding.rs`), the same numbering the pre-v5 - // i32 argument carried. - { "kind": "trap", "code": 24 }, - // Cooperative-threading built-ins (Explainer §thread.*, submodule pin - // 7c67611): `thread-index` and `thread-resume-later` carry `instance`; + // discriminant (`trap_encoding.rs`). + { "kind": "trap", "index": 2, "code": 24 }, + // Cooperative-threading built-ins: `thread-index` and + // `thread-resume-later` carry `instance`; // `thread-suspend`, `thread-yield`, `thread-suspend-then-resume`, // `thread-yield-then-resume`, `thread-suspend-then-promote`, - // `thread-yield-then-promote` carry `instance` + `cancellable`; - // `thread-new-indirect` is unchanged from v4. - { "kind": "thread-yield", "instance": 0, "cancellable": false }, - { "kind": "task-return", - "results": 0, // RAW wasmtime TypeTupleIndex — the FACT lookup key - // prepare-call passes at runtime - "resultType": 0, // interned plan.types index | null (null accepted on - // the wire; the producer always emits a tuple — a - // no-result task carries the empty tuple) - "options": 0 } + // `thread-yield-then-promote` carry `instance` + `cancellable`. + { "kind": "thread-yield", "index": 3, "instance": 0, "cancellable": false }, + { + "kind": "task-return", + "index": 4, + "instance": 0, + "results": 0, // RAW wasmtime TypeTupleIndex — the FACT lookup key + // prepare-call passes at runtime + "resultType": 0, // interned plan.types index | null (null accepted on + // the wire; the producer always emits a tuple — a + // no-result task carries the empty tuple) + "options": 0 + } // … ], // The loader builds the raw→interned task-return dictionary and rejects // contradictory mappings; the executor runs canon_task_return's // result-type check for FACT tasks (structural comparison against the - // task's declared result type, definitions.py:2395-2396). + // task's declared result type, definitions.py canon_task_return). // Canonical options table (Component::options), referenced by index from // trampolines and exports. Mirrors wasmtime_environ CanonicalOptions; @@ -150,13 +164,17 @@ plan never embeds it. // data_model: CanonicalOptionsDataModel::LinearMemory{memory, realloc} // (the Gc data model is rejected per descriptor-ir.md): "canonicalOptions": [ - { "instance": 0, "stringEncoding": "utf8", // utf8|utf16|latin1+utf16 - "memory": 0, // RuntimeMemoryIndex | null - "realloc": 0, // RuntimeReallocIndex | null - "postReturn": null, // RuntimePostReturnIndex | null - "callback": null, // RuntimeCallbackIndex | null - "async": false, "cancellable": false, - "coreType": { "params": ["i32","i32"], "results": ["i32"] } } + { + "instance": 0, + "stringEncoding": "utf8", // utf8|utf16|latin1+utf16 + "memory": 0, // RuntimeMemoryIndex | null + "realloc": 0, // RuntimeReallocIndex | null + "postReturn": null, // RuntimePostReturnIndex | null + "callback": null, // RuntimeCallbackIndex | null + "async": false, + "cancellable": false, + "coreType": { "params": ["i32", "i32"], "results": ["i32"] } + } ], // Component-level type table: descriptor-ir.md ValType/FuncType JSON. @@ -164,7 +182,7 @@ plan never embeds it. // families: ValTypes *and* function types tagged {"kind":"func", // "params": [{label,type}], "results": [...], "async": bool} — "func" is // not a ValType kind; consumers must discriminate. - "types": [ /* descriptor IR */ ], + "types": [/* descriptor IR */], // Resource tables, referenced by descriptor-IR own/borrow indices. // Index space = wasmtime TypeResourceTableIndex. @@ -176,15 +194,15 @@ plan never embeds it. // Imported resources, in ResourceIndex order; optional on the wire // (absent ⇒ empty). Executor obligation: // ResourceIndex = importedResources.length + DefinedResourceIndex. - "importedResources": [ { "import": 0 /* RuntimeImportIndex */ } ], + "importedResources": [{ "import": 0 /* RuntimeImportIndex */ }], // Stream/future tables: index spaces = wasmtime TypeStreamTableIndex / // TypeFutureTableIndex. Stream/future trampolines carry table indices; // these sections are what lets a consumer size and lift a copy buffer. // Digest-neutral: table sections do not enter the world digest (element // types reach it only via function types on the world surface). - "streamTables": [ { "element": /* ValType | null */ null, "instance": 0 } ], - "futureTables": [ { "element": /* ValType | null */ null, "instance": 0 } ], + "streamTables": [{ "element": /* ValType | null */ null, "instance": 0 }], + "futureTables": [{ "element": /* ValType | null */ null, "instance": 0 }], // Error-context tables: index space = // wasmtime TypeComponentLocalErrorContextTableIndex, emitted from @@ -193,16 +211,36 @@ plan never embeds it. // this section via a dedicated errorContextTableInstance(i) accessor — // never through resourceTables (loud PlanError on out-of-range, no ?? 0 // defaults). Digest-neutral. - "errorContextTables": [ { "instance": 0 } ], + "errorContextTables": [{ "instance": 0 }], // World surface. Import names use the component's exact import strings; // runtime import indices match wasmtime's RuntimeImportIndex order. // Import entries carry "path": string[] — wasmtime's RuntimeImportIndex // is (ImportIndex, Vec) walking into instance imports. - "imports": [ { "name": "…", "kind": "func", "type": 0, "path": [] } ], + "imports": [{ "name": "…", "kind": "func", "type": 0, "path": [] }], "exports": [ - { "kind": "lifted-func", "name": "greet", - "coreDef": { /* CoreDef */ }, "options": 0, "type": 0 }, + { + "kind": "lifted-func", + "name": "greet", + "coreDef": {/* CoreDef */}, + "options": 0, + "type": 0 + }, + { + "kind": "instance", + "name": "ns:pkg/interface", + "exports": [/* recursive */] + }, + { + "kind": "type", + "name": "resource-name", + "type": { "kind": "resource", "resource": 0 } + }, + { + "kind": "type", + "name": "value-name", + "type": { "kind": "value", "type": 0 } + }, // A component exporting one of its own embedded core modules // (wasmtime Export::ModuleStatic); n indexes plan.modules and names an // *embedded* entry by construction (FACT adapters are appended after @@ -220,41 +258,38 @@ plan never embeds it. Notes on specific entries: -- **Type exports index into `resourceTables`, not the `ResourceIndex` - space.** An export/import entry `{"kind": "type", "resource": n}` - carries a *resource-table* index (`TypeResourceTableIndex`, the same - space as descriptor-IR `own`/`borrow`), **not** a `ResourceIndex`. - Consequence: one resource type can be reachable through several distinct - table indices — e.g. a type export pointing at table 1 while the - functions' handles use table 0, both resolving to the same - `ResourceIndex` via `resourceTables[n].resource`. Consumers keying - per-resource state must key by the resolved `ResourceIndex`, treating - table indices as aliases. +- **Type exports index into `resourceTables`, not the `ResourceIndex` space.** + An export's `type: {"kind": "resource", "resource": n}` carries a + _resource-table_ index (`TypeResourceTableIndex`, the same space as + descriptor-IR `own`/`borrow`), **not** a `ResourceIndex`. Consequence: one + resource type can be reachable through several distinct table indices — e.g. a + type export pointing at table 1 while the functions' handles use table 0, both + resolving to the same `ResourceIndex` via `resourceTables[n].resource`. + Consumers keying per-resource state must key by the resolved `ResourceIndex`, + treating table indices as aliases. - **Module exports**: the executor surfaces the export as the platform's - compiled-module value — `WebAssembly.Module` in the JS runtime — reusing - the compilation the instantiation path already performs. Module exports - are **excluded** from the canonical world digest (digest.md's item rule: - only functions and resources contribute as export/import *items*; a - module export is not WIT-expressible and does not affect - positional-calling ABI shape, so a digest match stays ABI-sound). The - WIT-shaped conventions facade skips them (the type-export precedent); - they are available on the raw executor export surface only. -- **`Export::ModuleImport`** (re-export of an *imported* module) is - rejected at translation with a precise message. No conformance test - exercises it, and module *imports* have no instantiation story in the - runtime; lift both together if a consumer ever needs them. + compiled-module value — `WebAssembly.Module` in the JS runtime — reusing the + compilation the instantiation path already performs. Module exports are + **excluded** from the canonical world digest (digest.md's item rule: only + functions and resources contribute as export/import _items_; a module export + is not WIT-expressible and does not affect positional-calling ABI shape, so a + digest match stays ABI-sound). The WIT-shaped conventions facade skips them + (the type-export precedent); they are available on the raw executor export + surface only. +- **`Export::ModuleImport`** (re-export of an _imported_ module) is rejected at + translation as unsupported, as is imported-module instantiation. - **Structured error envelope**: translation failures emit `{"error": "", "errorDetail": {"phase": "validation" | - "unsupported" | "internal", "message", "detail"?}}`. `errorDetail` is - additive (consumers tolerate its absence); only `phase: "validation"` - may be scored as a correct + "unsupported" | "internal", "message", "detail"?}}`. + `errorDetail` is additive (consumers tolerate its absence); only + `phase: "validation"` may be scored as a correct `assert_invalid`/`assert_malformed` verdict. Body-validation failures in FACT-generated (non-embedded) modules classify as `internal`, never `validation`. - Runtime instance/memory/realloc **counts are derivable, not carried**; executors create state lazily. -- Adapter naming = static-module index; embedded `wasm_module_offset` - equals slice position (shim-asserted); `NameMap`/`IndexMap` iteration is +- Adapter naming = static-module index; embedded `wasm_module_offset` equals + slice position (shim-asserted); `NameMap`/`IndexMap` iteration is insertion-ordered (determinism holds). ## Determinism @@ -271,32 +306,24 @@ nondeterminism as a bug. - Execute `initializers` strictly in order; each op's semantics follow wasmtime-environ's documented behavior for the corresponding `GlobalInitializer` variant. -- Instantiate-time (not call-time) failure for any trampoline kind, - intrinsic, or op the executor doesn't support — with one carve-out: - capability-scoped built-ins whose absence affects only the exports that - use them (stream / future / error-context) may instantiate successfully - and fail at first call. That failure must be `PendingCapability`-shaped, - never a `Trap` (so it can never satisfy a conformance trap assertion). - Rationale: wit-bindgen guests routinely mix supported callback-ABI - exports with stream exports; instantiate-time refusal would make - supported exports unreachable over a capability their code never - touches. +- Referenced unsupported trampolines, intrinsics, and operations fail during + instantiation. Unreferenced table entries need not be materialized. + Unsupported runtime operations must report capability errors, not guest traps. + In particular, a blocking path reached without JSPI can fail at call time with + `NeedsJspi`; streams, futures, and error-context values themselves are + implemented. - Verify the canonical world digest when typed bindings are in play (docs/architecture.md §9, digest.md). -- The shim must fail translation with a clear error on any - wasmtime-environ construct not representable in this format (never - silently drop). +- The shim must fail translation with a clear error on any wasmtime-environ + construct not representable in this format (never silently drop). ## Open items -- Resource table details beyond dtor wiring (borrow bookkeeping lives in - the runtime; revisit when the shim emits resource-rich components). -- `values` section (the component-level value-definition feature): out of - scope (wasmtime parity, docs/architecture.md §7). -- Imported-module instantiation (`InstantiateModule::Import`) — not - emitted for our corpus; shim rejects with a clear error until - implemented. +- `values` section (the component-level value-definition feature): out of scope + (wasmtime parity, docs/architecture.md §7). +- Imported-module instantiation (`InstantiateModule::Import`) and re-export + remain unsupported. - The memory-identity half of `canon_task_return`'s options-equality check - remains a named open gap: `prepare-call.memory` is the adapter's - second-hand view and wasmtime's own check is one-sided — re-justified at - the site (intrinsics/fact_calls.ts / async_builtins.ts CONTRACT notes). + remains a named open gap: `prepare-call.memory` is the adapter's second-hand + view and wasmtime's own check is one-sided — re-justified at the site + (intrinsics/fact_calls.ts / async_builtins.ts CONTRACT notes). diff --git a/crates/bindgen/src/digest.rs b/crates/bindgen/src/digest.rs index 3101e70..3fb266a 100644 --- a/crates/bindgen/src/digest.rs +++ b/crates/bindgen/src/digest.rs @@ -2,13 +2,10 @@ //! //! See `runtime/src/digest/digest.ts` for the TypeScript counterpart, which //! must compute byte-identical canonical JSON (and therefore identical -//! sha256) from a *loaded plan* for the same WIT world. The full -//! normalization spec is documented in both files' module comments and is -//! the primary deliverable of this track (resolves the legacy shim -//! `worldDigest`, contracts/plan-format.md schema; the normative digest is -//! contracts/digest.md). +//! sha256) from a loaded plan for the same WIT world, within the supported +//! subset described below. contracts/digest.md is the normative digest spec. //! -//! ## Normalization spec (authoritative copy; keep in sync with digest.ts) +//! ## Normalization (keep in sync with digest.ts) //! //! Goal: two structurally-equivalent worlds — one parsed from WIT source, //! one recovered from a translated component's plan — must hash identically, diff --git a/crates/bindgen/tests/import_base.rs b/crates/bindgen/tests/import_base.rs index fdd786d..a57368a 100644 --- a/crates/bindgen/tests/import_base.rs +++ b/crates/bindgen/tests/import_base.rs @@ -1,12 +1,5 @@ //! Import-base resolution tests (issue #201). //! -//! Generated bindings used to hardcode `../../../src/...` specifiers, -//! calibrated to exactly one output directory in exactly one checkout — so -//! a binding written anywhere else had unresolvable imports, and the -//! world-digest handshake that `contracts/embedder-api.md` §"Module wiring -//! and instantiation" scopes to the generated typed entry point was -//! unreachable for consumers. -//! //! Three properties are pinned here: //! a. depth independence, proven end-to-end with `deno check`; //! b. the default base's version stays in sync with `runtime/deno.json`; @@ -124,10 +117,8 @@ fn header_records_the_import_base() { ); } -/// (a) The regression test this issue asks for: generate into a scratch -/// directory at a *different depth* from `runtime/tests/bindgen/generated/` -/// and prove the result typechecks. Under the old hardcoded `../../../src` -/// prefix this fails for any `--out` at another depth. +/// (a) Generate at a different depth from `runtime/tests/bindgen/generated/` +/// and verify that the imports resolve with `deno check`. #[test] fn generated_bindings_typecheck_at_an_unrelated_depth() { let root = repo_root().canonicalize().unwrap(); diff --git a/crates/translator-shim/README.md b/crates/translator-shim/README.md index 40cc781..5eaaa61 100644 --- a/crates/translator-shim/README.md +++ b/crates/translator-shim/README.md @@ -1,163 +1,91 @@ # translator-shim -`wasmtime-environ`'s component frontend (validate, resolve linkage, FACT -fused-adapter synthesis) behind the stable **plan v0** output format of -[contracts/plan-format.md](../../contracts/plan-format.md). Promoted from the -translator-spike (an earlier prototype crate); docs/architecture.md §4.1/§4.2. - -This crate is the only code in the repository that sees wasmtime's unstable -internal shapes; everything it emits is our own schema. Pinned: -`wasmtime-environ` (pinned git rev, see root Cargo.toml; crate version -49.0.0-dev), `wasmparser` 0.258 (wasm-tools). - -## Artifact set vs C-ABI envelope - -The contract defines the translation output as an artifact set: - -``` -plan.json (schema: contracts/plan-format.md) -adapters/.wasm FACT-generated core modules -``` - -Over the wasm C-ABI (`ts_translate`), the shim returns the same artifact set -packed into **one JSON envelope** — a shim-internal wire format, mapped 1:1: - -| Envelope field | Contract artifact | -|---------------------|-------------------------------------------------------| -| `plan` | the `plan.json` document (as a JSON object; its byte form is this crate's serde serialization, which is deterministic) | -| `adapters[i].file` | the artifact path, identical to `plan.modules[].file` (`adapters/.wasm`) | -| `adapters[i].wasm` | that artifact's bytes, base64 (standard alphabet, padded) | -| `error` | (error envelopes only) failure message (unchanged v0.1 meaning) | -| `errorDetail` | (error envelopes only) structured verdict `{phase, message, detail}` — see "Verdicts" below. **contracts v0.2 proposal**; additive, v0.1 consumers ignore it | - -Nothing else is in the envelope; consumers that want the on-disk artifact set -write `plan` and the decoded adapters out verbatim. The runtime-side decoder -is `runtime/src/plan/loader.ts` (`loadEnvelope`). - -Determinism (contract requirement): plan JSON uses fixed struct-field order, -no maps, integers only; adapter bytes are FACT output, deterministic for a -pinned toolchain. `translate twice ⇒ byte-identical envelope` is asserted by -`tests/translate.rs::determinism` and by the runtime e2e test. - -## Verdicts (`src/error.rs`) - -`translate` fails with a `TranslateError { phase, message, detail }`. The -`phase` is what makes the official suite's `assert_invalid` / -`assert_malformed` commands decidable: - -| phase | meaning | may be scored as a correct rejection? | -|---|---|---| -| `validation` | wasmtime's frontend rejected the input: the component is invalid or malformed | **yes** | -| `unsupported` | valid component, shape not representable in the plan format (module exports, `InstantiateModule::Import`, GC data model, …) | no — triage item | -| `internal` | shim invariant broken | no — bug | - -`assert_malformed` (decoding) and `assert_invalid` (type checking) are *not* -split: wasmparser reports both as `BinaryReaderError` and the distinction is -not recoverable without matching wasmtime's message text. Both are -`validation`, which is what both commands require. - -Note that `Translator::translate` does **not** validate core function -*bodies* — wasmtime defers that to its compiler backend, which we do not -have. The shim therefore runs the deferred `FuncToValidate`s itself; without -that, a component with an invalid nested core module (official suite -`test/validation/core-modules.wast:24`) would translate successfully and only -be rejected later by the JS engine. - -## API surface - -- `translate(&[u8]) -> Result` - — library. -- `to_envelope_json(&Translation) -> Result` — envelope. -- `cabi`: `ts_alloc` / `ts_translate` / `ts_dealloc` (wasm32 C-ABI; contract - in `src/lib.rs`). -- `examples/dump-plan.rs` — debugging: dump a component's plan - (`cargo run -p translator-shim --example dump-plan [--full]`). -- `examples/emit-testdata.rs` — regenerate `testdata/.wasm` from its - `.wat` using the pinned `wat` crate, for fixtures whose syntax is newer than - the installed `wasm-tools` CLI - (`cargo run -p translator-shim --example emit-testdata -- relend-borrow`). -- `examples/suite-inventory.rs` — triage: translate every component artifact - of the official suite (`cargo run -p testgen` first) and report - translated/rejected counts, rejection phases and the plan features seen per - directory (`cargo run -p translator-shim --example suite-inventory`). -- `driver.ts` — Deno smoke driver over the wasm32 build - (`deno run --allow-read driver.ts`). - -## Build - -``` -cargo test -p translator-shim # native tests -cargo build -p translator-shim --release --target wasm32-unknown-unknown -deno run --allow-read driver.ts # smoke -``` - -Some tests use the example fixture corpus (`examples/guests/build/*.wasm`, -gitignored); they skip with a notice unless `./examples/build.sh` has run. - -Size note: the default release wasm32 build is ~3.4 MiB. The translator-spike's -"1.66 MiB size-tuned" figure used size flags; reproduce without editing the -workspace manifest via: - -``` -cargo build -p translator-shim --release --target wasm32-unknown-unknown \ - --config 'profile.release.opt-level="z"' \ - --config profile.release.lto=true \ - --config profile.release.codegen-units=1 \ - --config 'profile.release.panic="abort"' \ - --config 'profile.release.strip=true' +The `wasmtime-environ` component frontend compiled as a native Rust library and +a wasm32 translator. It validates components, resolves linkage, synthesizes +FACT fused adapters, and emits **plan format 5**. The +[plan contract](../../contracts/plan-format.md) owns the schema; +[architecture](../../docs/architecture.md) explains the frontend/runtime split. +Dependency pins live in the root `Cargo.toml` and `Cargo.lock`. + +## Translation output + +The artifact set is `plan.json` plus `adapters/.wasm`. +Embedded core modules remain slices of the original component, so consumers +must retain that component binary too. + +The wasm C ABI returns one JSON envelope: + +| Field | Meaning | +| --- | --- | +| `plan` | Plan object, including component identity and producer metadata | +| `adapters[i].file` | Artifact path matching the plan's module entry | +| `adapters[i].wasm` | Adapter bytes encoded as padded standard base64 | +| `error` | Error envelope only: human-readable failure message | +| `errorDetail` | Error envelope only: `{phase, message, detail}` | + +Success and error fields are alternative envelope shapes. The runtime decoder +is [`loadEnvelope`](../../runtime/src/plan/loader.ts). Serialization and adapter +generation are deterministic for a fixed input and toolchain; native tests +compare repeated translations byte-for-byte. + +## Validation and corrections + +[`src/error.rs`](src/error.rs) separates three outcomes: + +| Phase | Meaning | Conformance rejection evidence? | +| --- | --- | --- | +| `validation` | Input rejected by the configured frontend or embedded core-body validation | Yes, within that feature configuration | +| `unsupported` | Accepted frontend output cannot be represented by this translator | No | +| `internal` | Translation or adapter-correction invariant failed | No | + +Malformed binary and type-invalid input share `validation`; callers must not +infer a finer distinction from message text. Validation is feature-configured: +`features()` in `src/lib.rs` enables the async extensions and other supported +proposals, but deliberately leaves `CM_VALUES` disabled to avoid unsupported +component value/start paths in the pinned frontend. A feature-disabled rejection +does not establish that the component is invalid under every spec feature set. + +Wasmtime's frontend defers core function-body validation to its compiler. +`validate_function_bodies` runs those validators here: invalid embedded bodies +are `validation` errors, while invalid FACT-generated bodies are `internal` +errors. Imported core-module instantiation/re-export and GC canonical options +are examples of unsupported mappings; static core-module exports are represented. + +[`fact_string_limits::correct`](src/fact_string_limits.rs) corrects the pinned +FACT generator's pre-realloc string checks to limit **source bytes** to +`2^28 - 1`, rather than destination width or retry-allocation size. It rewrites +only generated adapters, never embedded guest modules. The correction checks +the dependency pin and expected instruction/control-flow shapes, preserves +module length, and validates the corrected module. Drift is an internal error, +not a validation verdict. The integration regression is +[`fact_string_source_limits_test.ts`](../../runtime/tests/fact_string_source_limits_test.ts). + +## APIs and tools + +- `translate(&[u8]) -> Result` returns a plan and + adapter bytes. +- `to_envelope_json(&Translation)` serializes successful output; + `translate_to_envelope(&[u8])` returns a success or structured error envelope. +- The wasm exports are `ts_alloc`, `ts_translate`, and `ts_dealloc`; pointer + ownership and lengths are documented at `cabi` in [`src/lib.rs`](src/lib.rs). + This is a trusted pointer ABI, not a guarantee that arbitrary pointers or + upstream aborts can be converted to JSON errors. +- [`dump-plan`](examples/dump-plan.rs) inspects a component's plan; + [`emit-testdata`](examples/emit-testdata.rs) builds fixtures with the pinned + `wat` parser; [`suite-inventory`](examples/suite-inventory.rs) inventories + translation results from a generated WAST corpus. + +From the repository root: + +```sh +just shim +cargo test -p translator-shim +deno run --allow-read crates/translator-shim/driver.ts +cargo run -p translator-shim --example dump-plan -- component.wasm --full ``` -## wasmtime-environ API notes (feeds contract v0.1; pinned rev, see root Cargo.toml) - -Recorded here because plan-format.md left these underspecified; the mapping -code is `src/plan.rs`. - -- **Types**: obtained by calling `ComponentTypesBuilder::finish(&component)` - *after* translation; the resulting `ComponentTypes` is `Index`able by every - `Type*Index` and by `ModuleInternedTypeIndex` (core signatures for - `canonicalOptions.coreType`). Counts like `num_resource_tables()` exist - only on the builder — capture before `finish()`. -- **`Export` enum reality**: `LiftedFunction { ty: TypeFuncIndex, func: - CoreDef, options: OptionsIndex }`, `Instance { ty, exports: NameMap }` - (recursive), `Type(TypeDef)`, `ModuleStatic`/`ModuleImport` (rejected in - v0). Options live in `Component::options` - (`PrimaryMap`), *not* inline. -- **`CanonicalOptions` reality**: `memory`/`realloc` are nested inside - `data_model: CanonicalOptionsDataModel::LinearMemory(LinearMemoryOptions)`; - `Gc {}` is rejected. `StringEncoding::CompactUtf16` maps to the contract's - `latin1+utf16`. -- **`ExportItem::Index(EntityIndex)`** cannot be consumed by a JS embedder - (exports are name-addressed); the shim resolves indices to names via - `Module::exports` and emits `{ name, space }`. -- **FACT adapter imports** are already folded to `CoreDef`s by - `translate/adapt.rs` (`fact_import_to_core_def`): every §A intrinsic - arrives as `CoreDef::Trampoline` (Trap, EnterSyncCall, Transcoder, ...) or - a plain CoreDef (callee funcs, memories, instance-flags globals). The - per-adapter `intrinsics` manifest is those args zipped with the adapter's - import names, categorized by trampoline kind. -- **Instance flags**: FACT treats the flags global as a plain boolean - `may_leave` (no bit masks); initial value 1. -- **`ResourceDrop` has no `async` field** (`{ instance, ty }`) — - plan-format.md's example shows one. -- **`Component::imported_resources: PrimaryMap`** is emitted as the plan's `importedResources` - (contracts v0.2 proposal). `Component::resource_index` (`info.rs`) is - the mapping the runtime must reproduce: - `ResourceIndex = importedResources.len() + DefinedResourceIndex` (line - number not re-verified at the pinned rev). -- **Feature gates**: beyond the async set, the shim enables - `cm-fixed-length-lists`, `cm-map`, `cm-implements` and `cm-threading`, - because the official suite contains components it expects to *decode* which - use them. Verified over the whole corpus that this turns no - `assert_invalid`/`assert_malformed` case into an acceptance - (`--example suite-inventory`). -- **`CoreDef::UnsafeIntrinsic` is real**: wit-bindgen 0.60 async guests - (`context.get`/`context.set`) produce it (variants - `context-{get,set}-i32-{0,1}`, slot 0 in practice — that is where the - generated async executor keeps its task pointer). Plan **v1** - (`formatVersion: 1`, contracts/plan-format.md v0.3) emits it as - `{"kind": "unsafe-intrinsic", "intrinsic": ""}`, carrying - wasmtime's stable `UnsafeIntrinsic::name()` rather than the `#[repr(u32)]` - ordinal. All 21 variants are representable; the runtime implements the four - `context-*` ones and refuses the rest (raw host-memory access) at - instantiate time. +`just shim` builds the size-tuned wasm artifact and copies it into the translator +package. Some native tests skip when guest fixtures are missing; `just fixtures` +builds that corpus. The standalone smoke driver uses the checked-in `testdata/` +fixtures by default. For the full translation/runtime path, use +`just test-runtime` and `just conformance`. diff --git a/crates/translator-shim/src/error.rs b/crates/translator-shim/src/error.rs index 4a8d609..a6d0e83 100644 --- a/crates/translator-shim/src/error.rs +++ b/crates/translator-shim/src/error.rs @@ -8,8 +8,8 @@ //! saying no. This, and only this, is the verdict that satisfies the //! official suite's `assert_invalid` / `assert_malformed` commands. //! - **`unsupported`** — the component is valid, but uses a shape this -//! plan-format version cannot represent yet (async `CoreDef`s, GC data -//! model, module exports, …). A conformance run must *not* score these as +//! plan-format version cannot represent (e.g. the GC data model or +//! re-exported imported modules). A conformance run must *not* score these as //! correct rejections; they are triage items. //! - **`internal`** — a shim invariant broke. Always a bug here. //! @@ -19,12 +19,6 @@ //! string-matching wasmtime's messages. Both map to `validation`, which is //! sound for the suite (a rejection with the right phase is what both //! commands require). -// -// CONTRACT: contracts/plan-format.md v0.1 specifies the C-ABI error envelope -// as `{"error": ""}` with "no other field present". This module adds -// a sibling `errorDetail` object; `error` keeps its exact v0.1 meaning and -// value, so v0.1 consumers are unaffected. Proposed for contracts v0.2 (see -// the track report). use std::fmt; @@ -117,7 +111,7 @@ impl TranslateError { } } -/// Marker error: a valid component whose shape plan v0 cannot represent. +/// Marker error: a valid component whose shape the plan cannot represent. /// Constructed via [`unsupported!`]. #[derive(Debug)] pub struct Unsupported(pub String); diff --git a/crates/translator-shim/src/lib.rs b/crates/translator-shim/src/lib.rs index f989b3c..5c30363 100644 --- a/crates/translator-shim/src/lib.rs +++ b/crates/translator-shim/src/lib.rs @@ -1,9 +1,8 @@ //! translator-shim: `wasmtime-environ`'s component frontend (validation, //! linking resolution, FACT fused-adapter synthesis) behind a stable output -//! format — the **plan v0** of `contracts/plan-format.md`. +//! format defined in `contracts/plan-format.md`. //! -//! Promoted from translator-spike (`crates/translator-spike`). The spike's debug -//! `Summary` is replaced by the contract artifact set: +//! Translation produces the contract artifact set: //! //! - `plan.json` — the plan (schema: `src/plan.rs`) //! - `adapters/.wasm` — FACT-generated core modules @@ -65,7 +64,7 @@ pub struct Translation { /// Wasm feature set used for validation during translation. /// -/// `wasmparser` 0.252 defaults already include `component_model` and +/// `wasmparser` defaults already include `component_model` and /// `cm_async` (component-model-async). We additionally enable the async /// trailing features (mirroring wasmtime's /// `-W component-model-async=y,component-model-error-context=y`) and the @@ -74,9 +73,7 @@ pub struct Translation { /// suite files contain components the suite expects to *decode* (`module` / /// `module_definition` commands); with the gates off the shim rejects them /// with a feature-gate error, which is not a conformance verdict we want to -/// claim. Enabling them was checked against the whole corpus: it does not -/// turn any `assert_invalid`/`assert_malformed` case into an acceptance -/// (`cargo run -p translator-shim --example suite-inventory`). +/// claim. `suite-inventory` checks validation outcomes against the corpus. fn features() -> wasmparser::WasmFeatures { let mut f = wasmparser::WasmFeatures::default(); f.insert(wasmparser::WasmFeatures::CM_ASYNC); @@ -130,7 +127,7 @@ fn feature_names() -> Vec { .to_vec() } -/// Translate a component binary into plan v0 + adapter artifacts. +/// Translate a component binary into a plan and adapter artifacts. /// /// Runs wasmtime's full component frontend: parse + validate + type-check the /// component, resolve its linking structure to a flat initializer list, run @@ -378,10 +375,8 @@ pub fn translate_to_envelope(component_bytes: &[u8]) -> String { /// The C-ABI error envelope. /// -// CONTRACT: contracts/plan-format.md / translator-shim README pin the error -// envelope as `{"error": ""}` and "no other field present". The -// `error` string keeps exactly that meaning; `errorDetail` is an additive -// sibling carrying the structured verdict (phase + message). v0.2 proposal. +// `error` carries the message; `errorDetail` carries the structured verdict +// (phase + message), per contracts/plan-format.md. pub fn error_envelope_json(e: &TranslateError) -> String { serde_json::to_string(&serde_json::json!({ "error": e.message, @@ -392,7 +387,7 @@ pub fn error_envelope_json(e: &TranslateError) -> String { /// C-ABI surface for the wasm32 build (used by the Deno driver). /// -/// Contract (unchanged from the spike): +/// Contract: /// - `ts_alloc(len) -> ptr`: allocate `len` bytes (caller writes input here). /// - `ts_translate(ptr, len, out_len: *mut usize) -> out_ptr`: translate the /// component at `ptr..ptr+len`. Writes the output length to `*out_len` and diff --git a/crates/translator-shim/src/plan.rs b/crates/translator-shim/src/plan.rs index 9e91e06..44f8173 100644 --- a/crates/translator-shim/src/plan.rs +++ b/crates/translator-shim/src/plan.rs @@ -1,13 +1,10 @@ //! Mapping from `wasmtime_environ::component` translation output to the -//! plan v0 schema of `contracts/plan-format.md`. +//! schema of `contracts/plan-format.md`. //! -//! This module is the only code that sees both shapes (wasmtime's unstable -//! internals and our stable plan format); everything wasmtime-specific is -//! confined here per docs/architecture.md §4.1. +//! This module maps wasmtime's internal index spaces and types to the plan +//! consumed by the JavaScript runtime. //! -//! wasmtime-environ API reality this maps from (pinned rev, see root -//! Cargo.toml; recorded for the -//! contract-v0.1 review; see crate README): +//! Relevant wasmtime-environ shapes (revision pinned in root Cargo.toml): //! //! - `Component::exports: NameMap` //! + `export_items: PrimaryMap`; `Export` variants are @@ -41,48 +38,6 @@ use wasmtime_environ::component::{ use wasmtime_environ::{EntityIndex, ModuleInternedTypeIndex, PrimaryMap, WasmValType}; /// `formatVersion` this producer emits (contracts/plan-format.md). -/// -/// v1 (contracts/plan-format.md v0.3): additive — `CoreDef` gained the -/// `"unsafe-intrinsic"` variant (previously a hard `unsupported` rejection). -/// -/// v2: additive — `streamTables` / `futureTables`, mapping the -/// `streamTable` / `futureTable` indices the stream and future trampolines -/// already carried to their *element types*. Without them a consumer knows a -/// `stream.read` targets table 3 but not what a table-3 element is, so it -/// cannot size or lift the copy buffer at all. The same gap in the other -/// direction (`task_return_type`, a `TypeTupleIndex` with no mapping into -/// `plan.types`) was still open at v2 — see below. -/// -/// v3 (contracts/plan-format.md schema: the errorContextTables section and -/// the task-return raw `results` + interned `resultType` keys): closes both -/// v2 gaps. -/// * `errorContextTables` — the `error-context-transfer` trampoline's table -/// arguments live in the `TypeComponentLocalErrorContextTableIndex` space -/// and had no section, so the runtime resolved them through the -/// *resource*-table mapping (a different index space: silent mis-route in -/// a composition with an ErrorContext at a colliding slot). -/// * `task-return`'s `resultType` / raw `results` — see -/// `TrampolineDecl::TaskReturn`. -/// -/// Per the contract's compat rule ("changes require updating both producer -/// and consumer in the same commit and bumping `formatVersion`") the bump is -/// unconditional even though the change is additive. -/// -/// v4 (2026-08-17, polyengine#13): additive — `exports[]` gained the `"module"` -/// kind (`Export::ModuleStatic`, a component exporting one of its own -/// embedded core modules; previously a hard `unsupported` rejection). -/// `Export::ModuleImport` remains rejected, now with a precise message. -/// -/// v5: tracks the wasmtime-environ rev bump (see root Cargo.toml). Breaking, -/// not additive: `CoreDef::TaskMayBlock` was removed upstream (the -/// `"task-may-block"` core-def kind is gone); `Trampoline::Trap` gained a -/// `Trap` payload (`"trap"` trampolines now carry a `code` byte); the thread -/// trampoline set was renamed/expanded (`thread-suspend-to-suspended`, -/// `thread-suspend-to`, `thread-unsuspend`, `thread-yield-to-suspended` are -/// gone; `thread-resume-later`, `thread-suspend-then-resume`, -/// `thread-yield-then-resume`, `thread-suspend-then-promote`, -/// `thread-yield-then-promote` are new; `thread-index` gained an `instance` -/// field). pub const FORMAT_VERSION: u32 = 5; // --------------------------------------------------------------------------- @@ -102,38 +57,29 @@ pub struct Plan { pub types: Vec, /// Resource-table metadata referenced by `own`/`borrow` `resource` fields /// and by resource trampolines. Index space == wasmtime's - /// `TypeResourceTableIndex`. (Extension over the letter of plan-format.md, - /// which references "the plan's resource table" without defining it.) + /// `TypeResourceTableIndex`. pub resource_tables: Vec, /// Stream-table metadata, index space == wasmtime's /// `TypeStreamTableIndex`; referenced by the `streamTable` field of every /// `stream.*` trampoline. `element` is the `T` of `stream`, absent for - /// the zero-width payload (`stream`). Plan v2. + /// the zero-width payload (`stream`). pub stream_tables: Vec, /// Future-table metadata, index space == wasmtime's /// `TypeFutureTableIndex`; referenced by the `futureTable` field of every - /// `future.*` trampoline. Plan v2. + /// `future.*` trampoline. pub future_tables: Vec, /// Error-context-table metadata, index space == wasmtime's /// `TypeComponentLocalErrorContextTableIndex`; the space the /// `error-context-transfer` trampoline's `srcTable`/`dstTable` runtime - /// arguments live in (fact/trampoline.rs:3526-3539). Emitted from - /// `ComponentTypes::error_context_tables` (`TypeErrorContextTable`, - /// types.rs:1144-1151), which carries nothing but the owning instance — - /// hence no `element` here. Plan v3 — the errorContextTables section - /// (contracts/plan-format.md schema). + /// arguments live in. Emitted from `ComponentTypes::error_context_tables` + /// (`TypeErrorContextTable`), which carries only the owning instance, + /// hence no `element` here. pub error_context_tables: Vec, /// Resource types this component *imports*, in `ResourceIndex` order /// (entry `i` is `ResourceIndex(i)`). Defined resources follow: /// `ResourceIndex = importedResources.len() + DefinedResourceIndex`, /// exactly wasmtime's `Component::resource_index` /// (the pinned wasmtime-environ rev, see root Cargo.toml; `component/info.rs`). - /// - /// The `importedResources` field (contracts/plan-format.md schema) - /// closes this gap; the field - /// is a **v0.2 proposal**. Emitting it is - /// purely additive — v0.1 consumers ignore it, and it is empty for every - /// component that imports no resource type, which is every current fixture. pub imported_resources: Vec, pub imports: Vec, pub exports: Vec, @@ -236,9 +182,8 @@ pub struct AsyncTableDecl { pub instance: u32, } -/// One entry of the `errorContextTables` section (plan v3). An error-context -/// table has no element type — wasmtime's `TypeErrorContextTable` -/// (types.rs:1144-1151) is exactly `{ instance }`. +/// One entry of `errorContextTables`. An error-context table has no element +/// type: wasmtime's `TypeErrorContextTable` is exactly `{ instance }`. #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct ErrorContextTableDecl { @@ -266,9 +211,9 @@ pub enum CoreDefJson { /// are an unstable internal detail, while `name()` yields the stable /// spec-facing symbol (`"context-get-i32-0"`, `"u32-native-load"`, …). /// - /// All 21 variants are representable on the wire; the executor implements + /// All variants are representable on the wire; the executor implements /// only the four `context-{get,set}-i32-{0,1}` intrinsics (the canonical - /// `context.{get,set}` built-ins — definitions.py:2348/2358) and fails at + /// `context.{get,set}` built-ins) and fails at /// instantiate time on the rest. Emitting them all keeps the shim's job /// "faithful transcription" and moves the capability boundary into the /// runtime, where contracts/intrinsics.md already puts it. @@ -294,10 +239,10 @@ pub struct ExportItemJson { /// One `wasmtime_environ::component::Trampoline`, tag-for-tag. `index` is the /// trampoline's own index (redundant with array position, kept for -/// greppability). Type-table references: `type`/`results` point into the plan -/// `types` table; `resource` into `resourceTables`; `streamTable`/ -/// `futureTable`/`errorContextTable` are raw wasmtime table indices (task- -/// scheduler machinery, no plan-level table yet); `options` into `canonicalOptions`; +/// greppability). Type-table references: `type`/`resultType` point into the plan +/// `types` table; task-return `results` is a raw wasmtime `TypeTupleIndex`. +/// `resource`, `streamTable`, `futureTable`, and `errorContextTable` index +/// their respective plan table sections; `options` indexes `canonicalOptions`; /// `memory`/`callback`/`postReturn` are runtime extraction indices. #[derive(Debug, Serialize)] #[serde(tag = "kind", rename_all = "kebab-case", rename_all_fields = "camelCase")] @@ -309,23 +254,12 @@ pub enum TrampolineDecl { ResourceDrop { index: u32, instance: u32, resource: u32 }, BackpressureInc { index: u32, instance: u32 }, BackpressureDec { index: u32, instance: u32 }, - /// `task.return`. Plan v3 splits what v2 conflated into one `results` - /// field: - /// - /// * `results` is now the **raw** wasmtime `TypeTupleIndex` (u32), i.e. - /// verbatim the value FACT's `prepare-call` passes as its - /// `task_return_type` argument at runtime (fact.rs:47,584). Without - /// it a consumer cannot relate the callee's declared result type to - /// anything in the plan, which is exactly why - /// `canon_task_return`'s result-type check was skipped for FACT - /// tasks. - /// * `result_type` is that tuple interned into `plan.types` — the - /// task-return raw `results` + interned `resultType` keys - /// (contracts/plan-format.md schema). `Option` for wire - /// symmetry with the other nullable decl fields only: wasmtime's - /// `Trampoline::TaskReturn.results` is a plain `TypeTupleIndex` - /// (info.rs:789-796, no `Option`), so this producer always emits a - /// number — a no-result task carries the *empty tuple*, not `null`. + /// `task.return`: `results` is the raw wasmtime `TypeTupleIndex` passed + /// by FACT's `prepare-call` as `task_return_type`; `result_type` is the + /// same tuple interned into `plan.types`. Together they let the runtime + /// check the FACT callee's declared result type in `canon_task_return`. + /// `Option` is for wire symmetry with other nullable fields: this producer + /// always emits a number, including the empty tuple for no-result tasks. TaskReturn { index: u32, instance: u32, results: u32, result_type: Option, options: u32 }, TaskCancel { index: u32, instance: u32 }, @@ -810,8 +744,8 @@ impl<'a> PlanBuilder<'a> { }); } - // 6b. Imported resources, in ResourceIndex order (v0.2 proposal; see - // the `Plan::imported_resources` docs). Emitted *after* imports so + // 6b. Imported resources, in ResourceIndex order (see + // `Plan::imported_resources`). Emitted after imports so // the `import` back-references can be range-checked here. let mut imported_resources = Vec::new(); for (_, runtime_import) in component.imported_resources.iter() { diff --git a/crates/translator-shim/tests/cabi.rs b/crates/translator-shim/tests/cabi.rs index 1d040d8..f2260ff 100644 --- a/crates/translator-shim/tests/cabi.rs +++ b/crates/translator-shim/tests/cabi.rs @@ -72,8 +72,7 @@ fn cabi_roundtrip_all_testdata() { } /// Errors come back as an `{"error": ...}` envelope, not a panic/trap, and -/// carry the structured verdict in `errorDetail` (contracts v0.2 proposal; -/// `src/error.rs`). The `error` string keeps its v0.1 meaning. +/// carry the structured verdict in `errorDetail` (src/error.rs). #[test] fn cabi_error_envelope() { let json = roundtrip_bytes(b"not a component"); diff --git a/crates/translator-shim/tests/translate.rs b/crates/translator-shim/tests/translate.rs index 6b931e1..219dea0 100644 --- a/crates/translator-shim/tests/translate.rs +++ b/crates/translator-shim/tests/translate.rs @@ -1,6 +1,5 @@ -//! Plan-v0 emission tests: the translator-spike assertions adapted to the plan -//! schema, plus determinism and a golden-ish shape test for the hello -//! fixture (contracts/plan-format.md). +//! Plan emission, determinism, and fixture shape tests +//! (contracts/plan-format.md). use translator_shim::plan::{ CoreDefJson, ExportDecl, Initializer, ModuleEntry, ResourceTableDecl, TrampolineDecl, TypeDecl, @@ -100,7 +99,7 @@ fn linked_generates_fact_adapter() { .collect(); assert_eq!(adapter_inits.len(), t.adapters.len()); - // Manifest categories cover the translator-spike-observed intrinsic surface + // Manifest categories cover the fixture's intrinsic surface // (intrinsics.md §A): callee core-def, instance flags, trap + // enter/exit-sync-call trampolines. (`task-may-block` was a // `CoreDef::TaskMayBlock` category; upstream removed that variant at the @@ -382,9 +381,8 @@ fn resources_plan_shape() { .any(|e| matches!(e, ExportDecl::LiftedFunc { .. }))); } -/// wit-bindgen 0.60 async guests use `context.{get,set}`, which wasmtime 47 -/// models as `CoreDef::UnsafeIntrinsic`. Plan v0 rejected it outright; plan -/// v1 (contracts/plan-format.md v0.3) emits it as +/// The async guest uses `context.{get,set}`, which wasmtime models as +/// `CoreDef::UnsafeIntrinsic`. The plan emits it as /// `{"kind":"unsafe-intrinsic","intrinsic":""}`. /// /// This locks the wire shape *and* the symbol vocabulary: the guest uses @@ -536,7 +534,7 @@ fn transcoder_trampoline_shape() { } // --------------------------------------------------------------------------- -// Structured verdicts (src/error.rs; contracts v0.2 proposal) +// Structured verdicts (src/error.rs; contracts/plan-format.md) // --------------------------------------------------------------------------- /// Malformed bytes and invalid components are both `validation`: the shim's diff --git a/ct-runner/src/import-analysis.ts b/ct-runner/src/import-analysis.ts index 14bf148..7f88cb2 100644 --- a/ct-runner/src/import-analysis.ts +++ b/ct-runner/src/import-analysis.ts @@ -48,10 +48,8 @@ export interface ImportAnalysis { * WITHOUT instantiating anything (translate-only; `requiredImports` reads * only the plan). Detects: * - whether `test-context` is imported (only then does the runner provide - * it — pre-composed bundles with the provider already linked must work - * too, per the dispatch); - * - a caller/runner collision on `test-context` (the runner "merges, - * erroring on collisions" per the dispatch's import-wiring spec); + * it; pre-composed bundles already include their provider); + * - a caller/runner collision on `test-context`; * - every other top-level import key the suite needs but `provided` lacks, * via the same version-canonical resolution `instantiate` itself uses * (`ImportResolver`, contracts/embedder-api.md §"Version canonicalization"). diff --git a/ct-runner/src/main.ts b/ct-runner/src/main.ts index 6514e9a..d1e1caa 100644 --- a/ct-runner/src/main.ts +++ b/ct-runner/src/main.ts @@ -16,7 +16,7 @@ // translator-shim wasm explicitly — required when this CLI runs outside a // polyengine checkout (e.g. imported by URL at a release tag, with the wasm // taken from that release's `polyengine-translator-shim.wasm` asset; see -// docs/consumers.md and issue #16's interim release scheme). Inside a +// docs/consumers.md). Inside a // checkout it defaults to the local release build under `target/`. import { Translator } from "../../runtime/src/shim/mod.ts"; diff --git a/docs/architecture.md b/docs/architecture.md index 8f24fcc..37032ba 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,685 +1,563 @@ # polyengine — architecture and design decisions -A WebAssembly Component Model host built on the JS `WebAssembly` API, targeting -JSPI-capable engines. Primary development against Deno; conformance runs -against browsers. Aims for Component Model feature parity and compatibility -with wasmtime and wit-bindgen. **0.3.0 concurrency is the core deliverable.** +polyengine loads WebAssembly Component Model binaries on the JS +`WebAssembly` API. A wasm32 translator produces a linking plan and fused +adapters; a platform-neutral TypeScript runtime executes the plan, handles +the host ABI, and schedules Component Model tasks. -This document records the architecture and the decisions made, with rationale. -Section numbers **§1–§11 are stable** — they are cited from code comments and -contracts throughout the repo. Related documents: - -- [`consumers.md`](consumers.md) — the polymorph adoption track (jco replacement) -- [`references.md`](references.md) — canonical upstream links -- [`../contracts/`](../contracts/) — versioned interface contracts -- [`../AGENTS.md`](../AGENTS.md) — the development protocol -- open work: the [issue tracker](https://github.com/polymorph-components/polyengine/issues) - ---- +Section numbers and headings are stable because code and contracts link +to them. The [contracts](../contracts/) specify interfaces; +[security.md](security.md) describes trust boundaries; +[consumers.md](consumers.md) covers downstream integration; and +[references.md](references.md) records upstream sources and dependency pins. ## 1. Goals -- **Concurrency is the point.** Existing hosts already run non-async - components fine; this project exists to be a first-class host for Component - Model 0.3.0 concurrency — `async` lift/lower, tasks/subtasks, - `stream`/`future` — mapped natively onto the JS event loop and JSPI. - Sync-only operation is a supported subset, not a destination. -- Load, link, and run Component Model binaries (`.wasm` components) at runtime - on stock JS engines, using only the JS `WebAssembly` API. -- Component Model feature parity with wasmtime, tracked against the official - spec ([WebAssembly/component-model]). -- Compatibility with wasmtime-built and wit-bindgen-built guest components, - sync and async alike, made executable via imported conformance/test - suites. componentize-go output is a named second guest toolchain - (consumer-driven, [consumers.md](consumers.md)): callback-ABI async lifts + - async-lowered imports from Go's patched runtime — a differently-shaped - exerciser of the same ABI. -- **Adoption target: replace jco as the JS host for the polymorph - component family and experiment-mosh** ([consumers.md](consumers.md)). - Their JS-host legs were blocked on structural jco defects in exactly this - project's core territory (0.3 concurrency). They have no external - dependents, so embedder conventions co-evolve with them — designed against - real consumers, not in the abstract. -- **"Parity" means functional parity, not behavioral identity.** The bar is: - the same feature set, spec-conforming behavior, and wasmtime/wit-bindgen - guests running correctly. Where the spec sanctions a range of behaviors, - this host may — and does — diverge from wasmtime's choices (deterministic - FIFO scheduling per §6; deterministic NaN profile; JS-native host value - shapes; per-instance poisoning, §6). Wasmtime-identical observable - behavior is adopted only where (a) something external forces it — the - official suite's `assert_trap` matches message text, which is de facto - wasmtime wording — or (b) it is free by construction (the translation - frontend *is* wasmtime-environ, §4). Behavior mandated by the - spec/reference (e.g. the borrow-lending traps, per definitions.py) is spec - conformance, not wasmtime-matching, even when wasmtime exhibits it too. - The tie-breaking authority for semantic questions is the spec + - `definitions.py`, with wasmtime as corroborating evidence — never the - other way around. **One bounded exception** (operator decision): where - `definitions.py` contradicts the spec repo's *own wast corpus* and - wasmtime implements the corpus side, the corpus semantics — as wasmtime - actually implements it, verified against wasmtime source or trace, not as - inferred from the test alone — is adopted as the working assumption. Each - such case must be a named finding in - `upstream-component-model-repo-findings.md` (currently CM-3 only) and - flips back if upstream adjudicates the other way. Bare wasmtime behavior - with no corpus backing never supersedes the reference. Guard before - invoking the exception: a corpus assertion counts as semantic authority - only if it is **schedule-independent** — a test that two conforming - scheduler policies answer differently pins a policy, not semantics, and - is an upstream test defect rather than a conflict. -- TypeScript throughout the JS side: the runtime, the harness, and all - generated bindings. -- A performance story that can get fast later without rearchitecting. +- Load, link, and execute component binaries at runtime on JavaScript + engines, or consume artifacts translated at build time. +- Support Component Model 0.3 concurrency: async lift/lower, tasks and + subtasks, streams, futures, backpressure, and cancellation. Sync calls + use the same task machinery. +- Run guests from external toolchains, including Rust/wit-bindgen and + componentize-go, and composed workloads from the polymorph family. +- Pursue functional parity with the Component Model feature set supported + by wasmtime, tested against the spec and guest workloads. This is a + compatibility target, not a claim that every feature is implemented or + that passing tests proves conformance. Coverage and gaps are in §11. + +**Semantic authority.** The Component Model spec and its executable +reference, `third_party/component-model/design/mvp/canonical-abi/definitions.py`, +break ties. Wasmtime is corroborating evidence, not an overriding +authority. Reusing its frontend reduces duplicated implementation; it +does not prove the shim, runtime, or their integration correct. + +**Parity is functional, not behavioral identity.** Where the spec permits +choices, this runtime may differ from wasmtime: deterministic FIFO +scheduling, deterministic NaNs, and JS-native host value shapes. It also +has named divergences (§6). The conformance harness accommodates +engine-specific trap wording; diagnostic text is not the public API. + +**One bounded exception (operator decision):** when `definitions.py` conflicts with the spec +repository's own WAST corpus and wasmtime implements the corpus side, the +corpus semantics may be adopted as a working assumption. This requires +verification against wasmtime source or a trace, a named finding in +[upstream-component-model-repo-findings.md](../upstream-component-model-repo-findings.md), +and reversal if upstream adjudicates otherwise. Currently this applies +only to CM-3. A schedule-dependent assertion cannot invoke the exception: +if two conforming schedulers can answer differently, the assertion pins a +policy rather than semantics. Wasmtime behavior alone is insufficient. ## 2. Non-goals -- **WASI implementations in the core.** `wasi:*` host packages are out of - scope for the runtime itself; the finish line is Component Model support, - demonstrated with custom WIT worlds. Two sanctioned carve-outs, both - outside the core: WASI interface *shapes* are first-class design inputs - to the embedder conventions ([consumers.md](consumers.md)) — they are the - ecosystem's most important interfaces and the conventions must serve them - well — and a WASI provider *package* (`wasi/`, `@polyengine/wasi`; a - separate deliverable with consumer-driven scope: p2 cli/io/clocks/random - baseline + p3 clocks, plus à la carte network fragments the default - `wasi()` merge never carries — p3 `wasi:sockets` - (`@polyengine/wasi/sockets`: UDP, TCP client + listener; one - node-builtins backend serving Deno via its stable node compat and real - Node, [#4](https://github.com/polymorph-components/polyengine/issues/4)) - and the fetch-backed `wasi:http@0.3` outbound client - (`@polyengine/wasi/http`, riding the `@0.3` track like the rest of the - package; a `version` override keys exact ids for guests pinned to - pre-consolidation rc snapshots)). -- **Componentizing JS/TS.** Guests are components built by external toolchains - (Rust + wit-bindgen is the reference). No embedded-JS-engine work. -- **jco compatibility or reuse.** Ignored entirely — including at the - embedder-API level: we do not emulate jco's host conventions (thrown - bare `{tag, val}` payloads, its `Stream` objects, transpile-time async - enumerations); consumers port to our conventions - ([consumers.md](consumers.md)). Where we need prior art we take it from - wasmtime; where jco's conventions have known footguns (documented - defensively by the polymorph host modules themselves), we fix rather than - inherit. Replacing jco for the named consumers is a goal (§1); *being* jco - is not. -- **Pre-JSPI engines.** No JSPI fallback path for the stackful/blocking - forms. (The callback ABI — what wit-bindgen and componentize-go actually - emit — needs no JSPI at all, so the effective floor for consumer workloads - is "any modern engine"; JSPI is required only for sync-blocking forms.) +- **WASI in the runtime core.** Providers live in the separate + `@polyengine/wasi` package. Its default `wasi()` merge supplies captured + CLI I/O, clocks, entropy, and empty filesystem preopens. Real filesystem, + host stdio, sockets, and outbound HTTP providers are explicit opt-ins. + WASI interface shapes still inform the embedder API. +- **Componentizing JS/TS.** Guests are binaries from external toolchains; + this project does not embed a JS engine into components. +- **jco API compatibility.** Consumers use the conventions in + [contracts/embedder-api.md](../contracts/embedder-api.md), not jco's + host value shapes or transpilation options. +- **A JSPI fallback for stackful/blocking forms.** Callback-ABI async + execution does not itself need JSPI. A callback-ABI guest can still need + it when it calls a blocking sync import. Missing engine capabilities + are reported, not emulated. ## 3. Compatibility targets -Floor for the JSPI-dependent forms: engines supporting JSPI (proposal is -**phase 4**; minor API drift is still possible — it changed once already when -the `Suspender` object was removed). - -| Engine | JSPI status | Role | -|---|---|---| -| Deno ≥ 2.3.2 | on by default | primary dev target | -| Chrome/Chromium ≥ 137 | on by default | browser lane (exact Deno parity) | -| Firefox | flag: `javascript.options.wasm_js_promise_integration` | browser lane (pref flipped by the driver) | -| Safari / WebKit | JSPI works unflagged on WPE 26.5; JSPI + multi-memory both present in Safari Technology Preview; stable-Safari status: [#11](https://github.com/polymorph-components/polyengine/issues/11) | browser lane; pinned build capped by JSC's missing multi-memory — implemented and default-on in WebKit trunk (webkit-2342+ rolls reach effective parity; #11) | -| Node | on by default ≥ 26 | pinned runtime lane (`node-pinned`, v26.x: exact Deno parity, no flags; required gate). Node 24 LTS deliberately not laned: flag-gated JSPI (`--experimental-wasm-jspi`) whose older V8 13.6 vintage deviates on 2 corpus commands — see `harness/shell/expectations/node-pinned.ts` | -| Bun | on by default (1.3.x, vendored JSC) | pinned runtime lane (`bun-pinned`, findings-only until a track record): exact Deno parity under `BUN_JSC_useWasmMultiMemory=1` (driver-set; stock bun ships multi-memory off) — see `harness/shell/expectations/bun-pinned.ts` | - -Notes: - -- Deno and Chrome share V8, so Firefox and WebKit provide the real engine - diversity. SpiderMonkey JSPI is clean over the full corpus; JSC's JSPI - works unflagged, but the pinned JSC build lacks multi-memory, the actual - WebKit-lane cap (the CABI routinely needs >1 memory per core module; - default-on in WebKit trunk, #11). Engine trap-message wording differences - are normalized in the harness matcher, never in the runtime - (`TRAP_MESSAGE_EQUIVALENTS`, harness/src/runner.ts) — with them - reconciled, Firefox and trunk WebKit run at exact Deno-lane parity. -- The node/bun lanes add **embedding** coverage, not engine coverage (V8 and - JSC are already exercised above): module loading, event-loop integration, - and runtime I/O quirks — e.g. node's pooled `Buffer`, whose pool-backed - `.buffer` must never reach WebAssembly APIs (`tools/shell/host-node.mjs`). -- **Type reflection (js-types) is phase 3 and flagged everywhere** — function - signatures are not available from `WebAssembly.Module.imports()`. The - architecture below sidesteps this (the translator emits all type - information), but no design may assume type reflection exists. -- CSP: compiling from bytes requires `wasm-unsafe-eval`. The runtime - requires nothing beyond that — **a design invariant, not a default** - ([#8](https://github.com/polymorph-components/polyengine/issues/8)): no - code path may require full `unsafe-eval`. The specialized-JS executor is - emission-only (§8) — a deploy-time AOT step or a server-side first-load - cache import — never runtime `eval`/`new Function`. -- The runtime core is platform-neutral by contract (§4.3), pinned by - `runtime/tests/platform_purity_test.ts` — no `node:*` builtins, no Deno - APIs. +Core-wasm features and JSPI are separate requirements. For example, FACT +adapters may require multi-memory even when guest async calls use the +callback ABI. JSPI is needed for stackful async lifts and suspending sync +lowers (§5–§6). -## 4. Architecture +| Engine | JSPI / lane policy | +|---|---| +| Deno | Primary development runtime; JSPI enabled by default from 2.3.2 | +| Chrome/Chromium | JSPI enabled by default from 137; browser lane | +| Firefox / SpiderMonkey | Browser driver enables `javascript.options.wasm_js_promise_integration`; pinned shell lane is a required gate | +| WebKit / JavaScriptCore | JSPI works in the pinned WPE browser build, but that build lacks multi-memory. Newer trunk builds support both; stable Safari support remains tracked in [#11](https://github.com/polymorph-components/polyengine/issues/11) | +| Node | Pinned Node 26 lane requires no JSPI flag. The npm package's lower Node floor does not guarantee JSPI; Node 24's flag-gated implementation is not the conformance target | +| Bun | Pinned findings-only lane; driver enables `BUN_JSC_useWasmMultiMemory=1` | + +Exact shell versions live in [tools/shell/pins.json](../tools/shell/pins.json). +[Browser](../harness/browser/expectations/) and +[shell](../harness/shell/expectations/) expectations record lane-specific +differences. V8, SpiderMonkey, and JSC provide engine coverage; Node/Bun +also test module loading, I/O, and event-loop integration. These lanes do +not establish support for every release of an engine family. + +The runtime does not depend on WebAssembly JS type reflection. The +translator supplies signatures that `WebAssembly.Module.imports()` cannot +provide on the supported baseline. + +**CSP invariant:** compiling wasm bytes requires `wasm-unsafe-eval`, not +full `unsafe-eval`. The runtime does not use `eval` or `new Function`. +Any future specialized-JS executor must be emitted as importable modules +(§8), not generated and evaluated inside a browser at runtime. Core +platform neutrality is checked by +`runtime/tests/platform_purity_test.ts`; platform-specific loaders and +WASI backends sit outside it. -One deterministic pipeline, run either at first load (and cached) or ahead of -time — "AOT" and "runtime linking" are the same code executed at different -moments. +## 4. Architecture -``` - Rust (compiled to wasm32, runs everywhere) - ┌──────────────────────────────────────────┐ - component.wasm ───► │ translator = wasmtime-environ (validate, │ - │ resolve linkage) + FACT (fused adapters) │ - │ + shim (stable output format) │ - └──────────────┬───────────────────────────┘ - │ artifacts (bytes, content-addressed) - ▼ - ┌────────────────────────────────────────────────────┐ - │ plan: instantiation ops, type tables, CABI │ - │ descriptors, required-intrinsics list │ - │ core modules: byte ranges sliced from the component │ - │ adapter modules: FACT-generated core wasm │ - └──────────────┬─────────────────────────────────────┘ - │ - ▼ TypeScript (platform-neutral) - ┌────────────────────────────────────────────────────┐ - │ runtime: plan executor, host-boundary lift/lower, │ - │ resource tables, intrinsics, instance-state rules, │ - │ JSPI trampolines, 0.3 task scheduler (core) │ - └────────────────────────────────────────────────────┘ - - wit/*.wit ──► bindgen (Rust, wit-bindgen-core) ──► typed TS bindings - (verified against the plan at instantiate()) +Build-time and runtime translation use the same pipeline. Both currently +execute the host boundary through the descriptor interpreter. + +```text +component.wasm + | + v +wasm32 translator: wasmtime-environ + FACT + translator-shim + | + +-- plan: initializers, core-module ranges, types, CABI descriptors, + | required intrinsics, resource metadata + +-- FACT adapter modules + | + v +TypeScript runtime + original component bytes + +-- compile core modules and adapters with WebAssembly APIs + +-- instantiate and link in plan order + +-- interpret host-boundary lift/lower + +-- manage resources, tasks, streams/futures, and JSPI entries + +WIT source --> bindgen --> typed TS facade + expected world digest + | + checked at instantiation ``` ### 4.1 Translator: wasmtime's frontend compiled to wasm -We reuse wasmtime's "decide what to do" layer, which is separable from its -"do it" layer and has no native-code dependency: - -- `wasmtime-environ`'s component translator: parsing, validation, subtyping, - and resolution of the component's linking structure into a flat - instantiation plan. -- `wasmtime-environ::fact` (FACT): generates **fused adapters** — the glue for - cross-component calls (canonical-ABI lift composed with lower) — **as plain - core wasm modules** via `wasm-encoder`. - -Why this is the cornerstone decision: - -- **Wasmtime compatibility by construction.** We inherit wasmtime's - interpretation of the spec for the largest correctness surface (validation, - types, adapter semantics) and turn "compat with wasmtime" into a version pin. -- **It solves the JSPI stack-purity problem (§5) by construction** — all - cross-component call paths are wasm, never JS. -- **It removes the hardest codegen** (flattening, param spilling, string - transcoding, resource transfer, post-return) from our scope. - -Constraints and mitigations: - -- `wasmtime-environ` is an **internal, unstable API**. Mitigation: a thin Rust - **shim** crate owns the dependency and maps environ's output into our own - stable plan format. Wasmtime churn is confined to the shim. Pin wasmtime and - wasm-tools versions; upgrade deliberately (the staged bump is - [#1](https://github.com/polymorph-components/polyengine/issues/1)). -- FACT adapters import **host intrinsics** (string transcoders, - `resource-transfer-own/borrow`, enter/exit bookkeeping, trap). The TS - runtime implements this contract — specified in - **[contracts/intrinsics.md](../contracts/intrinsics.md)** — and the shim - emits the required-intrinsics list per component so the contract is explicit - at translation time, not discovered at instantiation. These intrinsics are - synchronous JS calls that return before any suspension can occur — - compatible with the JSPI frame rule. -- The translator ships as a **plain core wasm module** with a bytes-in/bytes-out - ABI (no components-all-the-way-down bootstrap). -- Size: 1.66 MiB size-tuned (~0.5 MiB gzip), sub-ms steady-state translation; - multi-MB consumer components translate in tens of ms. Being a real static - asset ≥ 128 kB, browsers code-cache the translator itself well — the most - expensive fixed cost of the pipeline is the part engines already handle. +`crates/translator-shim` compiles wasmtime's translation frontend to a +plain core wasm module with a bytes-in/bytes-out ABI: + +- `wasmtime-environ` parses and validates components, resolves types and + linkage, and flattens instantiation into a plan. +- FACT generates fused canonical lift/lower adapters as core wasm. These + perform cross-component conversion without a JS adapter frame between + guest calls, allowing suspension on those paths (§5). +- The shim maps environ's internal structures to the versioned + [plan format](../contracts/plan-format.md). + +The frontend and FACT remain upstream dependencies, not a replacement +for runtime conformance testing. FACT calls host intrinsics for operations +including transcoding, resource transfer, state bookkeeping, and traps. +The runtime implements that interface under +[contracts/intrinsics.md](../contracts/intrinsics.md); the plan lists the +required intrinsics. Host-boundary conversion is also this runtime's +responsibility, not FACT's. + +**Local FACT correction.** Adapters are not copied verbatim from the +pinned frontend: `crates/translator-shim/src/fact_string_limits.rs` +corrects recognized pre-realloc string guards to enforce the reference's +source-byte limit of `(1 << 28) - 1`, rather than the pinned generator's +destination-width/retry thresholds. Only FACT-generated adapters pass +through this correction; embedded guest modules are untouched. It +preserves module length, validates the result, and rejects unrecognized +guard shapes or an environ revision change. Every frontend pin update +must review whether to retain, revise, or remove this correction. The +producer matrix and drift checks are in +`crates/translator-shim/tests/fact_string_source_limits.rs` and +`runtime/tests/fact_string_source_limits_test.ts`. + +`wasmtime-environ` is an unstable internal API. Its git revision and the +matching wasm-tools release train are pinned in [Cargo.toml](../Cargo.toml) +and [Cargo.lock](../Cargo.lock). The shim contains dependency-specific +mapping code; upgrades require integration gates, not just a pin change. ### 4.2 Plan format -Specified in **[contracts/plan-format.md](../contracts/plan-format.md)**. -Summary of the fixed decisions: - -- Defined by us, versioned, **operational content only**: instantiation ops, - core-module slice ranges, adapter module references, canonical-ABI - descriptors for host-boundary functions, type tables, required intrinsics, - resource-type metadata (dtor references). -- **No WIT-level fidelity** (no docs, no feature gates, no aliasing - structure) — bindings generation reads WIT source instead (§9). This keeps - the format small and stable. -- Encoding: JSON (the simplest thing that round-trips); revisit only if - measurable. -- Deterministic: identical inputs (component bytes, translator build, flags) - produce identical artifacts. This is what makes caching trivial (§10). +[contracts/plan-format.md](../contracts/plan-format.md) defines the wire +format and artifact set: + +- JSON with strict format-version checking and structural validation. +- Operational data: initializers, module references, CABI descriptors, + type tables, required intrinsics, and resource/destructor metadata. +- Core modules referenced by byte ranges in the caller's original + component; generated adapters supplied separately. The plan does not + duplicate the component bytes. +- No WIT source fidelity: documentation, source-level aliases, and feature + gates are not preserved for bindgen (§9). + +Identical component bytes, translator build, and feature settings are +expected to produce identical artifacts. That identity supports caching +(§10); it is not an authenticity guarantee. ### 4.3 TS runtime -Platform-neutral core (dependencies: `WebAssembly` JS API, `TextEncoder`/ -`TextDecoder`, Promises — nothing else; pinned by -`runtime/tests/platform_purity_test.ts`). Responsibilities: +The runtime core requires only standard web-platform APIs, not +platform-specific APIs. Guarded ambient probes for diagnostics or the +scheduler seed may inspect Deno when available; they are not runtime +requirements. Platform-specific cache backends and asset loaders are +separate concerns. Core responsibilities are: -1. Plan executor: compile sliced core modules and adapters, instantiate in - plan order, wire imports/exports. -2. Host boundary: lift/lower per CABI descriptors (§8), `realloc`/ +1. Compile modules, instantiate in plan order, and wire imports/exports. +2. Lift/lower host values from CABI descriptors, including `realloc` and `post-return` handling. -3. Resource machinery: slab handle tables, own/borrow tracking (`num_lends`, - borrow invalidation at call return), dtor invocation (§7), FACT intrinsic - implementations. -4. Instance-state rules: `may_leave` enforcement and poisoned-instance - refusal — **JSPI enforces no Component Model invariant for us**; the - state discipline is ours and must hold while suspended. (The spec has no - reentrance gate — reentrance into a live instance is valid; the only - entry refusal is the poisoned-corpse divergence, §6.) -5. Task scheduler (§6): the 0.3 task/thread model is the runtime's core - structure, not an add-on — waitable sets, streams/futures, callback-ABI - event dispatch, backpressure, cancellation. Sync calls are the degenerate - case: a task driven to resolution before the call returns, exactly as in - the reference implementation. - -Above the raw boundary sits the **embedder conventions layer** -(`runtime/src/embedder/`, governed by -[contracts/embedder-api.md](../contracts/embedder-api.md)): camelCase facades, -branded `ComponentException`s, resources as classes in both directions, `Stream`/ -`Future` handles over web-native producers, and semver-canonical import -resolution matching the spec + wasmtime's `NameMap`. +3. Maintain resource handle tables, own/borrow transfers, lend counts, + borrow invalidation, and destructor calls. +4. Enforce Component Model state rules such as `may_leave`, and the + runtime's poisoned-instance refusal (§6). JSPI enforces none of these + rules. There is no separate reentrance gate into a live instance. +5. Schedule tasks, threads, waitables, streams/futures, callback events, + backpressure, and cancellation. Sync calls use tasks too. + +The [embedder layer](../runtime/src/embedder/) adapts the raw interpreter +boundary into camelCase facades, branded errors, resource classes, +`Stream`/`Future` handles, and version-canonical import resolution. Its +public behavior is governed by the +[embedder contract](../contracts/embedder-api.md), not the raw boundary. ## 5. The JSPI frame rule (load-bearing constraint) -From the JSPI spec ([js-promise-integration Overview]): - -> Only WebAssembly computations may be suspended: **only WebAssembly frames may -> be active between the call to a `promising` function and any call to a -> `Suspending` wrapped import** — a JS frame in between traps. - -Consequences baked into this design: - -- **Host boundary JS glue is safe.** A `Suspending`-wrapped import's JS runs to - completion and returns a Promise; suspension happens after it returns, so - host-side lift/lower in JS never sits on the suspended stack. -- **Cross-component glue must be wasm.** A JS adapter between components A and - B would trap the moment anything below it suspends. FACT adapters keep those - stacks pure wasm — this is why §4.1 is the cornerstone. -- **Component exports invoked from JS** that may transitively suspend must be - entered through `WebAssembly.promising` trampolines. This includes - JS-initiated resource drops (§7). -- **Guest-initiated cross-component dtor calls** route through generated wasm - (direct funcref call in the adapter/intrinsic path), not a JS bounce. +JSPI suspends wasm computations, not arbitrary JS stacks. Between an +entry through `WebAssembly.promising` and a `WebAssembly.Suspending` +import, an intervening JS frame prevents suspension. See the +[JSPI overview][js-promise-integration Overview]. + +- Host-boundary glue runs inside the suspending import and returns its + Promise before suspension. It is not an intervening frame. +- Cross-component ABI adapters are wasm. A JS adapter that calls another + guest would prevent a later suspension below it. +- JS calls into wasm that may suspend need a `promising` entry. The + executor also prepares such entries for suspension-capable host-initiated + resource drops. +- **Guest-initiated destructor calls are a current exception to the + pure-wasm path:** their dispatch contains a JS frame. They must complete + synchronously; attempting JSPI suspension traps. Pure-wasm destructor + dispatch is not implemented (§7). ## 6. Concurrency (the core deliverable) -Existing hosts handle non-async components adequately; 0.3.0 concurrency is -why this project exists. The runtime is therefore designed around the 0.3 -task model **from day one** — sync-only operation falls out as the degenerate -case, exactly as in the reference implementation (`definitions.py`, where -`canon_lift` always creates a Task/Thread and the sync path is a driving loop -over the same structures). This ordering was deliberate: retrofitting the task -model onto a sync-first runtime is the rearchitecting we were not allowed to -need. (It is also, empirically, the rearchitecting jco is stuck in — see -[consumers.md](consumers.md).) +The task model follows the executable reference's `Store`, `Task`, +`Thread`, and `Subtask` structures. Scheduling is cooperative: the JS +event loop supplies host settlements, and explicit queues determine guest +progress. There is no preemption. -Mapping the reference model onto the web platform: - -| Reference concept | Implementation | +| Reference operation | Runtime mechanism | |---|---| -| `Thread` (suspendable computation) | wasm activation entered via `WebAssembly.promising` | -| `Thread.wait_until` / blocking | call to a `Suspending` import returning a scheduler-controlled Promise | -| resume | scheduler resolves that Promise (event-loop turn) | -| scheduler | JS event loop + explicit ready queues; cooperative, matching the CM model — no preemption exists or is needed | -| `Waitable` / `WaitableSet` | host-side event structures; `wait` = suspension (stackful) or the callback return-code protocol (stackless) | -| callback ABI | no suspension at all: the scheduler invokes the callback export with events | -| sync `canon_lift` driving loop | same scheduler: pump ready threads until resolved, with the spec's deadlock trap. Async-typed exports have no such loop in the reference (`canon_lift` returns after the first resume, line 2189), so their driver exits on idle and the Promise stays pending for a later driver to settle — wasmtime `call_concurrent`, not `call_async` (#292; contracts/embedder-api.md §"Functions and async") | -| `Subtask`, backpressure, cancellation | direct ports of the reference structures | - -JSPI's three roles, precisely: - -1. **Stackful async lifts** (no-callback `async`) — the guest blocks mid-stack. -2. **Blocking sync lowers** — a caller waiting on an unresolved subtask - (`thread.wait_until(subtask.resolved)` in the reference). -3. **Sync guests over async host imports** — falls out of the same mechanism; - a useful capability, not a separate deliverable. - -The callback ABI needs no JSPI (stackless by design). Empirical confirmation -(reconfirmed by every consumer artifact): wit-bindgen emits **exclusively -callback-ABI async lifts**, and componentize-go likewise — running real async -guests requires the task core, not JSPI. - -Determinism: the reference scheduler makes explicitly nondeterministic -choices (`random.choice` over ready threads). **Decided:** deterministic -FIFO ready-queue by default; a seeded-shuffle mode (`POLYENGINE_SCHED_SEED` env -var) exercises the spec-allowed nondeterminism in tests, verified across -seeds. Documented at `runtime/src/task/scheduler.ts`. A load-bearing -architectural rule: **one driver per store** — concurrent `driveAsync` loops -can double-resume threads; between export calls the two fallback drivers -stand down whenever an export-call driver is live (the invariant and its -benignity argument are documented at the site in -`runtime/src/exec/boundary.ts`). There are exactly three drivers: export -calls, the host-activity pump (embedder stream/future operations landing -between calls), and the settlement pump, which services host-import -settlements that land while the store is driver-idle. The settlement pump is -what gives background tasks host-driven liveness between export calls (a -task parked on a waitable set whose pending host call is a clock resumes at -settlement time); wasmtime only delivers such wakeups while the embedder -dwells in `run_concurrent`, but a JS host's event loop is always dwelling, -so polyengine makes it unconditional. - -Named divergence ([#92](https://github.com/polymorph-components/polyengine/issues/92)): -**the async form of `subtask.cancel` is not atomic under jspi.** The -reference built-in returns `[BLOCKED]` with no suspension; polyengine parks the -caller on a determinacy wait so the BLOCKED/resolved answer matches the -reference's synchronous-delivery outcomes across the engine's mandatory -microtask hop (jspi pin (j), pinned by `cancellable.wast`). While parked, -other ready threads of the store may run, so sibling-task effects can become -observable across the single built-in call — a reordering *within* the -reference's own `Store.tick` freedom, taken one built-in early; every -interleaved sibling was already at a block point. Rationale and mechanics at -the site (`runtime/src/intrinsics/async_builtins.ts`, the determinacy park -in `createSubtaskCancel`); regression pinned across seeds by -`runtime/tests/cancel_bracket_race_test.ts`. - -**Host-import cancellation resolves promptly by default.** The reference -leaves a host callee's `on_cancel` to the embedding (`Store.invoke`, -definitions.py line 572); wasmtime hosts hand back a future whose drop *is* -cancellation. A JS Promise offers no such channel, so polyengine's lowered -host imports answer with the reference's prompt-cancel shape — -`on_cancel = () => on_resolve(None)` — resolving the subtask -CANCELLED_BEFORE_RETURNED and discarding the promise's eventual settlement -(never lowered, rejections unreported, deregistered from deadlock -accounting). This is a reference-legal host behavior, not a divergence; the -per-declaration `deferCancel()` brand (contracts/embedder-api.md -§"Functions and async") restores run-to-completion for imports with commit -points. The host operation itself is never interrupted — only delivery is -cancelled. `abortable()` closes that gap for hosts that can be stopped: a -marked import receives a fresh `AbortSignal` appended after its -WIT-declared parameters on every call, and the runtime aborts it a -microtask after the discard — never synchronously inside -`canon_subtask_cancel`, so a host abort listener never runs inside a live -guest activation. Whatever settlement the abort provokes (typically an -`AbortError` rejection) arrives with the subtask already resolved and lands -on the resolved-subtask guards, discarded like any other late settlement. - -Named divergence ([#173](https://github.com/polymorph-components/polyengine/issues/173)): -**per-instance poisoning is polyengine's only entry refusal.** A trap that -escapes a guest activation marks the instance's corpse -(`poisonedInstances`); every entry site refuses a marked instance -permanently, naming the original trap -([#145](https://github.com/polymorph-components/polyengine/issues/145)). -The reference has no instance-level trap state at all and wasmtime kills -the whole store on trap, so per-instance corpse semantics — sibling -instances of the same instantiation stay usable — is purely this runtime's -choice, pinned by `builtin-trap-poisons-instance.wast`'s substring -expectations and the runtime poisoning suites. The same-instance exemption -(`caller === callee` passes vacuously) is preserved in the refusal guard -for the dtor self-drop path. There is no reentrance gate besides this: -the spec removed `may_enter`/`entering_set` -([CM#705](https://github.com/WebAssembly/component-model/pull/705)), and -reentrance into a live instance — host-mediated or otherwise — is simply -valid. - -Named divergence ([#296](https://github.com/polymorph-components/polyengine/issues/296)): -**cancel-read/cancel-write of a copy that already COMPLETED reports -CANCELLED|count.** definitions.py's `cancel_copy` returns an already-armed -pending event verbatim, so cancelling a stream copy the guest never observed -completing would report COMPLETED|count; polyengine instead supersedes an -undelivered stream COMPLETED with CANCELLED, count preserved, following the -upstream suite (`test/async/big-interleaving-test.wast:1526-1531`, which -asserts `0x42` where the reference's own rule gives `0x40`) over -definitions.py itself. Mechanics at the site -(`runtime/src/intrinsics/stream_builtins.ts`, `takeCancelEvent`); the -definitions.py-vs-suite disagreement is tracked for upstream filing as -[CM-3](../upstream-component-model-repo-findings.md#cm-3-cancel_copy-returns-a-stale-completed-where-wasmtime-reports-cancelled). +| Stackful thread execution | Wasm activation entered through `WebAssembly.promising` | +| Blocking wait | `Suspending` import returning a scheduler-controlled Promise | +| Resume | Scheduler resolves or consumes the relevant settlement | +| Callback ABI | Scheduler invokes the callback export with events; no suspended wasm stack | +| Waitable / waitable set | Host-side event state, consumed by stackful waits or callback return codes | +| Sync `canon_lift` | Drive the task to resolution, retaining the reference's deadlock trap | +| Async `canon_lift` | Exit the driver on idle; an unresolved export Promise stays pending for later progress | + +JSPI is used for no-callback stackful async lifts, blocking sync lowers, +and sync guests calling host imports marked `suspending()`. A callback +ABI alone needs no JSPI, but a guest's blocking imports may still require +it. Rust guest fixtures and external Go consumer workloads exercise callback +lifts; the Go integration test skips when its external artifact is absent. + +**Scheduling policy.** The default is deterministic FIFO in waiting-list +order, not the time readiness became true; pending events use join order. +Tests can use seeded shuffling through `POLYENGINE_SCHED_SEED` to exercise +spec-permitted scheduling variation. See `runtime/src/task/scheduler.ts`. + +**Overlapping drivers.** Concurrent exports may run overlapping +`driveAsync` loops on one store. The invariant is that an activation +consumes a settlement at most once and never resumes from an obsolete +settlement. `runtime/src/exec/boundary.ts` enforces this with synchronous +awaiting-membership removal, memoized Promise tags, Promise-identity +checks, and per-store pending-resumption bookkeeping. + +The asynchronous host-activity and host-settlement pumps are fallback +drivers: they stand down cooperatively when another driver is active. +This is not a ban on synchronous pump participation: `HostActivity.pump()` +services settled activations and ticks ready threads before its async +fallback checks driver depth, including while an export driver is live. +Arrival notifications wake parked drivers so they can yield or reconsider their +waits. New host-call registrations also wake incumbent drivers rather +than leaving them parked on an obsolete snapshot of pending work. + +**Between-calls progress.** A host import settling can resume background +guest work even with no export call in flight. A task waiting for the +embedder's half of a stream/future remains pending until the embedder +acts. An idle async-typed export may remain pending indefinitely; sync-typed +exports retain deadlock detection. See the +[function contract](../contracts/embedder-api.md#functions-and-async). + +**Host-import cancellation.** By default, cancellation resolves the +subtask promptly as `CANCELLED_BEFORE_RETURNED` and discards late Promise +settlements. The result is not lowered, and the discarded call no longer +counts as an outstanding host dependency. This cancels delivery, not the +JS operation. `deferCancel()` instead keeps the import running to +completion; `abortable()` supplies a per-call `AbortSignal`. On discard, +the signal is aborted in a microtask, never inside the guest activation. +These are embedding policies permitted by the reference's host-callee +cancellation hook. + +Named differences from the reference or other hosts: + +- **Async `subtask.cancel` under JSPI is not atomic** + ([#92](https://github.com/polymorph-components/polyengine/issues/92)). + The runtime may park for a determinate cancellation result across the + engine's mandatory microtask hop. Ready sibling threads can run during + that park. See `createSubtaskCancel` in + `runtime/src/intrinsics/async_builtins.ts` and + `runtime/tests/cancel_bracket_race_test.ts`. +- **Per-instance poisoning** + ([#173](https://github.com/polymorph-components/polyengine/issues/173)). + A trap escaping a guest activation permanently poisons that instance; + later entry names the original cause. Sibling instances remain usable + unless the trap propagates into them. The reference has no instance-level + trap state; wasmtime's store-level trap handling is not this policy. + The same-instance exemption in `entryRefusal` permits destructor + self-drops. Reentrance into an otherwise live instance is valid. +- **Cancelling an unobserved completed stream copy reports + `CANCELLED|count`**, preserving the count rather than delivering the + pending `COMPLETED|count` + ([#296](https://github.com/polymorph-components/polyengine/issues/296)). + This is the sole §1 corpus/reference exception, + [CM-3](../upstream-component-model-repo-findings.md#cm-3-cancel_copy-returns-a-stale-completed-where-wasmtime-reports-cancelled). + See `takeCancelEvent` in `runtime/src/intrinsics/stream_builtins.ts`. ## 7. Canonical ABI decisions -Authority: [CanonicalABI.md] and its executable reference -(`design/mvp/canonical-abi/definitions.py`). Where the host has freedom, we -decide deliberately and document here. - -- **Strings.** Component strings are USV sequences; JS strings are WTF-16. - Lowering a JS string with lone surrogates uses WebIDL `USVString` - replacement semantics (U+FFFD). Guest→host lift via `TextDecoder`; - host→guest lower via `TextEncoder.encodeInto` directly into guest memory. - `latin1+utf16` is implemented in the v1 interpreter (wit-bindgen guests - themselves use utf8). -- **Numbers.** `u64`/`s64` ↔ `BigInt`; everything else ↔ `number`. - `list` ↔ `Uint8Array` (copy; views into guest memory are never - exposed — with one deliberate, scoped exception: the `stream` - direct-access sessions of contracts/embedder-api.md §"Streams and - futures" hand the callback a view over the peer guest's landing zone or - unread bytes, valid only for that synchronous callback, so an external - byte mover's last hop can BE the one ABI copy). Both directions are bulk - copies: lift via a `Uint8Array` slice, lower via `Uint8Array.set` - (issue #54 — the per-element interpreted store cost ~45 ns/byte and - capped host→guest byte traffic at ~22 MB/s). Stream payload copies share - these paths, and u8 stream chunks stay `Uint8Array` through host buffers - too, so a host-side stream read costs exactly the one rendezvous copy. - Lists of the other flat element types (bool, s8, u16–u64/s16–s64, - f32/f64) keep their plain-array host shapes but also copy bulk, through - TypedArray views with the deterministic profile's NaN canonicalization - preserved in both directions (issue #67); the platform's - little-endianness is a named assumption checked once, with the DataView - per-element path as the big-endian fallback. `char` stays per-element - (its lift is per-element USV validation). -- **Memory views** are re-acquired after any call that can grow memory - (`ArrayBuffer` detach on `memory.grow`). -- **Resources.** Host-facing handles are classes with `Symbol.dispose` - (TS `using`), an explicit `[Symbol.dispose]()`/`drop()`, and a - `FinalizationRegistry` backstop for leaks. (Backstop-vs-teardown ordering - policy: open, [#10](https://github.com/polymorph-components/polyengine/issues/10).) -- **Destructors.** Per spec (CanonicalABI.md §`canon resource.drop`): the dtor - is a core function `[rep] -> []`, invoked as a normal **non-async** - cross-component call — *"the destructor may not block. However, the - destructor may spawn a cooperative thread that does."* Dtor entry into a - live instance is valid; a poisoned implementing instance refuses (§6 - divergence, with the same-instance exemption preserved for self-drops), - and a trapping dtor poisons the **implementing** instance - (`runtime/src/cabi/handles.ts` `callDtorGated`). - Host policy: - - CM-level blocking in a dtor → deterministic trap (falls out of general - sync-task rules). - - Host-import latency is invisible to CM semantics; a dtor calling a - `Suspending` host import is legal but needs a suspension-legal stack: - JS-initiated drops (`using`, FinalizationRegistry) enter via a `promising` - trampoline (`ResourceTypeInfo.dtorHost`, wired by the executor in jspi - mode for suspension-capable dtors — a non-suspendable dtor keeps the - exact synchronous path, avoiding the promising microtask hop's - one-turn entered window; the async entry bracket is held until the - activation settles, tracked in `pendingHostCalls`). **Known - limitation**: a *guest*-initiated drop reaches the dtor through a JS - trampoline frame, not the §5 pure-wasm funcref path — a Suspending - import under it is a deterministic JSPI frame-rule trap, not a - supported suspension. The pure-wasm dispatch path is future machinery; - until then §5's "guest-initiated dtor calls route through generated - wasm" is aspiration, not description. - - Host-held own handles carry lend tracking mirroring `num_lends` - ([#86](https://github.com/polymorph-components/polyengine/issues/86)): - drop/GC-backstop defer while lent; a backstop dtor trap poisons the - implementing instance and lands on the host-failure channel (never - `catch {}`-swallowed). - - Upstream spec findings related to drops and backpressure are tracked in - [upstream-component-model-repo-findings.md](../upstream-component-model-repo-findings.md), - the single source for component-model issue/PR filing. Implementation is - sync-only drop regardless of upstream timing. -- **Component `value` imports/exports** (the component-level `value` - definition feature): wasmtime doesn't implement them; excluded from parity - scope. Note the official suite's `test/values/` directory is **not** this - feature — it is plain canonical-ABI value-passing tests (`canon lift` with - memory options) and is fully in scope. -- **Reentrance**: the spec has no reentrance gate; the only entry refusal is - the poisoned-instance divergence (§6, enforced per §4.3 item 4). +The [Canonical ABI][CanonicalABI.md] and pinned `definitions.py` govern +lift/lower behavior. Public host shapes are specified in the +[embedder contract](../contracts/embedder-api.md#value-mapping-normative). + +**Strings.** Host strings are plain JS strings without encoding provenance. +Lowering treats them as UTF-16 code units and replaces lone surrogates +with U+FFFD (WebIDL `USVString`). UTF-8 and UTF-16 lifts use fatal +`TextDecoder`s; latin1 uses a byte-to-code-point mapping because the +WHATWG `latin1` label means Windows-1252. All three CABI encodings, +including `latin1+utf16`, are implemented. + +The current UTF-8 lowering path copies an ASCII prefix directly, then on +non-ASCII input reallocates to the worst-case size, uses +`TextEncoder.encode`, copies the encoded suffix, and shrinks if needed. +UTF-16 uses explicit little-endian encoding; compact strings begin as +latin1 and widen when necessary. There is no `encodeInto` fast path. +See `runtime/src/cabi/strings.ts`. + +**Numbers and lists.** `u64`/`s64` use `bigint`; other numeric types use +`number`. NaNs follow the deterministic profile. `list` uses copied +`Uint8Array`s; other lists retain ordinary array host shapes. Flat numeric +lists use bulk TypedArray paths on little-endian hosts, with NaN +canonicalization and a DataView fallback for big-endian hosts. `char` +requires per-element Unicode scalar validation. + +**Memory lifetime.** Views are reacquired after calls that can grow memory. +Ordinary lifted lists never expose guest memory. The explicit +`stream` direct-access API is the narrow exception: its callback may +access the peer's landing zone or unread bytes only during that +synchronous callback. See +[Streams and futures](../contracts/embedder-api.md#streams-and-futures). + +**Resources.** Host-facing handles are classes with `drop()` and +`Symbol.dispose` for explicit disposal, plus a `FinalizationRegistry` +backstop. Finalization is not deterministic cleanup. Host-held owns track +lends; explicit disposal invalidates the wrapper immediately but defers +destruction while lent, as does the backstop. Backstop +failure is reported through the host-failure channel, not swallowed. +Backstop-versus-teardown policy remains tracked in +[#10](https://github.com/polymorph-components/polyengine/issues/10). + +**Destructors.** `canon_resource_drop` lifts a core `[rep] -> []` +destructor with synchronous canonical options. The destructor may not +Component-Model-block, though the spec permits spawning an explicit +thread that blocks without preventing the destructor's implicit thread +from returning. This does not imply support for the deferred explicit-thread +built-ins (§11). Both guest- and host-initiated drops of guest resources use +`createDtorEntry` in `runtime/src/exec/boundary.ts`, creating a fresh +synchronous task and implicit thread rather than borrowing the caller's +task. A missing destructor still goes through that lift machinery. + +Guest-initiated drops use the synchronous drive and must finish before +returning; thenables are refused. Their JS dispatch frame also prevents +JSPI suspension (§5). Host-initiated drops can use a `promising` entry +for a suspension-capable destructor: host-import latency does not itself +constitute CM blocking. `drop(): void` does not wait for that activation's +tail; the store drives completion, and asynchronous failures surface on +the host-failure channel. The completion Promise is not itself registered +as external host work; genuine host imports register their own waits. + +Destructor entry into a live instance is permitted. A trapping destructor +poisons its implementing instance, with propagation able to poison its +caller too. Poisoned-instance refusal retains the same-instance exemption +for self-drops. Host-implemented resources have no guest implementing +instance to enter or poison. + +**Component `value` definitions.** Component-level `value` imports/exports +are excluded from the wasmtime parity target because the frontend does not +implement them. The official `test/values/` directory tests ordinary CABI +value passing and remains in scope. ## 8. Performance strategy -Requirement: not critical now; must become fast **without rearchitecting**. - -- **Cross-component calls are already the fast path**: FACT adapters, pure - wasm, no JS in the hot path. Nothing to do later. -- **Host boundary** has two executors over one IR — specified in - **[contracts/descriptor-ir.md](../contracts/descriptor-ir.md)**: - - The shim emits **CABI descriptor tables** (a compact ops IR per function). - - v1: a generic interpreter walks descriptors. CSP-clean, everywhere. - This is what ships today; it has been fast enough for every consumer - gate so far. - - v2 (when a gap is measured — gated on - [#17](https://github.com/polymorph-components/polyengine/issues/17), not the calendar; - [#8](https://github.com/polymorph-components/polyengine/issues/8)): a generator from the - same descriptors to specialized JS **modules** — emission-only, never - `eval`/`new Function` (§3's CSP invariant). One mechanism, two - invocation times: a deploy-time AOT step, or — on server hosts - (Deno/Node, no CSP) — first-load emission into a cache directory and - `import()`; pre-warming and freezing that cache *is* the AOT step. - Browsers running dynamically-loaded components stay on the interpreter. - The generated-module contract is AOT-shaped from day one (explicit - linking context; no closure capture of live runtime state) so both - invocation times share one artifact. Two executors over one IR double - as a differential-testing oracle — exercised by importing emitted - modules, i.e. the production delivery mechanism itself. -- Disciplines adopted from the start because they're hard to retrofit: slab - handle tables; no per-call closure/object allocation on hot paths; view - reuse with grow-aware invalidation; `encodeInto` for strings. -- Future options, noted not planned: JS string builtins (now widely shipped) - for string-heavy host boundaries. Deploy-time unbundling (real URLs per - module → engine code-cache hits) is a packaging concern independent of the - executor choice and lives on the caching track (§10, - [#7](https://github.com/polymorph-components/polyengine/issues/7)). +The shipping host-boundary executor is the generic CABI descriptor +interpreter, governed by +[contracts/descriptor-ir.md](../contracts/descriptor-ir.md). FACT already +generates wasm adapters for cross-component conversions, but these may +call runtime intrinsics; they are not a guarantee of a JS-free hot path. +Current optimizations include bulk list copies, byte-stream paths, slab +handle tables, and grow-aware memory views. String implementation details +are in §7. + +**Future, not implemented:** emit specialized JS modules from the same +descriptors when measurements justify the additional executor +([#8](https://github.com/polymorph-components/polyengine/issues/8), +[#17](https://github.com/polymorph-components/polyengine/issues/17)). +The proposed delivery is deploy-time emission, or server-side emission +into a cache followed by `import()`, never `eval`/`new Function`. +Dynamically loaded browser components would retain the interpreter. +Differential tests between interpreted and emitted execution belong to +that work; they are not a current gate. + +Translation time, artifact size, and call throughput depend on component, +engine, and build configuration. There is no universal startup or +throughput bound. Measure the relevant workload before adding another +execution path or changing deployment packaging. ## 9. Bindings generation -- A Rust CLI crate (`crates/bindgen`) built on **wit-bindgen-core**, consuming - `wit_parser::Resolve` + `WorldId`. **WIT source is the input** — the plan - cannot reproduce high-fidelity bindings (docs are lost in binaries, feature - gates are resolved away, aliasing is flattened) and the - bindings-before-any-component workflow requires WIT anyway. -- Output: TypeScript — typed world/interface APIs, `.d.ts`, resource classes - (`using`-compatible), JSDoc from WIT doc comments, honoring - `@since`/`@unstable` gates. -- Generated bindings import the runtime through a configurable base - (`--import-base`), defaulting to the versioned JSR specifier - `jsr:@polyengine/runtime@^` (derived at build - time, never hand-written). A path or URL base addresses files - (`{base}/{module}/mod.ts`); a bare or registry base addresses package - exports (`{base}/{module}`) — `--import-base --help` states the rule and - its scheme fallback in full. The in-repo fixtures use the relative base - `../../../src` so `deno check` stays offline and on this checkout's source. -- Host-facing value conventions (error model, stream/future wrappers, - variant/option/result shapes, resource classes, module-per-interface - authoring) are governed by the embedder conventions contract - (**[contracts/embedder-api.md](../contracts/embedder-api.md)**). Bindgen - also emits host-side types for **import worlds** (what an embedder must - provide), not only export-side facades — the consumers' host modules - ([consumers.md](consumers.md)) are the reference consumers of that surface. -- **Skew protection, the wasmtime way**: the generator embeds a canonical - structural digest of the expected world into the bindings - ([contracts/digest.md](../contracts/digest.md)); `instantiate()` verifies it - against the loaded component's types (already computed by the translator) - and fails fast with a useful diff. Compile-time fidelity from WIT; load-time - truth from the binary. -- Secondary, degraded mode: bindings from a component binary via its decoded - types (structure only, no docs). For third-party components; never primary. -- Guest-side bindings are stock wit-bindgen (Rust et al.) — that toolchain is - the compatibility target, exercised by its own runtime tests (§11). -- Version pinning: wit-parser/wasm-tools pinned to the same versions as the - translator's wasmtime, so WIT feature resolution matches. +`crates/bindgen` reads a WIT file or directory through `wit_parser::Resolve` +and selects a `WorldId`. It emits world/interface types, resource class +declarations, host-provider types, an expected structural world digest, +and a TypeScript instantiation wrapper. Guest-side bindings still come from external toolchains +such as wit-bindgen. + +WIT is the source input because component translation does not retain its +source-level fidelity. The current CLI emits a `.ts` file; it does not +offer a separate `.d.ts` or component-binary generation mode. `.d.ts` +files in the npm distribution come from the package build, not this CLI. + +WIT documentation/stability-aware output and a degraded binary-input mode +remain design targets, not current CLI capabilities. The latter could +recover structural types but not WIT source documentation. + +The generated instantiation wrapper checks its digest against the loaded +plan, then delegates to the runtime, which constructs the facade and +resource classes and adapts values. Generated resource declarations are +not implementations that callers can import as constructors. The digest +check follows [contracts/digest.md](../contracts/digest.md). +This detects structural skew between bindings and component types. It +does not authenticate the plan (§10). + +`--import-base` controls runtime imports. A path or URL addresses source +files (`{base}/{module}/mod.ts`); a bare or registry specifier addresses +package exports (`{base}/{module}`). The default JSR range derives from +`runtime/deno.json` at build time. **Development manifests name the next +release**, which may not exist in the registry: use an explicit compatible +released base or the local source when generating from a checkout. +In-repo fixtures use `../../../src`. + +The translator and bindgen share a pinned wasm-tools release train. +Host value shapes, import resolution, errors, async behavior, and resource +classes remain governed by the +[embedder contract](../contracts/embedder-api.md). ## 10. Caching -Two independent layers; nothing may *depend* on the second. - -1. **Artifact cache (ours, bytes only).** `runtime/src/cache/` — - content-addressed by `(component sha256, translator build hash, features)`; - deterministic translation makes this trivial. Storage: Cache API in - browsers (`webCache`), a cache directory in Deno (`dirCache`). Skips the - translation stage on reload. The plan is stored, never the component bytes - (the plan slices the component by offset; whoever holds the cache key - already holds the bytes). -2. **Engine code caches (opportunistic).** Chrome's wasm code cache is keyed - by URL but anchored to the **HTTP resource cache entry** (invalidation via - 304/200 semantics + V8 version), applies only to - `compileStreaming`/`instantiateStreaming`, and only to modules **≥ 128 kB** - after full tier-up. Consequences: - - Service-worker-**synthesized** responses get streaming *compilation* but - no persistent code cache (no HTTP cache entry to anchor to). Same - conclusion in Firefox (alt-data on HTTP cache entries). Safari: no - persistent wasm code cache known. - - FACT adapters are kilobytes — under the threshold, never code-cached - anyway. Only large sliced core modules matter; recurring cost is - re-tier-up CPU, not startup latency (Liftoff is fast). - - If a deployment has a build step: run the translator there (same wasm, - under Deno) and publish artifacts at real URLs → full engine caching with - zero tricks. Optionally warm via service-worker install-time - `compileStreaming` of those real URLs. - - Empirical verification of the code-cache behavior is open: - [#7](https://github.com/polymorph-components/polyengine/issues/7). - -Trust boundary (recorded lean, not yet forced by anything): trust locally-run -translation; never trust artifacts that did not come from the local cache -keyed by component hash. The runtime re-validates plan structure at load -(strict `formatVersion`, schema checks) but does not re-verify that artifacts -faithfully derive from the component bytes. Consequence for embedders — write -access to a cache root is worth about what write access to the component files -is worth: see [security.md](security.md) "The artifact cache is a trust input", -which carries the pre-warmed read-only-cache recipe. - -No cache failure may fail a translation -([#196](https://github.com/polymorph-components/polyengine/issues/196)): a -`get`/`put`/self-heal-eviction failure — an unwritable root included — degrades -to a fresh translation, reported only through `translateCached`'s opt-in -`onCacheError`. That is what makes a read-only cache root a usable deployment -rather than a crash. The public `evict()` still throws for explicit callers. +**Artifact cache.** `runtime/src/cache/` stores the plan and FACT adapters, +not the original component bytes. Its key includes the component SHA-256, +translator build hash, and feature settings. `webCache()` uses the Cache +API; `dirCache()` uses a Deno filesystem directory. A hit skips +translation, not core-module compilation or instantiation. + +`translateCached` requires a non-null translator `buildHash`, including on +a hit. `Translator.create(bytes)` computes it; +`Translator.fromExports(ns, { buildHash })` accepts a known asset hash. +The packaged loader's Deno wasm-module path currently supplies no hash, +so that translator cannot be used directly with `translateCached`. + +Cache I/O failures must not fail an otherwise valid translation: +`translateCached` falls back to fresh translation on `get` failure and +returns fresh artifacts even if `put` fails. Backend self-heal eviction +is best-effort; explicit `evict()` still reports errors. +`onCacheError` reports caught `get`/`put` failures, while backend failures +already converted to misses need not produce a callback. Component +validation failures still propagate. Correctly populated entries can be +read from a read-only cache, with a translator available for misses. + +**Current directory-cache limitation:** `put` preserves adapter names such +as `adapters/0.wasm` beneath its own `adapters/` directory but does not +create the additional nested directory. Normal adapter-bearing writes +therefore fail and `translateCached` returns fresh artifacts without +populating that entry. Adapter-free writes are not implicated, and +correctly laid-out entries can still be read. Verify actual +`fromCache: true` results after prewarming; successful translation alone +does not establish that the cache was populated. + +**Engine code caches.** These are independent, opportunistic platform +optimizations; correctness and artifact caching do not depend on them. +V8's [published code-cache description](https://v8.dev/blog/wasm-code-caching) +ties persistence to streaming compilation, HTTP cache entries, module +size, and tier-up. Thresholds and policies are engine/version details, +not polyengine guarantees. Synthesized responses or sliced component +bytes should not be assumed to get the same persistent cache behavior as +standalone modules served at real URLs. Deployment-specific verification +is tracked in [#7](https://github.com/polymorph-components/polyengine/issues/7). + +**Trust.** Plans and adapters are trusted inputs whether produced locally, +loaded from a cache, or shipped as build artifacts. The runtime checks +structure, format versions, and component-byte identity, but does not +prove that supplied artifacts are what the translator would produce. +Protect them like executable inputs. See +[The artifact cache is a trust input](security.md#the-artifact-cache-is-a-trust-input) +for deployment guidance. ## 11. Conformance and testing -There is no single official conformance suite; the corpus is assembled: - -| Source | What | How used | -|---|---|---| -| [WebAssembly/component-model] `test/` | official, growing WAST suite: `binary/`, `validation/`, `linking/`, `resources/`, `values/`, `async/` | git submodule; primary gate, all directories in scope. Independent check on the wasmtime-frontend reuse. | -| same repo, `design/mvp/canonical-abi/definitions.py` + `run_tests.py` | executable CABI reference | lift/lower edge-case tests ported to TS unit tests (`runtime/tests/`) | -| wit-bindgen runtime tests | guest programs exercising bindings | Rust guests, sync and async (wit-bindgen + `wasm-tools component new`), run against our host = the executable wit-bindgen-compat claim (`examples/guests/`) | -| wasmtime `tests/misc_testsuite/component-model/` | engine-grade wast corpus | supplementary coverage | -| polymorph conformance matrices (webcrypto/websocket/webrtc/tls, driven by polymorph-test) | per-interface implementation×environment conformance suites over real WIT surfaces | consumer lane ([consumers.md](consumers.md)): `ct-runner` executes them | -| experiment-mosh gates + minimized repros (`compose-async-tdz`) | composed 3-component client: mixed sync/async exports, background pumps, resources re-exported across interfaces, componentize-go guest | strongest known real-workload exercisers — this family surfaced ≥5 distinct jco defect classes no WAST corpus expresses (`tools/smoke-c0/`) | - -Harness pipeline: an offline Rust step (`crates/testgen`) converts `.wast` -into JSON commands + `.wasm` binaries by driving the `wast` and -`json-from-wast` crates (the `wasm-tools json-from-wast` implementation) as -libraries — the same conversion wasmtime's own wast runner performs -in-process; the schema is upstream's, unmodified. The harness classifies -each binary as core module vs component from its preamble (V8 cannot even -validate component binaries). The TS harness executes the JSON identically -under `deno test` and in browsers (`tools/browser/run-lane.ts`: static -server + automated Chromium / Firefox-with-pref / WebKit, with per-lane -expectation overlays and stale-delta detection) — and directly under engine -*shells* and server runtimes (`tools/shell/run-lane.ts`: SpiderMonkey `js`, -JSC `jsc`, and node/bun via a host preamble; same classification machinery, -no browser). Because the corpus is engine-shaped, the per-push/PR engine -gates are the **pinned shell lanes** (`sm-pinned` = the Firefox-release -shell matching the browser lane, `jsc-pinned` = a sha256-mirrored trunk -build, `node-pinned` = the node ≥ 26 runtime; `bun-pinned` rides along -findings-only until it has a track record; `tools/shell/pins.json`); -browser lanes run post-merge, verifying the embedding and shipped-channel -configs and gating the prerelease. Trunk/nightly shells and a Deno-canary -probe run weekly as findings-only canaries (`.github/workflows/canary.yml`) -with a capability-probe preamble that surfaces wasm-proposal landings -(multi-memory, GC, EH, memory64, …) before the corpus exercises them. - -Also planned: differential testing of the interpreter vs emitted -specialized-JS modules (§8, [#8](https://github.com/polymorph-components/polyengine/issues/8)); -differential fuzzing -against native wasmtime with `wasm-smith`-generated components -([#9](https://github.com/polymorph-components/polyengine/issues/9)). - -Epistemic note: because our frontend *is* wasmtime's, wasmtime-derived tests -partly test wasmtime against itself — weight the official suite and -definitions.py ports accordingly. And passing suites is necessary, not -sufficient: the consumer workloads found real defects that no WAST corpus -expresses — which is why the in-repo smoke legs (`just smoke-tls`, -`just smoke-c0`) are kept as gates, and the consumers' own matrices as -high-yield sanity checks ([consumers.md](consumers.md)). +| Source | Coverage | +|---|---| +| Pinned [Component Model][WebAssembly/component-model] `test/` corpus | Binary format, validation, linking, resources, values, and async behavior | +| `definitions.py` and `run_tests.py` | Reference-derived lift/lower tests and fixtures in `runtime/tests/` | +| Rust/wit-bindgen fixtures | Real guest ABI and runtime integration in `examples/guests/` | +| External componentize-go consumer artifacts | `wasi/tests/integration_engine_go_test.ts` skips when the external artifact or shim is absent; consumer smoke tools provide additional external coverage | +| Polymorph conformance suites and consumer smoke gates | Host-provider interfaces, composed components, background tasks, and resource flows; see [consumers.md](consumers.md) | +| `runtime/tests/conventions/` | Committed transcripts of the public host ABI, gated with protocol versioning | + +Wasmtime's component-model tests are supplementary reference material +([references.md](references.md)), not an additional corpus executed by the +current generation or gate paths. + +`crates/testgen` uses `wast` and `json-from-wast` to convert WAST into JSON +commands and wasm binaries. The TS harness distinguishes core modules +from components by their preambles and executes the commands across Deno, +browsers, engine shells, and server runtimes. Engine trap-message +normalization belongs to `TRAP_MESSAGE_EQUIVALENTS` in +`harness/src/runner.ts`, not the runtime. + +Expected failures are classified, not counted as conformance. Known +classes include deferred thread support +([#12](https://github.com/polymorph-components/polyengine/issues/12)), +sync scheduling gaps +([#249](https://github.com/polymorph-components/polyengine/issues/249)), +and upstream-unimplemented features +([#248](https://github.com/polymorph-components/polyengine/issues/248)). +Per-lane overlays distinguish engine limitations from runtime failures. +The current base classification is in [harness/src/xfail.ts](../harness/src/xfail.ts). +Unexpected failures and stale expected failures fail their gate. + +The [justfile](../justfile) is the command surface; CI job bodies live in +[.github/justfile](../.github/justfile). Required PR checks use the pinned +shell lanes alongside core tests. Browser lanes run post-merge and gate +prereleases; findings-only lanes do not become required checks merely by +running there. Weekly canaries probe newer runtimes and engine features. +`just gates` adds consumer smoke checks that require external checkouts. + +Passing suites is evidence, not proof. Wasmtime-derived tests share code +with the translator; spec-derived tests and independent consumer workloads +cover different failure modes. Native-wasmtime differential fuzzing is +future work ([#9](https://github.com/polymorph-components/polyengine/issues/9)), +as is interpreter-versus-emitted-module testing (§8). ## 12. Risks -| Risk | Severity | Mitigation | -|---|---|---| -| wasmtime internal API churn (`wasmtime-environ` is internal/unstable) | medium, recurring | shim isolation + version pinning (a git rev of `main` since the 49-dev bump, not a crates.io release); upgrades are deliberate events ([#1](https://github.com/polymorph-components/polyengine/issues/1)) | -| JSPI phase-4 drift | medium | small trampoline surface, centralized; track proposal | -| Safari: stable-channel JSPI status unverified | accepted | floor is explicit; JSPI unflagged on WPE 26.5, and Safari Technology Preview carries JSPI + multi-memory — stable channel is the remaining gap ([#11](https://github.com/polymorph-components/polyengine/issues/11)); callback-ABI consumers don't need it | -| JSC/SpiderMonkey engine gaps | medium | Deno-first dev; file upstream. SpiderMonkey JSPI is clean over the full corpus (pref-flipped); **JSC's real gap was missing multi-memory** (capped the WebKit lane — the CABI routinely needs >1 memory per module), not JSPI. Resolved in WebKit trunk: default-on from the webkit-2342 playwright roll, lane at effective parity there ([#11](https://github.com/polymorph-components/polyengine/issues/11)) | -| Testing wasmtime-with-wasmtime blind spots | medium | official suite + definitions.py ports as independent checks; consumer suites as real-workload sanity checks | -| CSP variance in embedders | low | baseline needs only `wasm-unsafe-eval` — an invariant, no path may require full `unsafe-eval` (§3); specialized JS is emission-only, deploy-time or server-side cache import ([#8](https://github.com/polymorph-components/polyengine/issues/8)) | -| Consumer coupling churn: 7+ downstream repos tracking pre-1.0 plan/contract formats | medium | caret-honest registry releases ([#16](https://github.com/polymorph-components/polyengine/issues/16)): still 0.x/unstable, compatible within a minor line, breaking changes bump the minor — consumers couple by caret; `pre-` prerelease artifacts (exact pins) and git refs track `main` between releases; strict formatVersion equality already fails loud | -| Consumer scope creep pulling WASI implementations into the core | medium | the wasi package is a separate deliverable with consumer-driven scope; §2 non-goal stands | -| Host-boundary perf vs jco's generated JS (v1 interpreter) | low-medium | translation throughput measured (multi-MB components in tens of ms); cutover benches tracked with [#8](https://github.com/polymorph-components/polyengine/issues/8) | +| Risk | Current response / limitation | +|---|---| +| Unstable wasmtime frontend API | Pin dependencies, isolate mapping in the shim, and run integration gates on upgrades | +| Engine capability gaps | Explicit per-lane expectations; do not infer stable Safari or older runtime support from trunk results | +| JSPI frame restrictions | Wasm adapters and centralized entry wrapping; guest-initiated suspending destructors remain unsupported | +| Scheduler and shared-frontend blind spots | Seeded scheduling, reference tests, regression fixtures, and consumer gates; none establish exhaustive parity | +| Host-boundary interpreter cost | Measure workloads; specialized-module emission remains future work | +| Pre-1.0 interface churn | Compatible minor-line releases, explicit breaking-version events, and strict plan-format checks | +| Artifact or host-provider authority | Trusted artifacts and explicit capability grants; WASI confinement is not a hostile-guest sandbox ([security.md](security.md)) | +| Resource cleanup timing | Prefer explicit disposal; finalization and teardown ordering have limits (§7) | [WebAssembly/component-model]: https://github.com/WebAssembly/component-model [CanonicalABI.md]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md diff --git a/docs/consumers.md b/docs/consumers.md index 5ca0fed..35bd6c3 100644 --- a/docs/consumers.md +++ b/docs/consumers.md @@ -1,136 +1,71 @@ -# polyengine — consumer adoption: the polymorph track - -The first production consumers are the [polymorph-components] family — -`polymorph-{webcrypto,websocket,webrtc-datachannels,tls,test,iroh}` — and -experiment-mosh (a mosh client/proxy tunneled over the iroh endpoint -component). All run the same triangle {wasmtime host, JS host, in-guest -provider}. Their JS host was jco (a pinned fork), whose structural defects -in exactly this project's core territory — scheduler task admission, -cross-task wakeups, composed async cross-component calls, cancellation, -codegen emission — blocked their plans; replacing jco there is this -project's adoption target ([architecture.md §1](architecture.md)), and -jco-convention compatibility is explicitly not part of it -([§2](architecture.md)) — the consumers have no external dependents and -port to conventions designed fresh -([contracts/embedder-api.md](../contracts/embedder-api.md)). - -The cutover is delivered: every standing consumer matrix runs polyengine as -its JS host. polymorph-webcrypto additionally retains jco rows by design — -the Chromium side of its platform-gap ledger. What replacing jco does not -replace: componentize-js/-go (guest production — out of scope per -[architecture.md §2](architecture.md); their output components are ordinary -inputs to us) and the wasmtime host legs (the native story). - -## Standing conventions - -- **Co-evolution, not compatibility.** Conventions are designed against - the consumers' host modules as reference implementations; they port; both - sides upgrade deliberately. Registry releases are caret-honest — still - 0.x/unstable, compatible within a minor line, breaking changes bump the - minor — so consumers couple via caret constraints - (`jsr:@polyengine/*@^0.4.0`), with `pre-` GitHub release - artifacts and git references for tracking `main` between releases. -- **WASI interfaces are design inputs even though implementations stay - out of core.** The conventions must make wasi p2 idioms (pollables, io - streams, error-code enums, resource-heavy surfaces) and p3 idioms - (stream/future-bearing signatures, async resource methods, - error-context) natural to implement in JS — whoever adopts this host - writes shims against these conventions, and the broader ecosystem's - most important interfaces are exactly these. The `wasi/` package is the - executable check. -- **The application owns the import map** (the WICG import-maps stance as - a family convention). Host-module packages import `@polyengine/*` by - bare specifier and carry **no** mapping for it in any config a consumer - resolves through — standalone dev/test mappings live outside the package - directory (sibling repo root), because Deno applies package-local config - to package files and a package-carried pin silently overrides the - consumer's root import map (observed in the wild: four extra runtime - copies, one per sibling pin). Consumers assert the invariant - mechanically: after `deno install`, the resolved graph contains - **exactly one** polyengine source (for a vendoring consumer: zero remote - polyengine URLs in the lockfile — a one-line CI guard). Cross-boundary - brands are process-global symbols via `@polyengine/protocol` - (contracts/embedder-api.md §"Module identity"), so a violation degrades - to a diagnosed inefficiency instead of a latent `instanceof` failure — - and host modules MUST import `@polyengine/protocol` at most (§"The - host-ABI surface and its version": the runtime's exported surface is - application-only), keeping runtime selection entirely with the deploying - application. -- **Their suites are engine sanity checks, not gates** (operator ruling). - This family surfaced multiple distinct jco defect classes that no WAST - corpus expresses (long-lived composed workloads, background pumps, - cross-task wakeups, codegen-shape triggers) — and several polyengine - runtime defects the same way — so running them is high-yield. But - everything on both sides is unstable and co-evolves in tandem: a - consumer-suite delta is a finding to triage, never a blocker for - upstreaming or release. - -## Pins and the scope rename - -This repository moved from `lann/deltic` to -[`polymorph-components/polyengine`](https://github.com/polymorph-components/polyengine) -and its JSR scope from `@deltic` to `@polyengine` — a hard break with no -compatibility shim, deliberately **not** synchronized with the consumers: -JSR versions are immutable, so `@deltic/*` through `0.2.1` stays resolvable -forever and an un-migrated consumer keeps building; each repo migrates in -one step, on its own schedule. What a consumer changes when its turn comes: - -| Surface | Before | After | -|---|---|---| -| package specifiers | `jsr:@deltic/{runtime,protocol,translator,wasi,ct-runner}` | `jsr:@polyengine/…` | -| version line | `0.2.x` | continues at `0.3.0`+ (lower numbers would collide with surviving deltic-era tags, #206) | -| cross-copy brands | `Symbol.for("deltic.witError/1")` and siblings | `Symbol.for("polyengine.componentException/1")` and siblings — see [embedder-api.md](../contracts/embedder-api.md) §"Module identity" | -| environment | `DELTIC_TRANSLATOR`, `DELTIC_SCHED_SEED`, `DELTIC_DRIVE_TRACE` | `POLYENGINE_*` | -| release assets | `deltic-embedder.mjs`, `deltic-translator-shim.wasm` | `polyengine-*` | -| ct-runner envelope target | `deltic/host` (the CLI default) | `polyengine/host` | - -The brand keys are the one that fails quietly: a graph mixing a `@deltic` -copy and a `@polyengine` copy produces two disjoint brand namespaces, so -cross-copy checks simply return false rather than reporting a copy -conflict. Migrate a repo's engine dependency in one step, never partially. - -Deltic-era `v*` GitHub releases were deleted during the rename; their tags -survived (which is why the `@polyengine` line starts at `0.3.0`), and every -deleted release has a surviving `pre-` prerelease twin at the -same commit carrying byte-identical assets — the prerelease tags are what -consumers actually pin when fetching release assets. Identifiers the -consumers own — lane names (`deltic-deno`), directories (`host-deltic`), -expectation keys, CI check names — are theirs to rename or keep; where this -document names one, it is quoting their spelling. - -Every repo gates module identity on ONE resolved engine version (pin gates; -iroh's "one runtime, no raw URLs" identity gate). Upgrades ride the -lockstep line: runtime and translator move together by construction. - -## Deno substitutes for Node - -Node is **not a consumer requirement.** Deno functionally substitutes -across the whole consumer capability surface — WebRTC via -`node-datachannel/polyfill` (the polymorph Node legs' exact dependency) or -`werift` (pure TS), built-in `WebSocket` and WebCrypto, UDP/TCP via native -and node-compat APIs — all verified by the consumers' own polyengine legs. -Node stays a nearly-free *distribution* target via npm (the callback-ABI -path needs no JSPI flag) plus the pinned `node-pinned` conformance lane -(architecture.md §3). - -## In-repo consumer artifacts - -The reference host modules (`ports/{websocket,webcrypto,webrtc}`) and the -iroh endpoint exam were developed here and upstreamed as the consumers' own -copies; their repos run the living gates (history: `git log -- ports`, -`git log -- exams/iroh-endpoint`). What remains in-repo: - -| Path | What | Gate | -|---|---|---| -| `ct-runner` | L3 runner for the polymorph-test L1 contract; published in the lockstep set as `@polyengine/ct-runner`; polymorph-test's own glue pins it | golden/schema/shard/tags battery + CLI tests (`just test-ct-runner`); translate-only import analysis against the websocket consumer checkout; `tools/smoke-tls` drives it as source | -| `tools/smoke-c0` | consumer smoke legs (experiment-mosh `compose-async-tdz` repro, iroh exec-model probe incl. later export calls against a live detached pump, websocket suite translation, componentize-go translator throughput) | `just smoke-c0` | -| `tools/smoke-tls` | polymorph-tls conformance under polyengine ([#18](https://github.com/polymorph-components/polyengine/issues/18)) | `just smoke-tls`: translate 8/8; suites: zero failures, zero xfails on every composition — tag gating ([#25](https://github.com/polymorph-components/polyengine/issues/25), `ct-runner/src/tags.ts`) schedules per-target inapplicable cases to `not-applicable` exactly like their harness legs | - -Deferred consumer surfaces: iroh UDP direct path -([#4](https://github.com/polymorph-components/polyengine/issues/4)). - -Defects found in consumer code while running their artifacts are tracked in -[`../upstream-consumer-findings.md`](../upstream-consumer-findings.md) -(filing: [#15](https://github.com/polymorph-components/polyengine/issues/15)). - -[polymorph-components]: https://github.com/polymorph-components +# Consumer workloads + +The [polymorph-components](https://github.com/polymorph-components) family and +[`wosh`](https://github.com/lann/wosh) supply real composed workloads alongside +the WAST corpus. They exercise long-lived background tasks, cross-task wakeups, +resource-heavy host interfaces, mixed sync/async calls, and components produced +by Rust, componentize-js, and componentize-go. A small WAST or Rust fixture is +useful for isolating a defect, but does not replace this workload scope. + +Polyengine replaces the JavaScript component host, not guest-production tools +or native wasmtime hosts. Consumer suites are integration sanity checks and +sources of findings, not a claim that every consumer configuration works or an +independent release requirement. The local `just gates` recipe includes the +available smoke tools; CI cannot assume those external checkouts exist. + +## Dependency conventions + +- The application selects the runtime through its import map or package + manager. Host-provider packages import `@polyengine/protocol` at most, and + must not carry a package-local runtime mapping that overrides the application. + Keep standalone development mappings outside the provider package. +- Keep runtime and translator on the same lockstep release line. Protocol is + independently versioned. Published dependencies use compatible version + constraints and lockfiles; tracking `main` requires explicit git revisions or + `pre-` release-asset pins. See [Consuming](../README.md#consuming) + and the [host-ABI contract](../contracts/embedder-api.md) rather than copying a + version number from this document. +- Check the resolved graph after dependency updates. The intended application + graph has one engine source; a vendored application should not also resolve + remote copies. Cross-copy brands help diagnose mistakes but do not make + duplicate runtimes desirable. +- For an old `@deltic` consumer, migrate the entire engine dependency graph to + `@polyengine` together. The scopes use different brand namespaces, so a partial + rename can silently break cross-copy checks. Consumer-owned lane and directory + names need not match the package scope. + +WASI p2 pollables, I/O streams, and resources, and p3 streams, futures, and async +methods are part of the workload requirements. Implementations live outside +core, including this repository's [`wasi/`](../wasi/) package. Platform-specific +host choices and engine requirements belong to each consumer and +[architecture](architecture.md), not to a blanket Deno/Node equivalence claim. + +## Read-only smoke inputs + +External consumer checkouts are **read-only** during engine work. Check their +git status before and after any verification that uses them. Do not install, +regenerate, or build into those trees; any separately authorized build must use +scratch output or a redirected `CARGO_TARGET_DIR`. Do not update consumer pins +as a side effect of an engine test. + +The smoke tools consume prebuilt artifacts. Their paths are defined in +[`tools/smoke-c0/common.ts`](../tools/smoke-c0/common.ts), with `POLYMORPH_ROOT` +and `WOSH_ROOT` overrides. Record the consumer revision, artifact identity, and +engine revision when interpreting a result. An artifact on disk is not proof +that it was built from the checkout's current source. + +| In-repo tool | Scope | +| --- | --- | +| [`ct-runner`](../ct-runner/) | Runs the polymorph-test suite contract and emits structured per-case results; its own fixtures and tests live here | +| [`tools/smoke-c0`](../tools/smoke-c0/) | Composed async repro, iroh execution-model probe including later calls against a live detached pump, componentize-go translation timing, and websocket suite translation/import inspection | +| [`tools/smoke-tls`](../tools/smoke-tls/) | Translates TLS compositions and executes self-contained suites through ct-runner and WASI; per-target tags distinguish inapplicable cases | + +Missing consumer artifacts can be logged as skips without failing the smoke +process. A successful exit alone therefore does not establish that the intended +workloads ran. Report which artifacts executed, which were translation-only, +and all skips or not-applicable cases; do not retain an old all-green count as +a current guarantee. + +Consumer-code findings belong in +[`upstream-consumer-findings.md`](../upstream-consumer-findings.md). Filing them +in a foreign repository requires the operator's authorization. diff --git a/docs/references.md b/docs/references.md index a037c14..7f7e9e3 100644 --- a/docs/references.md +++ b/docs/references.md @@ -1,86 +1,99 @@ # polyengine — references -Canonical links contributors (human or agent) are likely to need. Versioned -links are pinned to the versions this repo pins; re-pin them together with -the dependency. +Use the checked-in dependency pins when investigating behavior. Links to +upstream `main` show current upstream work, not necessarily this checkout's +semantics. The authority policy, including the sole CM-3 exception, is in +[architecture §1](architecture.md#1-goals). ## Component Model spec (submodule: `third_party/component-model`) -- Explainer (text format, grammar, validation): - https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md -- Canonical ABI (lift/lower, options, built-ins, invariants): - https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md -- **Executable CABI reference** (the tie-breaking authority for runtime - semantics): - https://github.com/WebAssembly/component-model/blob/main/design/mvp/canonical-abi/definitions.py - — with `run_tests.py` and `diff.py` alongside -- Binary format: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Binary.md -- Concurrency model (0.3 tasks/streams/futures): - https://github.com/WebAssembly/component-model/blob/main/design/mvp/Concurrency.md -- WIT: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md -- Shared-nothing linking: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md -- Official WAST suite: https://github.com/WebAssembly/component-model/tree/main/test -- User-facing CM documentation: https://component-model.bytecodealliance.org/ -- Our upstream findings tracker: - [upstream-component-model-repo-findings.md](../upstream-component-model-repo-findings.md) +The [submodule](../third_party/component-model/) pins the spec, executable +reference, and WAST corpus used by this checkout. Its principal sources are: + +| Source | Purpose | +|---|---| +| [Explainer](../third_party/component-model/design/mvp/Explainer.md) | Text format, types, validation, canonical interface names | +| [Canonical ABI](../third_party/component-model/design/mvp/CanonicalABI.md) | Lift/lower, canonical options, built-ins, state rules | +| [definitions.py](../third_party/component-model/design/mvp/canonical-abi/definitions.py) | Executable semantic reference; use function names rather than line numbers | +| [run_tests.py](../third_party/component-model/design/mvp/canonical-abi/run_tests.py) | Reference tests; [diff.py](../third_party/component-model/design/mvp/canonical-abi/diff.py) compares reference and prose | +| [Binary format](../third_party/component-model/design/mvp/Binary.md) | Component binary encoding | +| [Concurrency](../third_party/component-model/design/mvp/Concurrency.md) | Tasks, streams, futures, and concurrency model | +| [WIT](../third_party/component-model/design/mvp/WIT.md) | Interface language | +| [Linking](../third_party/component-model/design/mvp/Linking.md) | Shared-nothing linking | +| [WAST suite](../third_party/component-model/test/) | Official test corpus | + +Current upstream sources are at +[WebAssembly/component-model](https://github.com/WebAssembly/component-model). +The [Component Model book](https://component-model.bytecodealliance.org/) +is introductory documentation, not the semantic tie-breaker. +Local discrepancies belong in +[upstream-component-model-repo-findings.md](../upstream-component-model-repo-findings.md). ## JSPI and engine support -- JSPI proposal Overview (the frame rule lives in "Restriction"): - https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md -- V8 JSPI introduction: https://v8.dev/blog/jspi -- Engine feature matrix: https://webassembly.org/features/ (data: - https://github.com/WebAssembly/website/blob/main/features.json) -- V8 wasm code caching (the URL/HTTP-cache anchoring facts in - [architecture.md §10](architecture.md)): - https://v8.dev/blog/wasm-code-caching -- Stack-switching proposal (JSPI's core-wasm sibling, context only): - https://github.com/WebAssembly/stack-switching +- [JSPI overview](https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md): + `promising`, `Suspending`, and the restriction on intervening JS frames. +- [V8 JSPI introduction](https://v8.dev/blog/jspi). +- [WebAssembly feature matrix](https://webassembly.org/features/): useful + context; project coverage is recorded in + [architecture §3](architecture.md#3-compatibility-targets), + [browser expectations](../harness/browser/expectations/), and + [shell expectations](../harness/shell/expectations/). +- [Shell pins](../tools/shell/pins.json): exact tested shell/runtime builds. +- [V8 wasm code caching](https://v8.dev/blog/wasm-code-caching): published + engine policy, not a portable cache guarantee. See + [architecture §10](architecture.md#10-caching). +- [Stack-switching proposal](https://github.com/WebAssembly/stack-switching): + related core-wasm work, not polyengine's current scheduling mechanism. ## wasmtime internals (pinned: wasmtime-environ **49.0.0-dev+4675ee1**, a git rev of `main`) -- Source at the pinned rev: - https://github.com/bytecodealliance/wasmtime/tree/4675ee16b703b33948073a5ff6b961367371e7a1/crates/environ/src - — notably `component::{Translator, Component, GlobalInitializer, CoreDef, - Trampoline, CanonicalOptions}` and `fact::Import`. FACT: `src/fact.rs` - (+ `src/fact/`), component translation: `src/component/` -- FACT design note ("polyfill for the component model in JS environments" is - an intended consumer): - https://github.com/bytecodealliance/wasmtime/blob/4675ee16b703b33948073a5ff6b961367371e7a1/crates/environ/src/component/translate/adapt.rs -- Wasmtime component wast tests (supplementary corpus): - https://github.com/bytecodealliance/wasmtime/tree/main/tests/misc_testsuite/component-model +The revision is declared in [Cargo.toml](../Cargo.toml) and resolved in +[Cargo.lock](../Cargo.lock). Update source links with the pin. + +- [Environ at the pinned revision](https://github.com/bytecodealliance/wasmtime/tree/4675ee16b703b33948073a5ff6b961367371e7a1/crates/environ/src): + component translation and plan structures under `component/`; fused + adapter generation in `fact.rs` and `fact/`. +- [Adapter translation](https://github.com/bytecodealliance/wasmtime/blob/4675ee16b703b33948073a5ff6b961367371e7a1/crates/environ/src/component/translate/adapt.rs): + how component linkage is translated into FACT adapters. +- [Component-model tests at the same revision](https://github.com/bytecodealliance/wasmtime/tree/4675ee16b703b33948073a5ff6b961367371e7a1/tests/misc_testsuite/component-model): + supplementary reference material, not a corpus executed by the current + project gates or an independent check of the reused frontend. ## Toolchain crates (pinned versions in lockfiles) -- wasm-tools repo (CLI + crates): https://github.com/bytecodealliance/wasm-tools -- `wast` crate (component-aware wast parsing, used by testgen): - https://docs.rs/wast/ -- `json-from-wast` crate (wast → JSON commands + binaries, used by testgen; - the `wasm-tools json-from-wast` implementation): https://docs.rs/json-from-wast/ -- `wasmparser` (0.258.x — must match wasmtime-environ): https://docs.rs/wasmparser/ -- `wasm-encoder`: https://docs.rs/wasm-encoder/ -- `wit-parser` (bindgen input): https://docs.rs/wit-parser/ -- wit-bindgen (guest toolchain, pinned **0.60.0**): - https://github.com/bytecodealliance/wit-bindgen — `generate!` macro docs: - https://docs.rs/wit-bindgen/0.60.0/wit_bindgen/macro.generate.html -- wit-bindgen runtime tests (compat corpus): - https://github.com/bytecodealliance/wit-bindgen/tree/main/tests +[Cargo.toml](../Cargo.toml) and [Cargo.lock](../Cargo.lock) govern the host +toolchain; individual [guest crates](../examples/guests/) pin their own +dependencies (for example, [hello](../examples/guests/hello/Cargo.toml)). +The wasm-tools release train must +agree with the wasmtime frontend. + +- [wasm-tools](https://github.com/bytecodealliance/wasm-tools): CLI and + libraries, including `wasmparser`, `wasm-encoder`, `wit-parser`, `wast`, + and `json-from-wast`. +- [wasmparser](https://docs.rs/wasmparser/0.258.0/wasmparser/): parsing and validation. +- [wasm-encoder](https://docs.rs/wasm-encoder/0.258.0/wasm_encoder/): core and component binary generation. +- [wit-parser](https://docs.rs/wit-parser/0.258.0/wit_parser/): bindgen's WIT input. +- [wast](https://docs.rs/wast/): WAST parsing; use the version in the lockfile. +- [json-from-wast](https://docs.rs/json-from-wast/): testgen's JSON-command and wasm-artifact conversion. +- [wit-bindgen](https://github.com/bytecodealliance/wit-bindgen) and its + pinned [0.60.0 `generate!` documentation](https://docs.rs/wit-bindgen/0.60.0/wit_bindgen/macro.generate.html): + guest bindings, distinct from polyengine's host-facing `crates/bindgen`. ## JS platform specifics -- WebIDL `USVString` conversion (our string-lowering semantics): - https://webidl.spec.whatwg.org/#idl-USVString -- `String.prototype.toWellFormed` (ES2024): - https://tc39.es/ecma262/#sec-string.prototype.towellformed -- `TextEncoder.encodeInto`: - https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder/encodeInto -- `TextDecoder` labels (note: "latin1" label decodes windows-1252, hence the - hand-rolled latin1 in `runtime/src/cabi/strings.ts`): - https://encoding.spec.whatwg.org/#names-and-labels -- `FinalizationRegistry` (resource backstop — read the caveats): - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry -- Explicit resource management / `using` (TS 5.2+, `Symbol.dispose`): - https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-2.html -- WebAssembly JS API spec: https://webassembly.github.io/spec/js-api/ -- Deno runtime docs (workspaces, `deno test`, `--v8-flags`): - https://docs.deno.com/runtime/ +- [WebIDL `USVString`](https://webidl.spec.whatwg.org/#idl-USVString) and + [`String.prototype.toWellFormed`](https://tc39.es/ecma262/#sec-string.prototype.towellformed): + lone-surrogate replacement during string lowering. +- [`TextEncoder.encode`](https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder/encode): + the current non-ASCII UTF-8 lowering path in `runtime/src/cabi/strings.ts`. +- [Encoding labels](https://encoding.spec.whatwg.org/#names-and-labels): + `TextDecoder("latin1")` means Windows-1252, not the CABI's latin1 mapping. +- [`FinalizationRegistry`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry): + nondeterministic resource backstop, not guaranteed cleanup. +- [Explicit resource management](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-2.html): + `using` and `Symbol.dispose`. +- [WebAssembly JS API](https://webassembly.github.io/spec/js-api/). +- [Deno documentation](https://docs.deno.com/runtime/). + +For deployment authority and cache trust, see [security.md](security.md). diff --git a/docs/security.md b/docs/security.md index a8fc34d..83ebd64 100644 --- a/docs/security.md +++ b/docs/security.md @@ -1,245 +1,239 @@ # Security posture -What this project's WASI implementation does and does not guarantee, and -what an embedder has to do itself. - -Read this before granting a guest filesystem or network access. +Read this before granting a component filesystem, network, or process +access. This page covers WASI provider authority and translated-artifact +trust; it is not a security certification of the engine or its dependencies. ## The headline -**The filesystem and network confinement in `@polyengine/wasi` is not a -security mechanism.** It is a correctness mechanism: it makes a guest's -requests land where the embedder said they should, and it refuses the -obvious ways of asking for something else. It is not a boundary that -holds against a guest actively trying to escape, and it is not a -substitute for an OS- or runtime-level sandbox. - -If untrusted input can flow into the paths a guest opens — or if the -guest itself is untrusted — the embedder must impose its own boundary. -Recipes are at the bottom of this page. - -## Why path confinement can never be a boundary +**The path and request checks in `@polyengine/wasi` are correctness +mechanisms, not a sandbox for hostile guests.** They reject invalid or +out-of-scope requests, but do not replace an OS- or runtime-level boundary. +If a guest is untrusted, or untrusted input controls its paths or network +destinations, the embedder must constrain the host process independently. -The filesystem backends confine guest paths to their preopens. The -provider layer resolves guest paths textually (`..` cannot climb past a -preopen, absolute paths and NUL are refused), and the node backend adds -physical containment on top: it resolves each path against the preopen's -real path before handing anything to the OS, and refuses what lands -outside. - -That is worth having, and it closes the escapes a guest reaches for -first. It cannot be a security boundary, for reasons that have nothing to -do with how carefully it is written: - -- **Hardlinks.** A hardlink inside a preopen pointing at an inode outside - it resolves entirely "within" the tree. No amount of path checking sees - this, because by every path-shaped measure it *is* inside. -- **Bind mounts and mount points.** Same shape: the path stays inside the - preopen, the bytes are somewhere else entirely. -- **Cross-process races.** Between the moment the implementation resolves - a path and the moment the OS acts on it, another process can swap a - component for a symlink. Closing this requires anchoring every - operation to a file descriptor (`openat`-style), which `node:fs` does - not expose. -- **It rests on the backend's own resolution logic.** Symlink resolution - is subtle, and platform behavior is not uniform: this project has found - cases where the same code confines correctly under one set of runtime - permission flags and not another (see the `PLATFORM TRAPS` note in - `wasi/src/filesystem_node.ts`). Every such quirk is a bug we can fix; - the class of such quirks is not one we can close by being careful. - -The web backend (OPFS) is different in kind: it has no host namespace to -escape and no symlinks, so containment there is structural rather than -checked. The browser's origin sandbox is the boundary. +Wasm memory isolation does not limit authority deliberately supplied +through host imports. Custom providers and injected transports are trusted +host code; their access and policy are the embedder's responsibility. ## What the implementation gives you -**Preopens are explicit grants.** There is no default and no ambient -filesystem. A guest sees exactly the directories the embedder named: +The default `wasi()` merge provides captured CLI I/O, clocks, entropy, and +empty filesystem preopens. Host authority is opt-in through +`filesystem-node`, `filesystem-web`, `sockets`, `http`, and `cli-stdio`. + +**Explicit filesystem grants, read-only by default.** There is no ambient +host filesystem. Preopens name the directories exposed to the guest: ```ts filesystemNode({ preopens: { "/": "./sandbox" } }) ``` -**Read-only by default.** Write access is opt-in for the whole -filesystem implementation: +Write access is a single option for the whole provider, not a per-preopen +permission: ```ts filesystemNode({ preopens: { "/": "./data" }, writable: true }) ``` -Without `writable`, every mutating operation is refused with the WIT -`read-only` error code — writes, creation, truncation, deletion, -rename, link and symlink creation, and timestamp changes. This is -deliberately a single package-level flag rather than a per-preopen -permission. Per-preopen permissions form a lattice, and the -two-descriptor operations (`link-at`, `rename-at`) are edges between -cells: each is a place where the check can attach to the wrong -descriptor, letting a guest bridge from a read-only preopen to a -writable one. wasmtime-wasi had a vulnerability of that shape. One flag -has no lattice to bridge, and the proof obligation becomes a closed -enumeration — every mutating operation refuses — which is checkable -against the WIT. - -**Outgoing HTTP requests can be scoped by name.** `http()` takes an -`allowRequest` policy, evaluated on the assembled request before anything -is dispatched: +Without `writable`, mutations are refused with the WIT `read-only` error: +writes, creation, truncation, deletion, rename, link and symlink creation, +and timestamp changes. A single flag avoids mixed read/write grants on +two-descriptor operations such as `link-at` and `rename-at`. It does not +remove the confinement limits below. + +**Name-level HTTP policy.** `http()` accepts `allowRequest`, evaluated on +the assembled request before dispatch: ```ts http({ allowRequest: ({ url, method }) => - url.protocol === "https:" && url.hostname.endsWith(".example.com") && + url.protocol === "https:" && url.hostname === "api.example.com" && method === "GET", }) ``` -`true` (the default) is unscoped egress; `false` denies everything while -leaving the types and resources usable; a callback may be async. A -callback that throws denies — a predicate that fails must not fail open. -Refusals reach the guest as the WIT `HTTP-request-denied` error code, and -a refused request never drains the guest's body stream. Because the -implementation does not follow redirects, each hop is a fresh request and -is checked again — unless an embedder supplies its own transport via -`http({ fetch })` and re-enables redirect following, which is trusted -code and can undo this. - -This is a name-level check only, which is the whole of what a -`fetch`-based host can express. See "What it does not give you" below for -what that leaves open. - -**Capabilities are à la carte.** The `wasi()` batteries merge carries -only ambient, side-effect-benign capabilities. Everything that grants -real host authority is a separate, deliberate import: `filesystem-node`, -`filesystem-web`, `sockets`, `http`, `cli-stdio`. +The default `true` allows unscoped egress; `false` denies requests while +leaving the types and resources usable. A predicate may be asynchronous. +Returning false or throwing denies with the WIT `HTTP-request-denied` +error; a denied request does not drain the guest's body stream. + +The default transport uses `redirect: "manual"`. It does not silently +follow a redirect past the check: a follow-up request must re-enter the +provider and be checked again. An injected `http({ fetch })` transport is +trusted and can undo this property by following redirects itself. + +## Why path confinement can never be a boundary + +This provider's path checks are not sufficient as a hostile-guest +boundary. The provider resolves paths relative to preopens, refuses +absolute paths and NUL, and prevents `..` from climbing past a preopen. +The node backend also resolves physical paths and checks that they remain +under the preopen's real path before making OS calls. + +These checks leave several limits: + +- **Hardlinks:** a name inside a preopen can refer to an inode also named + outside it. Path containment does not establish data provenance. +- **Bind mounts and mount points:** an in-tree path can expose another + filesystem or host subtree. +- **Cross-process races:** another process can replace a path component + between resolution and the OS operation. Avoiding that check/use race + requires descriptor-relative operations such as `openat` with suitable + resolution constraints; `node:fs` does not expose the needed interface. +- **Backend/platform behavior:** symlink and path handling differ across + runtimes and permission configurations. The backend contains explicit + workarounds in [filesystem_node.ts](../wasi/src/filesystem_node.ts), but + those checks are not an independent isolation layer. + +The OPFS backend has no host path namespace or symlinks. Its boundary is +the browser's origin sandbox, not a path check. That does not isolate a +guest from other data the embedder exposes within the same origin. ## What it does not give you -- **No network scoping** for `sockets()`, which grants the host's - network reach wholesale: no allowlist, no address check, no TCP/UDP - toggle. A guest with `sockets()` can talk to anything this process can - reach, including loopback services and cloud instance metadata - endpoints. Tracked by +- **No socket policy.** `sockets()` grants the process's network reach, + including TCP/UDP client and listener operations. There is no address + allowlist or protocol toggle. Loopback, link-local services, and cloud + metadata endpoints may be reachable. Tracked in [#200](https://github.com/polymorph-components/polyengine/issues/200). -- **No address-level check for `http()` either.** `allowRequest` sees the - URL a guest asked for, not the address it resolves to, so it cannot - refuse a name that resolves to loopback, a link-local address or a - cloud metadata endpoint — and the resolution can change between the - check and the connection. Closing that requires resolving and - connecting ourselves rather than delegating to `fetch`, which is not - possible in a browser at all: name resolution and connection happen - inside the network stack, and JS never observes an address. -- **No stdio scoping.** `cliStdio()` grants the host process's stdin, - stdout, stderr, environment and arguments. -- **No protection against a hostile guest reaching outside a preopen** - by the means listed above. +- **No HTTP address policy.** `allowRequest` sees the URL, not the resolved + address. An allowed name can resolve to a private, loopback, or metadata + address, and resolution can change before connection. A fetch-based + provider cannot pin the connection to an address it checked; browsers + do not expose that control to JS. Name allowlisting alone is not an + address-level egress boundary. +- **No default process-data isolation with `cliStdio()`.** It exposes the + process's stdin/stdout/stderr, environment, arguments, and working + directory by default. Sources and sinks can be overridden. Exit throws + by default, but `exitProcess: true` permits process termination. +- **No hostile-path isolation from preopen checks alone.** Read-only + prevents provider-mediated mutation; it does not prevent disclosure + through an unsafe preopen or the path mechanisms above. +- **No guest preemption.** Cooperative task scheduling does not interrupt + a guest that keeps executing. Do not treat Promise timeouts as CPU or + memory quotas; isolate and limit the execution environment as needed. ## Imposing a real boundary -These are the mechanisms that actually hold, in rough order of how much -they cost you. All of them are the embedder's or the operator's job: -nothing in this package can apply them on your behalf, and a library that -refused to start without them would only push the decision into your -source code, where it would travel to production disguised as a -deliberate choice. +Choose and test deployment controls for the threat model. The WASI +package does not configure them. -**Scoped runtime permissions.** The cheapest real boundary, and it lives -in the deployment rather than the code: +**Scoped runtime permissions** reduce process authority. For example: ```sh -# Deno -deno run --allow-read=/srv/sandbox --allow-write=/srv/sandbox app.ts +# Deno: adjust readable paths for the application and its assets too. +deno run --allow-read=/srv/app,/srv/sandbox --allow-write=/srv/sandbox /srv/app/app.ts -# Node -node --permission --allow-fs-read=/srv/sandbox --allow-fs-write=/srv/sandbox app.js +# Node: filesystem permissions, not a network sandbox. +node --permission --allow-fs-read=/srv/app --allow-fs-read=/srv/sandbox --allow-fs-write=/srv/sandbox /srv/app/app.js ``` -Both restrict the whole process, so the host application is bound by the -same limits as the guest. Note that Node's permission model has no -stable network dimension, so it constrains the filesystem grants above -but not `sockets()` or `http()`. - -**Landlock** (Linux 5.13+) restricts a process to a set of paths with -specified access rights, enforced by the kernel against resolved inodes -— so it is not fooled by symlinks, hardlinks or bind mounts. It is -unprivileged and irreversible once applied, which makes it a good fit -for a server process that takes its grants at startup. - -**Containers, seccomp, VMs.** The conventional answer when the guest is -genuinely untrusted, and the only one that also contains the host -application. - -**Give the guest no host namespace at all.** If the component does not -need live host files, back its filesystem with something you fully own — -an in-memory tree or an image file. Containment stops being a property -you check and becomes one that cannot be expressed otherwise. This is -what the OPFS backend gets for free. +These apply to the host application as well as guest-triggered operations; +they do not distinguish two callers within one process. Keep unrelated +permissions disabled and review each runtime's limitations. In particular, +[Node's permission model](https://nodejs.org/api/permissions.html) explicitly +does not guarantee protection against malicious code. Network controls +depend on the runtime version; filesystem grants are not a destination +allowlist. Follow the deployed version's permission documentation rather +than assuming the filesystem example establishes network isolation. + +**Kernel and process isolation.** Landlock can irreversibly reduce Linux +filesystem authority without privilege. Its protection depends on the +available ABI, handled rights, filesystem topology, inherited descriptors, +and which threads are restricted. It is not a blanket guarantee that any +directory allowlist rejects every hardlink or mount exposure. Follow the +[kernel documentation](https://docs.kernel.org/userspace-api/landlock.html) +and apply the policy before exposing authority to untrusted execution. + +Containers or VMs, with appropriately configured filesystem and network +access, can isolate the whole host process. Seccomp can further restrict +system calls but is not by itself a filesystem-path or network-destination +policy. Include resource limits and inherited handles in the deployment +review; merely selecting an isolation technology does not configure it. + +**No host namespace.** If the guest does not need live host files, use an +isolated in-memory filesystem or controlled image instead. OPFS similarly +avoids the host namespace, under the browser's origin boundary. ## The artifact cache is a trust input -The artifact cache (`@polyengine/runtime`'s `runtime/src/cache/`, see -[architecture.md §10](architecture.md)) stores a translated plan and its -adapter modules so a reload can skip the translator. It is host-side: it -never passes through the WASI filesystem, and no guest can reach it -through any interface this package exposes. But it stores its entries in -the same host namespace a guest's preopens are carved out of, and **a -cache entry is worth roughly what the component binary is worth.** - -The plan drives module slicing, adapter selection, import wiring and -initializers. What the runtime verifies on a cache hit is real but -narrower than it looks: - -- the stored entry agrees with the requested key, and the plan's recorded - component hash matches it; -- the plan is structurally valid (`loadPlan`); -- the caller's component bytes hash to `plan.component.sha256` - (`verifyComponent`); -- with generated typed bindings, the world digest matches the one the - bindings were built from. - -What nothing verifies is that the plan is *the plan the translator would -have produced for those bytes*. The world digest is computed from the -plan itself, so it catches skew and coarse substitution, not tampering. -Write access to the cache root therefore buys about what write access to -your component files buys. - -**Keep the cache root outside every preopen tree.** Not a child, not a -sibling you also preopen, not a parent. And note that scoped runtime -permissions cannot enforce this for you: one process needs write access -to the cache root, so at that layer the cache and a writable preopen are -indistinguishable. What separates them is the WASI provider's own path -handling — which the top of this page says is not a boundary. The -read-only default is what makes the common case safe: a guest that cannot -write anywhere cannot poison a cache. - -**The stronger recipe: a pre-warmed, read-only cache.** Translate at -build time, ship the cache directory as an artifact, and run with no -write access to it: +The [artifact cache](../runtime/src/cache/) stores translated plans and +adapter modules to skip translation on reload +([architecture §10](architecture.md#10-caching)). It is host-side, not a +WASI interface. A filesystem cache can nevertheless share a host namespace +with guest preopens. **Treat cache contents and shipped translation +envelopes as trusted executable inputs.** + +The cache and instantiation pipeline check different properties: + +- Bundled cache backends check stored metadata against the requested key + and recorded component hash. +- `loadPlan` checks the plan's structure and format version. +- `verifyComponent` checks the caller's component bytes against the plan's + recorded length and SHA-256. +- Generated facades check the expected world digest against the plan's + type information. + +None proves that the supplied plan and adapters are what the translator +would produce from those bytes. The digest is derived from the plan, so +it detects binding skew, not authenticity. A custom `ArtifactCache` is +also trusted to honor its interface; `translateCached` does not independently +authenticate its results. Cache write access can substitute executable +artifacts without changing the original component file. + +**Keep the cache separate from all preopens.** Do not expose its root, an +ancestor, or an alias through the guest filesystem. Separation alone is +not a hostile-guest boundary: process permissions cannot distinguish a +legitimate host cache write from a guest-induced write by the same +process. The provider's read-only default prevents mutation through that +provider, not through other grants or a compromised host. + +**Prefer a pre-warmed, read-only cache for fixed deployments.** Translate +in a trusted build step and deny the production process write access to +the cache and application artifacts. **Verify actual hits after warming:** +the current `dirCache.put` does not create the nested directory required +by normal `adapters/.wasm` names. Those adapter-bearing writes fail, +but `translateCached` still returns a successful fresh translation. +Adapter-free writes are unaffected by this limitation, and correctly +laid-out entries can still be read. Check `fromCache: true` before relying +on prewarming; this documentation does not resolve the backend limitation. + +The intended permission split is: ```sh -# build step (writes the cache) +# Build step: writes trusted cache artifacts. deno run --allow-read --allow-write=/srv/cache warm.ts -# production (cannot write the cache at all) -deno run --allow-read=/srv/app,/srv/cache --allow-write=/srv/state app.ts +# Production: cache and application are readable, only state is writable. +deno run --allow-read=/srv/app,/srv/cache --allow-write=/srv/state /srv/app/app.ts ``` -Cache hits work normally without write access. A miss, a stale entry -after a layout-version bump, or an unreadable root degrades to a fresh -translation rather than an error — no cache failure fails a translation. -Pass `onCacheError` to `translateCached` if you want the degradation to -be visible in your logs. This is the only arrangement that keeps a -compromise of the running process from becoming a persistent one. - -**Use `dirCache` on servers, not `webCache`.** Deno's Cache API needs no -permission flag, so it cannot be scoped or denied, and — absent -`--location` — it stores in a user-global bucket shared by every Deno -program that user runs, any of which can overwrite entries under the same -cache name. It also does not exist on Node or Bun. `webCache` is the -browser backend, where it is the only option, there is no permission -model to leverage, and the platform partitions storage by origin. +Hits work without write access. A miss, stale layout, or unreadable root +falls back to fresh translation, which still requires a translator and +valid component bytes. Cache `get`/`put` failures do not fail an otherwise +valid translation. `onCacheError` reports failures caught by +`translateCached`; backend failures converted directly to misses may be +silent. Explicit `evict()` calls can still throw. + +A read-only cache prevents persistence through that cache; it does not +prevent persistence through other writable state or replace host isolation. +Build-time envelopes deployed without a translator are another option, +but must be protected as trusted artifacts rather than treated as a cache +that can repair itself. + +**On Deno servers, prefer `dirCache` to `webCache`.** Deno's Cache API has +no permission flag and, without `--location`, uses a user-global bucket +that other Deno programs under that user can modify. It is not constrained +by filesystem permission flags. `dirCache` allows an explicit filesystem +policy. It requires Deno; other server runtimes can supply their own +`ArtifactCache`. `webCache` is intended for browsers, where storage is +partitioned by origin, not by component or runtime instance. Same-origin +code is part of that trust boundary. ## Reporting -Security issues in this project should be reported through the -repository's issue tracker, or privately to the maintainers if the report -would itself be a working escape. +Report security issues through the +[repository tracker](https://github.com/polymorph-components/polyengine/issues), +or privately to the maintainers when public reproduction details would +enable an escape before a fix is available. diff --git a/examples/README.md b/examples/README.md index bd55c94..b3fcea6 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,197 +1,70 @@ -# examples/ +# Examples and guest fixtures -Two things live here, for two audiences: +## Embedder examples -## Embedder examples (start here if you're using polyengine) +Each example pairs WIT, a Rust guest, and a self-checking TypeScript host: -Complete, self-contained, self-checking WIT + Rust-guest + TS-host pairs. -Each directory can be copied out of the repo and built as-is; each `run.sh` -builds the guest component and runs the host under Deno (`just examples` -runs both, and CI does too — these cannot silently rot). +| Example | Focus | +| --- | --- | +| [`hello-world/`](hello-world/) | Translate, instantiate, and call one export with no imports | +| [`kitchen-sink/`](kitchen-sink/) | Values, fallible and suspending imports, resources, streams/futures, and explicit synchronous calls | -| example | what it teaches | -|---|---| -| [`hello-world/`](hello-world/) | the smallest complete embedding: translate → instantiate → call one export; no imports | -| [`kitchen-sink/`](kitchen-sink/) | a representative tour: imports (sync / fallible / **suspending**), resources both directions, **streams and futures** (natural producers in, handles out), and the non-obvious value spellings (enum, variant, flags, outermost vs nested option/result, the option-boxing rule) | +From the repository root, `just examples` builds the translator and runs both. +The scripts require Deno, Rust with `wasm32-unknown-unknown`, and `wasm-tools`. +Kitchen-sink also requires JSPI. Engine support is documented in +[architecture](../docs/architecture.md); the +[embedder contract](../contracts/embedder-api.md) defines the API. -The normative reference behind both is -[`contracts/embedder-api.md`](../contracts/embedder-api.md). +These examples use this repository's Deno workspace to resolve packages. When +copying one elsewhere, configure `@polyengine/runtime`, `@polyengine/translator`, +and, for kitchen-sink, `@polyengine/protocol` through your own import map or +package manager. The guest source is self-contained, but the directory alone +does not supply standalone package resolution. -## Rust guest fixture corpus (`guests/`) +## Rust guest fixture corpus -Guest components built with **wit-bindgen** (the compatibility target of this -project, docs/architecture.md §1/§11). The TS host runs these as its executable -wit-bindgen-compat claim. Each guest is a pure computational reactor — **no -WASI imports** — so componentization needs no wasip1 adapter. +[`build.sh`](build.sh) builds the fixture guests used by runtime, WASI, and +ct-runner tests. The guest WIT and Rust source are the inventory of their exact +interfaces; the following groups describe their test purpose rather than an +exhaustive exported-function list. -## Corpus +| Guests under `guests/` | Coverage | +| --- | --- | +| `hello`, `values`, `resources` | Value roundtrips, own/borrow handles, observable resource destruction | +| `async-probe`, `context-user`, `cancel-import` | Callback-ABI async calls, interleaving, context slots, host-import cancellation | +| `stream-echo`, `stream-pass`, `future-user`, `future-import`, `resource-stream` | Stream/future input and output, identity transfer, sync imports returning futures, resource elements, early reader drop | +| `tcp-echo`, `http-fetch` | WASI p3 socket/HTTP workloads, including detached serving tasks and streamed bodies | +| `fs-probe`, `net-probe` | WASI p2 filesystem and sockets through Rust `std::fs` / `std::net` | +| `test-suite` | ct-runner pass/fail/skip, diagnostics, and budget handling | -| Component | World (WIT) | Exports | Size (release) | -|---|---|---|---| -| `hello.component.wasm` | [`guests/hello/wit/world.wit`](guests/hello/wit/world.wit) | `greet: func(name: string) -> string` | ~20 KB | -| `values.component.wasm` | [`guests/values/wit/world.wit`](guests/values/wit/world.wit) | 17 `echo-*` funcs, one per type shape: bool, u64, s64, f32, f64, char, string, record, variant, enum, flags, option, option-nested, result, list\, list\, tuple | ~24 KB | -| `resources.component.wasm` | [`guests/resources/wit/world.wit`](guests/resources/wit/world.wit) | interface `counters`: `counter` resource (constructor, `increment`, `get`, static `merge`) + free funcs over own/borrow handles (`make-counter`, `sum-both`, `bump`, `consume`) + `live-counters` (observes destructor runs) | ~24 KB | -| `async-probe.component.wasm` | [`guests/async-probe/wit/world.wit`](guests/async-probe/wit/world.wit) | CM 0.3 async: `wait-then-double: async func` (yields once), `sum-stream: async func(stream)`, `future-add: async func(future, u32)` | ~57 KB | -| `context-user.component.wasm` | [`guests/context-user/wit/world.wit`](guests/context-user/wit/world.wit) | Context-local-storage (slot 0) via interleaved concurrent activations: `interleave: async func(count: u32) -> u32` (spawns `count` locally-concurrent tasks, each yielding a different number of times) | ~48 KB | -| `stream-echo.component.wasm` | [`guests/stream-echo/wit/world.wit`](guests/stream-echo/wit/world.wit) | `echo-doubled: async func(input: stream) -> stream` — consumes AND produces a stream in one export | ~60 KB | -| `future-user.component.wasm` | [`guests/future-user/wit/world.wit`](guests/future-user/wit/world.wit) | `double-future: async func(f: future) -> u32` (awaits an imported future); `make-future: async func(x: u32) -> future` (resolves an exported one) | ~64 KB | -| `future-import.component.wasm` | [`guests/future-import/wit/world.wit`](guests/future-import/wit/world.wit) | Host imports with future-bearing results (contracts/embedder-api.md §"Streams and futures"; the `wasi:sockets@0.3` TCP shapes reduced to `u32`): `next-value: func() -> future`, `send-sink: func(stream) -> future`, `recv-pair: func() -> tuple, future>`, driven by `run-next`/`run-send`/`run-recv` exports (`run-send` writes the stream only after the sync import returns — the livelock probe) | ~64 KB | -| `resource-stream.component.wasm` | [`guests/resource-stream/wit/world.wit`](guests/resource-stream/wit/world.wit) | Streams of OWNED HOST RESOURCES (contracts/embedder-api.md §"Streams and futures"; the `wasi:sockets@0.3` TCP `listen` shape): world-level `resource ticket { value: func() -> u32 }`, `tickets: func(count: u32) -> stream`, driven by `sum-tickets` (drains; per-element dtors) and `take-then-drop` (abandons the reader mid-stream — the un-taken-element release probe) | ~50 KB | -| `tcp-echo.component.wasm` | [`guests/tcp-echo/wit/world.wit`](guests/tcp-echo/wit/world.wit) | The REAL `wasi:sockets@0.3.0` TCP surface (wit/deps vendored verbatim from upstream): `echo-client` dials/streams/FINs/drains (the wosh client shape); `start-echo-server` binds/listens and serves connections from a detached task, returning `tuple>` — the wasi integration gate (`tests/integration_sockets_test.ts`) drives both over live loopback sockets | ~108 KB | -| `http-fetch.component.wasm` | [`guests/http-fetch/wit/world.wit`](guests/http-fetch/wit/world.wit) | The REAL `wasi:http@0.3.0-rc` outbound surface (wit/deps/wasi-http types.wit verbatim, worlds.wit reduced): `get`/`post-echo` exports construct requests (`fields`, trailers futures), drive `client.send`, and drain streamed response bodies — the wasi package's fetch-backed integration gate (`tests/integration_http_test.ts`) | ~120 KB | -| `test-suite.component.wasm` | [`guests/test-suite/wit/tests.wit`](guests/test-suite/wit/tests.wit) (vendored verbatim from polymorph-test's `polymorph:test@0.1.0`) | Implements the `suite` world: imports `test-context`, exports `tests` (`all: async func() -> list`). Six deterministic cases exercising pass/fail/skip, multi-message diagnostics, and a measurable-time case for budget plumbing — the ct-runner's (`../../ct-runner/`) fixture. | ~61 KB | -| `fs-probe.component.wasm` | [`guests/fs-probe/wit/world.wit`](guests/fs-probe/wit/world.wit) | The REAL `wasi:filesystem@0.2` surface via std::fs — built for **wasm32-wasip2** (wasi-libc + preview1 adapter emit a finished component; `build.sh` skips `component new` for it), so `run: func() -> result` drives create/write/read/append/seek/list/rename/delete plus the NotFound error path through the exact linkage of a ported CLI program — the wasi package's filesystem integration gate (`tests/integration_fs_test.ts`) | ~68 KB | -| `net-probe.component.wasm` | [`guests/net-probe/wit/world.wit`](guests/net-probe/wit/world.wit) | The REAL `wasi:sockets@0.2` surface via std::net — **wasm32-wasip2** like fs-probe: `run` drives a TCP listener + client self-echo over loopback and a UDP pair (connected mode included) entirely inside the guest, through wasi-libc's poll-shaped driving (two-phase start/finish ops looping on would-block, `pollable.block`, wasi:io socket streams) — the wasi package's 0.2-sockets integration gate (`tests/integration_net_test.ts`) | ~168 KB | +Not every guest is a WASI-free reactor. Most build a core module for +`wasm32-unknown-unknown`, then use `wasm-tools component new`; some explicitly +import WASI interfaces. `fs-probe` and `net-probe` instead build for +`wasm32-wasip2`, which emits a finished component without that conversion step. - -Every `echo-*` function returns its input unchanged: the host asserts -roundtrip equality for arbitrary vectors (lift/lower tests). The `resources` -guest counts live instances so destructor invocation is observable from -outside (`live-counters`). - -## Rebuild +From the repository root: ```sh -./build.sh +just fixtures ``` -Requires: Rust with the `wasm32-unknown-unknown` target, `wasm-tools` on -PATH. `wasmtime` optional (smoke run). Outputs go to `guests/build/` -(gitignored). Per guest the script runs: - -1. `cargo build --release --target wasm32-unknown-unknown` -2. `wasm-tools component new .wasm -o .component.wasm` -3. `wasm-tools validate --features component-model[,cm-async]` -4. `wasm-tools component wit` (world round-trip sanity) - -Each guest crate has an **empty `[workspace]` table** in its `Cargo.toml` (the -repo root cargo workspace does not include `examples/`), pins -`wit-bindgen = "=0.60.0"`, has a committed `Cargo.lock`, and uses a small -release profile (`opt-level = "s"`, `lto = true`, `codegen-units = 1`, -`panic = "abort"`, `strip = "debuginfo"`). - -### Toolchain (validated against) - -| Tool | Version | -|---|---| -| wit-bindgen (crate, proc-macro) | **0.60.0** (pinned `=0.60.0`) | -| Rust | 1.96.0 stable, target `wasm32-unknown-unknown` | -| wasm-tools CLI | 1.247.0 | -| wasmtime CLI (smoke run only) | 47.0.1 | - -No extra tool installs needed: bindings come from the `wit_bindgen::generate!` -proc macro, not the wit-bindgen CLI. - -## Async findings - -**Status: CM 0.3 async guests build on stable Rust today; built and smoke-run -here.** Details: - -- wit-bindgen 0.60.0 generates Component Model - [async ABI](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md) - bindings on **stable** Rust (1.96). The crate's `async` cargo feature is a - **default feature**; `generate!({ async: true })` or `async func` in WIT - turns it on per export. No nightly, no CLI tools, no unstable rustc flags. -- WIT `async func`, `stream`, `future` all parse in the macro and - round-trip through `wasm-tools component new` + `component wit` (wasm-tools - 1.247). -- Validation needs `--features component-model,cm-async` (wasm-tools names it - `cm-async`; there are further `cm-async-stackful`/`cm-async-builtins` - refinements, not needed for these guests). Without it, validation fails on - `context.get` — proof the binary genuinely uses 0.3 async builtins. -- **Every async export is lifted with the stackless callback ABI**: - `canon lift ... async (callback ...)`; core exports come in pairs - `[async-lift]NAME` + `[callback][async-lift]NAME`. wit-bindgen's Rust - backend never emits stackful async lifts, which matches docs/architecture.md §6's build - order (task core + callback ABI first — JSPI paths are only needed for - blocking sync-lowers over async, not to run these guests' exports). -- Canonical builtins used by the generated runtime: `task.return`, - `task.cancel`, `waitable-set.{new,poll,drop}`, `waitable.join`, - `context.{get,set}` (slot 0), and the full `stream.*`/`future.*` suites - (new/read/write/cancel-read/cancel-write/drop-readable/drop-writable). - Notably **no `canon yield`**: `wit_bindgen::yield_async()` is implemented - via the callback return-code protocol, not the `yield` builtin. -- wasmtime 47.0.1 executes the async component with **default flags** - (`wasmtime run --invoke 'wait-then-double(21)'` → `42`, including a real - yield suspension + resume). Stream/future-typed exports can't be invoked - from the CLI (WAVE has no stream/future literals) — they await the host - harness. - - Guest-side helpers available for later corpus growth: `block_on`, - `spawn_local` (feature `async-spawn`, adds `futures`), `yield_async`, - `backpressure_inc/dec`, stream/future writer halves (`wit_stream::new()`, - `wit_future::new()` in generated bindings). - -## Async corpus expansion: demand-side inventory - -Guests were added to give the task-core/scheduler and streams phases -concrete, minimal fixtures per canonical built-in. Canonical imports per -guest (`wasm-tools print *.component.wasm | grep -oE '\[[a-z0-9_-]+\]' | -sort -u`). The common base set — `async-lift`, `callback`, -`context.{get,set}`(slot 0), `task.{cancel,return}`, -`waitable-set.{new,poll,drop}`, `waitable.join`, and **no `canon yield`** -(`wit_bindgen::yield_async()` is implemented via the callback return-code -protocol, not the `yield` builtin) — is shared by every guest below: - -| Guest | Canonical built-ins imported beyond the base set | -|---|---| -| `context-user` | None — `spawn_local`'s locally-concurrent tasks are still driven by the one export's callback-ABI event loop; no additional canonical built-ins are needed to interleave them. This means context-slot isolation across interleaved activations is entirely a **guest-side** (wit-bindgen runtime) concern from the host's point of view — the host only ever sees one `context.get`/`context.set` pair per callback invocation, exactly as for a single non-interleaved task. | -| `stream-echo` | `async-lower` and the full `stream.*` suite: `stream.new`, `stream.read`, `stream.write`, `stream.cancel-read`, `stream.cancel-write`, `stream.drop-readable`, `stream.drop-writable`. `async-lower` appears here (and in `future-user`) but not in the pure-yield guests — worth the streams phase confirming why (candidate explanation: the generated stream-forwarding task itself contains an async call shape lowered via `canon lower ... async`, from `spawn_local`'s internal task machinery, but this needs the streams-phase owner to confirm against `definitions.py`, not asserted here). | -| `future-user` | `async-lower` and the full `future.*` suite: `future.new`, `future.read`, `future.write`, `future.cancel-read`, `future.cancel-write`, `future.drop-readable`, `future.drop-writable`. | - -**Stream-producer viability (wit-bindgen 0.60.0, stable Rust 1.96):** -`wit_stream::new()` (the per-world generated wrapper around -`wit_bindgen::rt::async_support::stream_support::stream_new`) is usable -directly, no extra feature flags beyond `async-spawn` (needed only for this -corpus's background-forwarding pattern — a stream/future `write` is a -rendezvous that only completes once the far end reads it, so it cannot be -awaited before the reader half is returned to the caller; the fix is to -`spawn_local` the writing loop and return the reader immediately). `stream- -echo.component.wasm` builds, validates (`component-model,cm-async`), and -round-trips its world with no divergence from the future-only guests. This -answers the task brief's open question: **producing a stream from a -wit-bindgen 0.60 Rust guest needs no unstable feature, only the -already-established `async-spawn` pattern used for `future-user`.** - -All guests build with `cargo build --release --target -wasm32-unknown-unknown` on stable Rust 1.96, validate with `wasm-tools -validate --features component-model,cm-async` (wasm-tools 1.247), and -round-trip their worlds via `wasm-tools component wit`. `context-user` is -also smoke-run in `build.sh` via `wasmtime run --invoke`; `stream-echo`/ -`future-user` share the CLI limitation noted above (WAVE has no -stream/future literals) and await the host harness. - -## Notes for the host implementation - -Shape of wit-bindgen 0.60 core modules (inspect: `wasm-tools print`): - -- **String encoding is utf8** on every lift/lower - (`string-encoding=utf8`); wit-bindgen Rust never emits utf16/latin1. -- Sync exports: `canon lift (core func $f) (memory $m) (realloc $cabi_realloc) - string-encoding=utf8 (post-return $cabi_post_NAME)`. A `cabi_post_NAME` - post-return is emitted **per export that returns indirect data** (e.g. - `greet`, the string/list echoes); the host must call it after copying - results out. -- Core module exports: the lifted funcs, `memory`, `cabi_realloc`, plus a - versioned `cabi_realloc_wit_bindgen_0_60_0` alias; `__data_end`/ - `__heap_base` globals are exported too (ignorable). -- Sync guests (hello/values/resources) have **zero core imports**. The async - guest imports only canonical intrinsics under module names `$root` / - `[export]$root` (e.g. `[waitable-set-new]`, `[task-return]NAME`), which - `wasm-tools component new` wires to canon builtins — still no WASI, no - adapter. -- Resources: dtors are plain core funcs; dropping an own handle inside the - guest (e.g. `consume`, `merge`) runs the dtor synchronously. Use - `live-counters` to assert dtor runs from the host side. -- The wit-bindgen version and world are embedded in a custom section - (`component-type:wit-bindgen:0.60.0:...:encoded world`) of the core module; - `wasm-tools component new` consumes it (metadata produced with wasm-tools - 0.254 internals decodes fine with the 1.247 CLI). - -### wasmtime CLI invocation notes - -`wasmtime run --invoke '' ` works for all scalar/ -aggregate types (WAVE syntax: `some("x")`, `err("bad")`, `{read, exec}`, -`(9, "nine", 9.25)`, `label("hi")`). Limitations found: functions returning -**resource handles** trap the CLI's result printer (wasm-wave "unsupported -value type"), and stream/future arguments aren't constructible — both are -host-harness territory, not corpus defects. +This requires both Rust targets and `wasm-tools`. It writes components to +`examples/guests/build/` and shared Cargo output to `examples/guests/target/` +(both gitignored). Each component is validated with its required feature set +and its WIT is printed. If `wasmtime` is on PATH, the script also smoke-tests +selected scalar/aggregate exports. That CLI smoke does not exercise the full +resource, stream, or future host API; the integration tests do. + +Guest crates are separate Cargo workspaces with committed manifests and +lockfiles. Consult those for dependency pins rather than a copied tool-version +table. Bindings come from the `wit_bindgen::generate!` macro; no separate +wit-bindgen CLI is needed. + +## Async producer pattern + +The async Rust fixtures use wit-bindgen's callback ABI. A stream/future write +waits for the far end to read, so a guest returning a reader must not first +await its writer. The producer runs in a `spawn_local` task and the export +returns the reader immediately; see [`future-user`](guests/future-user/) and +[`stream-echo`](guests/stream-echo/). This fixture pattern is not a substitute +for the runtime's cancellation and ownership rules in the embedder contract. diff --git a/examples/build.sh b/examples/build.sh index 904c9cd..9081472 100755 --- a/examples/build.sh +++ b/examples/build.sh @@ -5,7 +5,7 @@ # 1. cargo build --release --target wasm32-unknown-unknown # (pure computational reactors: no WASI imports, so no wasip1 adapter) # 2. wasm-tools component new core module -> component -# 3. wasm-tools validate (component-model; + cm-async for async-probe) +# 3. wasm-tools validate (component-model; + cm-async for async guests) # 4. wasm-tools component wit print the component's world (round-trip sanity) # # Outputs land in guests/build/ (gitignored). All guest crates share @@ -23,7 +23,7 @@ export CARGO_TARGET_DIR="$PWD/guests/target" GUESTS="hello values resources async-probe context-user stream-echo stream-pass future-user future-import resource-stream tcp-echo http-fetch test-suite fs-probe net-probe cancel-import" # Most guests are pure computational reactors on wasm32-unknown-unknown; -# fs-probe and net-probe build for wasm32-wasip2 ON PURPOSE — std::fs / +# fs-probe and net-probe build for wasm32-wasip2: std::fs / # std::net through wasi-libc is the linkage under test, and the wasip2 # target emits a finished component (no `component new` step). target_for() { @@ -76,8 +76,7 @@ if command -v wasmtime >/dev/null 2>&1; then check 'label("hi")' values.component.wasm 'echo-variant(label("hi"))' check '{read, exec}' values.component.wasm 'echo-flags({read, exec})' check '0' resources.component.wasm 'live-counters()' - # Component Model 0.3 async export (callback ABI): runs on wasmtime 47 - # with default flags; exercises yield suspension + task.return. + # Callback-ABI async exports exercise yield suspension and task.return. check '42' async-probe.component.wasm 'wait-then-double(21)' check '6' context-user.component.wasm 'interleave(4)' echo "smoke run OK" diff --git a/examples/hello-world/README.md b/examples/hello-world/README.md index b689e97..c65354d 100644 --- a/examples/hello-world/README.md +++ b/examples/hello-world/README.md @@ -1,30 +1,31 @@ -# hello-world — the smallest complete embedding +# Hello World -One exported function, no imports. Three files matter: +One exported function, no imports: -| file | role | -|---|---| -| [`wit/world.wit`](wit/world.wit) | the contract: `greet: func(name: string) -> string` | -| [`guest/src/lib.rs`](guest/src/lib.rs) | the Rust guest (wit-bindgen) implementing it | -| [`host.ts`](host.ts) | the host: translate → instantiate → call | +| File | Role | +| --- | --- | +| [`wit/world.wit`](wit/world.wit) | `greet: func(name: string) -> string` | +| [`guest/src/lib.rs`](guest/src/lib.rs) | Rust implementation using wit-bindgen | +| [`host.ts`](host.ts) | Load the packaged translator, instantiate, call, and check the greeting | -Run it: +From the repository root: ```sh -just shim # once, from the repo root: builds the translator -./run.sh # builds the guest component, runs the host +just shim +./examples/hello-world/run.sh ``` -What to notice: +Requires Deno, Rust with `wasm32-unknown-unknown`, and `wasm-tools`. The script +builds the guest, componentizes and validates it, type-checks the host, and runs +it with read access to the example's `build/` directory. Package resolution +comes from the repository workspace; see [examples](../README.md) before +copying the directory out. -- **Exports are Promise-shaped** — `await component.exports.greet(...)` - even though this guest is synchronous. One calling convention for sync - and async guests (contracts/embedder-api.md §"Functions and async"). -- **Strings just work** — the guest returns a heap-allocated string; the - canonical ABI's realloc dance is the runtime's problem, not yours. -- **The imports record is empty** — this world imports nothing. For the - full imports story (interfaces, resources, error model, suspending - imports) continue to [`../kitchen-sink`](../kitchen-sink). +`instantiate({ componentBytes, translator }, {})` supplies the empty import +record. `await component.exports.greet(...)` uses the default asynchronous host +calling convention even though the guest function is synchronous. The runtime +copies the returned string out of guest memory and handles canonical post-return +cleanup. -The authoritative reference for everything the host sees is -[`contracts/embedder-api.md`](../../contracts/embedder-api.md). +Continue with [kitchen-sink](../kitchen-sink/) for imports and other value shapes. +The [embedder contract](../../contracts/embedder-api.md) is the API reference. diff --git a/examples/hello-world/host.ts b/examples/hello-world/host.ts index 3c0e0d7..00a15c6 100644 --- a/examples/hello-world/host.ts +++ b/examples/hello-world/host.ts @@ -10,7 +10,7 @@ // // Inside this repository `@polyengine/runtime` and `@polyengine/translator` // resolve through the Deno workspace; a published consumer uses the same -// specifiers via JSR/npm (polyengine#16 tracks packaging). +// specifiers via JSR/npm. import { instantiate } from "@polyengine/runtime/embedder"; import { defaultTranslator } from "@polyengine/translator"; diff --git a/examples/hello-world/run.sh b/examples/hello-world/run.sh index 77e50d9..09e5e97 100755 --- a/examples/hello-world/run.sh +++ b/examples/hello-world/run.sh @@ -3,8 +3,7 @@ # examples/build.sh, scoped to this directory: cargo core module -> # `wasm-tools component new` -> validate -> run host.ts under Deno. # -# Prerequisites: the translator shim (`just shim` from the repo root; a -# published consumer gets a prebuilt Translator instead — polyengine#16). +# Prerequisite: `just shim` from the repo root. Published packages include the shim. set -euo pipefail cd "$(dirname "$0")" diff --git a/examples/kitchen-sink/README.md b/examples/kitchen-sink/README.md index 0898599..6f9295b 100644 --- a/examples/kitchen-sink/README.md +++ b/examples/kitchen-sink/README.md @@ -1,74 +1,57 @@ -# kitchen-sink — a representative tour of the embedder API +# Kitchen Sink -One world exercising the surfaces an embedder actually touches: +A WIT world, Rust guest, and self-checking host covering the main embedder +conventions. [`host.ts`](host.ts) numbers the demonstrations: -| surface | WIT | guest | host | -|---|---|---|---| -| enum / record / variant / flags | `types` interface | `describe`, `classify`, `scale`, `allowed` | §4 | -| outermost `option` → `undefined \| T` | `find` | | §5 | -| return-place `result` → resolve / throw `ComponentException` | `lookup` | | §5 | -| nested option/result as plain data + the boxing rule | `survey`, `maybe-maybe` | | §5 | -| host-implemented imports: sync, fallible, **suspending** | `notify` interface | `run-batch` | §2 | -| host-implemented resource (ctor / method / static / dispose) | `notify.channel` | `run-batch` | §3 | -| guest-implemented resource (`using`) | `api.counter` | `Counter` | §6 | -| streams: producers in, `Stream` handle out | `tally`, `countdown` | §8 | §8 | -| futures: Promise in, EAGER `Future` handle out | `promised-double`, `deferred-answer` | §9 | §9 | -| `sync()`: the synchronous view of a WIT-sync export | `allowed` | | §10 | +| Surface | WIT or guest entry | Host section | +| --- | --- | --- | +| Enum, record, variant, flags | `describe`, `classify`, `scale`, `allowed` | 4 | +| Options, fallible returns, nested results, option boxing | `find`, `lookup`, `survey`, `maybe-maybe` | 5 | +| Sync, fallible, and suspending imports | `notify`; driven by `run-batch` | 2, 7 | +| Host resource class and disposal | `notify.channel` | 3 | +| Guest resource class with `using` | `api.counter` | 6 | +| Stream producers and chunked reads | `tally`, `countdown` | 8 | +| Promise input and eager future handle output | `promised-double`, `deferred-answer` | 9 | +| Explicit synchronous export view | `allowed` via `sync()` | 10 | -Run it: +From the repository root: ```sh -just shim # once, from the repo root: builds the translator -./run.sh # builds the guest component, runs the host +just shim +./examples/kitchen-sink/run.sh ``` -What to notice: - -- **The guest cannot tell which imports suspend.** `read-sensor` and - `channel.send` are sync WIT functions; the host implements them with - Promises and marks them — `suspending(fn)` (call form) and - `@suspending` (decorator on the class method). The guest's Rust is - oblivious; its wasm frame parks on JSPI and resumes. Marking has costs - (a continuation hop per call, illegal from `start` functions) — see - the §2c comment in [`host.ts`](host.ts). -- **Match errors on the brand, never the message.** `lookup`'s err side - arrives as a thrown `ComponentException` with `.payload`; any *unbranded* throw - from a host import is a host bug and traps the component. -- **The option rule is per-chain.** An option inside a `list` is still - the outermost of its own chain (`undefined | T`); boxing to - `{ kind: "some" | "none" }` happens only for option directly inside - another option — `maybe-maybe` pins all three depths. -- **Resources are classes on both sides.** The host's `Channel` class is - handed over as-is (the runtime calls `[Symbol.dispose]` when the guest - drops its handle); the guest's `counter` comes back as a constructible - class the host can `using`-scope. -- **Streams lower from natural producers and lift as handles.** Pass an - array / ReadableStream / AsyncIterable where the guest expects a - `stream`; a guest-produced stream arrives as a `Stream` handle - whose `for await` yields *chunks* (`number[]` batches; `Uint8Array` for - `u8`). Guest-side, every stream/future write is a rendezvous — the - producer halves run in `spawn_local` tasks (see the guest doc comments). -- **A future-typed result is the one exception to Promise-shaped - exports**: `deferredAnswer()` returns an eager `Future` handle - synchronously (a Promise wrapper would adopt the thenable handle and - make `drop`/`cancel` unreachable). Awaiting the handle yields the value. -- **`sync()` reclaims the synchronous form of a WIT-sync export**, for a - handler that must decide before it returns (cancelable-event dispatch, - DOM's `preventDefault()` — even an already-resolved Promise only lets its - continuation run on a later microtask, too late once the handler has - returned). `sync(api.allowed)` works here for real: this instantiation - runs in JSPI mode (`read-sensor` is `suspending()`), so the call exercises - the SYNC_ENTRY re-entry path — it succeeds because `allowed` never - reaches the suspending import and so completes without parking. A guest - call that DOES try to park fails loudly instead (`NeedsJspi` / - `SyncEntryBusy` / trap) — `sync()` is for exports known to complete - synchronously, not a way to force one that doesn't. - -Deliberately absent (to stay approachable): async-typed *imports* and -`error-context` — see `contracts/embedder-api.md` until an example covers -them. - -The authoritative reference is -[`contracts/embedder-api.md`](../../contracts/embedder-api.md); if this -example and the contract disagree, the contract wins (and the example's -self-checks should have caught it — run them). +Requires Deno with JSPI, Rust with `wasm32-unknown-unknown`, and `wasm-tools`. +The script builds and validates the guest, type-checks the host, and runs its +assertions. Package resolution comes from the repository workspace; see +[examples](../README.md) for use outside this checkout. + +## Details worth following + +- `read-sensor` and `channel.send` are WIT-sync functions implemented with + Promises. `suspending(fn)` and the `@suspending` decorator come from + `@polyengine/protocol`; their marks select JSPI for this instantiation. + Suspending imports cannot be reached during core start-function execution. +- A return-position WIT `result` maps to a returned success value or a branded + `ComponentException` carrying `.payload`. Unbranded host exceptions trap; do + not classify errors by message text. +- An option nested directly inside another option needs boxing. An option + inside a list starts its own chain and still uses `undefined | T`. + `maybe-maybe` demonstrates the distinction. +- Host resources are supplied as classes, with `[Symbol.dispose]` called when + the guest drops the owned handle. Guest resources arrive as classes whose + instances can be scoped with `using`. +- Stream inputs accept natural producers. Lifted `Stream` iteration yields + chunks, not individual elements (`Uint8Array` for `u8`). Guest producers run + in background tasks so they can return a reader before waiting on writes. +- A direct `future` result returns its handle eagerly, not wrapped in a + Promise: Promise resolution would adopt the thenable and hide its lifecycle + methods. Await the handle to obtain the value. +- `sync(api.allowed)` gives a synchronous view of a WIT-sync export that does + not park. The example uses it for a cancelable-event-style handler that + cannot await. It does not force a suspending call to complete synchronously; + see the contract for admission and failure conditions. + +Async-typed imports and error-contexts are not demonstrated here. Consult the +[embedder contract](../../contracts/embedder-api.md) for those surfaces and the +complete ownership, cancellation, and error rules. diff --git a/examples/kitchen-sink/host.ts b/examples/kitchen-sink/host.ts index a1c2712..7d742a6 100644 --- a/examples/kitchen-sink/host.ts +++ b/examples/kitchen-sink/host.ts @@ -91,7 +91,7 @@ const imports = { // §2b — fallible import (result return-place): return the ok value; // throw `new ComponentException(payload)` for the err side. Any OTHER throw is a - // host bug and traps the component — the anti-footgun inversion. + // host bug and traps the component. parseId: (raw: string): number => { const n = Number(raw); if (!Number.isInteger(n) || n < 0) { diff --git a/examples/kitchen-sink/run.sh b/examples/kitchen-sink/run.sh index 3a7d3db..ad95a5e 100755 --- a/examples/kitchen-sink/run.sh +++ b/examples/kitchen-sink/run.sh @@ -3,8 +3,7 @@ # examples/build.sh, scoped to this directory: cargo core module -> # `wasm-tools component new` -> validate -> run host.ts under Deno. # -# Prerequisites: the translator shim (`just shim` from the repo root; a -# published consumer gets a prebuilt Translator instead — polyengine#16). +# Prerequisite: `just shim` from the repo root. Published packages include the shim. set -euo pipefail cd "$(dirname "$0")" diff --git a/harness/README.md b/harness/README.md index 93881d4..98efd10 100644 --- a/harness/README.md +++ b/harness/README.md @@ -1,201 +1,123 @@ # Conformance harness -The wast→JSON conformance pipeline from docs/architecture.md §11: an offline Rust step -(`crates/testgen`) converts the official Component Model `.wast` suite into -JSON command files plus extracted binaries, and a TypeScript runner (this -directory, Deno) executes the JSON. The component runtime does not exist yet; -every command that needs it is recorded as a skip with reason -`pending-runtime`, so the same harness runs green today and becomes the -conformance gate as the runtime lands. +`crates/testgen` converts the pinned Component Model WAST suite into JSON commands +and extracted binaries. This directory executes those commands under Deno; +browser and engine-shell runners reuse the harness with lane-specific +expectations. See [architecture](../docs/architecture.md) for the parity policy +and platform scope. + +A green run means no unexpected failures or passing stale xfails in the +executed corpus. It is **not proof of full conformance**: known failures, skipped +directives, capability gaps, and scheduling-profile exclusions remain visible +and must be included when reporting results. ## Running +From the repository root: + ```sh -deno task conformance # regenerate harness/generated/ via testgen, then test -deno task gen # just (re)convert the wast suite -deno task test # just run the tests against harness/generated/ +just shim # rebuild the translator for the current source +just conformance # regenerate the corpus and run the Deno harness ``` -`deno task test` prints a per-directory summary at the end: -`{commands, executed, passed, failed, pending-runtime, unsupported-directive}`. - -testgen can also be run directly (from anywhere in the repo): +From `harness/`: ```sh -cargo run -p testgen # whole suite -> harness/generated/ -cargo run -p testgen -- binary validation # subset of test subdirectories -cargo run -p testgen -- --test-dir D --out-dir D2 +deno task gen # regenerate harness/generated/ with testgen +deno task test # run against existing corpus and shim +deno task conformance # gen + shim-check + test ``` -Output is deterministic: same suite + same testgen build → byte-identical -`harness/generated/` (sorted traversal, stable JSON field order, no -timestamps or absolute paths). +`shim-check` builds only when the artifact is absent; it does not detect a stale +shim. `CONFORMANCE_EXECUTOR=core-only deno task test` selects the JS-WebAssembly +pipeline-sanity stub, not a component conformance run. It needs no shim. -## Generated layout +Testgen also accepts directory subsets and explicit locations, from the repo root: +```sh +cargo run -p testgen -- binary validation +cargo run -p testgen -- --test-dir D --out-dir D2 ``` + +## Results and exclusions + +The per-directory summary reports `commands`, `executed`, `passed`, `failed`, +`xfail`, `pending-runtime`, `pending-capability`, and `unsupported-directive`. +`executed` includes passes and both known and unexpected failures; skips are +separate. + +- `src/xfail.ts` records known failures by generated file and WAST source line, + with a reason and tracking issue. A matching failure is an xfail, not a pass. + The Deno summary fails if an xfailed command passes. This check does not prove + that every listed entry was reached, or that a failure still has its original + cause; review the reason when triaging a changed result. +- `PendingRuntimeError` becomes a skip. A `pending-capability:` prefix selects + the more specific counter. `RuntimeExecutor` retains message-based capability + classification in `CAPABILITY_MARKERS`; these skips are not validation + verdicts or evidence that all async behavior is unsupported. +- Text/quote artifacts and unimplemented directives are recorded as + `unsupported-directive` rather than passed. +- Under `POLYENGINE_SCHED_SEED`, files in `DETERMINISTIC_PROFILE_ONLY` are ignored + by Deno because their guests assert reference-profile scheduling order. They + are absent from the command summary, so report the ignored-test count too. +- Browser and shell deltas live in `browser/expectations/` and + `shell/expectations/`. Their lane runners check for unexpected deviations and + stale deltas; a lane with accepted deltas is not an all-pass corpus. + +All generated suite directories, including `async/` and `values/`, are run. +Component-level value imports/exports remain outside the runtime's stated +parity scope; do not confuse that feature with ordinary canonical-ABI values. + +## Execution + +[`CommandExecutor`](src/executor.ts) separates command bookkeeping from engine +semantics. The runner owns instance names and the current default instance, and +resets executor state after each file. The Deno test gives each file a timeout; +a stalled file is recorded as failed rather than disappearing from the summary. + +[`RuntimeExecutor`](src/runtime-executor.ts) translates components, loads the +current plan format, and calls the internal plan executor. It exercises the raw +canonical value boundary, not the public embedder facade. Async exports are +awaited; WAST calls use `trapOnIdle: true` so an unresolved blocking call that +goes idle is a deadlock verdict rather than a permanently pending Promise. + +Only a structured `TranslateError` with phase `validation` counts as component +rejection for `assert_invalid` or `assert_malformed`. Translation does not +distinguish those two phases; `unsupported` and `internal` errors are not valid +rejection evidence. Instantiation errors are mapped separately according to the +command's expected trap or link failure. + +The plain core-module path delegates to `CoreOnlyExecutor`, which validates and +compiles but does not instantiate the module. The separate core definition/ +instance path can instantiate a compiled module with empty imports. General +core invocation and component `get` are not implemented. `register` records +instances but does not wire them into component host imports. These are harness +limits, not claims about the embedder API. + +## Generated data and comparison + +```text harness/generated/ - manifest.json # {"files": ["async/cancel-stream.json", ...]} - /.json # command file, one per .wast - /..wasm # extracted binary (module or component) - /..wat # quote forms kept as text + manifest.json # list of generated command files + /.json # one command file per WAST file + /..wasm # extracted binary + /..wat # quoted text artifact ``` -## JSON schema - -testgen drives the `wast` parser and the `json-from-wast` crate (the -implementation of `wasm-tools json-from-wast`, itself the component-aware -successor of WABT's `wast2json`) as libraries, so the schema is exactly -upstream's: the serde derives in `json-from-wast`'s `src/lib.rs` -(`Wast`, `Command`, `WasmFile`, `Action`, `Const`) are the specification; -`src/schema.ts` mirrors the subset this suite exercises. Points worth knowing -when reading the files or writing xfail entries: - -- `line` is 1-based and, for the module- and action-bearing asserts - (`assert_invalid/malformed/unlinkable/uninstantiable/trap/return`), is the - line of the *inner form* (`(component ...)` / `(invoke ...)`), not the - `(assert_...` line. It is the key used by `src/xfail.ts` and the lane - expectation overlays. -- Artifacts are `filename` + `module_type: "binary" | "text"`; nothing in - the JSON says whether a binary is a core module or a component. The runner - classifies from the preamble (`artifactKind` in `src/runner.ts`: exact - core preamble → module, anything else → component; in this suite every - top-level artifact is a component). -- Values are `{"type": ..., "value": ...}`. Scalars are decimal strings - (floats as IEEE754 bit patterns; core-value expectations may also be - `nan:canonical` / `nan:arithmetic`), `bool` is a JSON boolean, `record` is `[name, value][]`, - `variant` is `{case, payload?}`, `result` is `{Ok: v|null}` / - `{Err: v|null}`, `option` is `v | null`, `flags` is `string[]`. -- `(component quote "...")` forms are written as `.wat` with - `module_type: "text"` (5 in the current suite, all `assert_malformed`); - the runner records them as skip(`unsupported-directive`). - -## Executor contract (provisional) - -`src/executor.ts` defines `CommandExecutor` — the interface a future -polyengine runtime must implement to make this harness execute for -real. It is deliberately minimal and **will change**; it exists so runner -and runtime evolve against one concrete seam: - -- `validate(artifact)` → verdict (never throws for bad input; malformed vs - invalid need not be distinguished) -- `instantiate(artifact, expect)` → `InstanceRef`; `expect` (`"success"` / - `"trap"` / `"link-error"`) lets a partial executor decline verdicts it - cannot deliver honestly -- `define(name?, artifact)` / `instantiateDefinition(defName?, instanceName?)` - — the `module_definition` / `module_instance` pair -- `register(as, instance?)` -- `invoke(target?, field, args)` / `get(target?, field)` → outcome - (`returned values` | `trapped message`; traps are outcomes, not exceptions) -- `reset()` — drop all per-file state (executor state is per `.wast` file) - -The runner owns instance *naming* and the "current default instance" rule; -the executor owns everything semantic. Anything the executor cannot do yet -throws `PendingRuntimeError`, which the runner records as a -skip(`pending-runtime`) rather than a failure. - -`CoreOnlyExecutor` is the pipeline-sanity stub: core modules are validated -and compiled with the JS `WebAssembly` API; **all component-layer operations -throw `PendingRuntimeError`**. This is forced, not lazy: V8 rejects the -component preamble (`00 61 73 6d 0d 00 01 00`) outright, so -`WebAssembly.validate` returns `false` for valid and invalid components -alike — no component verdict can come from the JS API. That layer is -pinned by a unit test in `tests/runner_unit_test.ts`. It stays available via -`CONFORMANCE_EXECUTOR=core-only deno task test` for pipeline sanity (no shim -build required). - -`RuntimeExecutor` (`src/runtime-executor.ts`) is the real thing, driving -`runtime/`'s public API (`@polyengine/runtime/{shim,exec,cabi,plan}`, -consumed read-only — this is Track A's territory): - -- `validate` / component `instantiate`: `Translator.translate` (the wasm32 - shim under Deno) is the verdict — a structured `{error}` envelope (surfaced - as a thrown `PlanError`) means invalid/malformed; the JS API can't - distinguish the two either, so neither does this. -- successful translation feeds `instantiateComponent` (plan v0 → compiled - core modules → task-model-backed export surface); its `Trap` / - `UnsupportedFeatureError` / `PlanError` outcomes map to - `TrapError`/`LinkError`/`PendingRuntimeError` by the command's expected - outcome (`assert_uninstantiable` vs `assert_unlinkable` vs a plain - instantiation gap). -- `module_definition`/`module_instance` reuse the same translate/instantiate - path against a small per-file definition table (by name, or "most recent"). -- `invoke` calls the export as a plain JS function - (`component.exports[field](...)` — see `runtime/src/exec/boundary.ts` - `createLiftedFunction`) with arguments converted from wast-JSON `Value` to - the runtime's `ComponentValue` host shapes (`src/value-mapping.ts`); the - arity of the raw JS return (`undefined`/bare-value/array, see - `resultsToHost`) is reconstructed into a proper result list using the - export's `FuncType.results.length` (recomputed from the plan, since a - single `list`-typed result is otherwise indistinguishable from a - multi-result array). -- `get` (a core `global.get` wast action) has no component-level equivalent - in this suite; declined honestly as `pending-runtime` rather than guessed. -- capability gaps the sync executor is expected to hit (async canonical - options, stream/future values, error-context — the task scheduler's scope) - are recognized by message substring (`CAPABILITY_MARKERS` in - `runtime-executor.ts`) and reported as skip(`pending-capability: ...`) — a - precise subset of `pending-runtime` naming the exact missing feature, - rather than a generic skip or a false failure. - -### Value comparison - -`src/value-mapping.ts` converts both directions against the runtime's -`ComponentValue` (definitions.py's semantics, our representation — -variant/enum/option/result as -`{kind: label, value: payload}` objects, tuple as despecialized record, -flags as `{label: boolean}`, `list` as `Uint8Array`): `toComponentValue` -for invoke arguments, `compareValue`/`compareValues` for `assert_return` -(recursive, type-directed by the *expected* value's own tag — no separate -`FuncType` needed on either side, matching how the runtime itself never -exposes one across its export-call boundary). - -Floats compare bit-exact: the expected bit-pattern string is decoded via a -shared `DataView` scratch buffer and compared against the actual value's -re-encoded bits, except `nan:canonical`/`nan:arithmetic` expectations, which -match by NaN pattern class instead of exact bits. In practice the runtime's -deterministic NaN profile (`runtime/src/cabi/float.ts`) always produces -exactly the canonical NaN bit pattern, so both classes are satisfied by -every NaN the runtime returns — but the pattern-class check is written -generally in case a less-deterministic engine's NaN ever needs it. - -### Trap-message matching - -`runner.ts`'s `trapMatches` compares by substring first (the suite's own -convention), then falls back to a small checked-in table -(`TRAP_MESSAGE_EQUIVALENTS`) of confirmed-equivalent wording pairs, e.g. the -suite's `"unknown handle index N"` vs. the runtime's -`"table index out of range"`/`"table entry empty"` (both are `trapIf(...)` -call sites in `runtime/src/cabi/handles.ts` — same semantic condition, -independently-authored text). A message pair not in the table is a plain -substring failure, not a silent pass — the table only encodes *confirmed* -equivalences, not a permissive fuzzy match. - -### Triage: xfail list - -`src/xfail.ts` is a checked-in `{file, line, reason}` list (line = the -command's 1-based source line, stable across regen) for commands that fail -today for a known, understood cause outside harness territory (a -translator-shim encoding gap, a sync-vs-task-scheduler semantic gap, etc.) — distinct -from "unexpected regression". `tests/conformance_test.ts` treats a failure -matched in `XFAIL` as `xfail` in the summary rather than `failed`, and it -does not fail the surrounding `Deno.test`. It is *not* auto-verified against -the actual outcome (an xfail'd command that starts passing again isn't -flagged) — periodically diff the summary's `xfail` column against -`XFAIL.length`. - -`test/async/` and `test/values/` are deliberately **not** triaged into -`xfail.ts` — docs/architecture.md §7 excludes `test/values/` from parity scope entirely -(wasmtime doesn't implement component `value` imports/exports), and -`test/async/` is task-scheduler scope; both are expected to show real -failures against the sync-only executor and are left as visible `failed` -counts rather than suppressed, so the summary keeps signaling exactly how -much of the suite the current implementation should be judged against -(binary, linking, resources, validation). - -### Wire-up - -`deno task conformance` = `deno task gen` (testgen regen) + `deno task -shim-check` (build the wasm32 shim if the artifact is missing) + `deno task -test`. `CONFORMANCE_EXECUTOR=core-only` switches `deno task test` back to -the JS-API-only stub. +Generation is deterministic for the same suite and testgen build. The schema +comes from `json-from-wast`; [`src/schema.ts`](src/schema.ts) mirrors the subset +used by this harness. Assertion `line` values identify the inner form's 1-based +source line, not necessarily the enclosing assertion. They remain stable only +while the WAST source does. + +The JSON artifact metadata does not identify core module versus component. +`artifactKind` in [`src/runner.ts`](src/runner.ts) recognizes the exact core +preamble and sends everything else to the component translator. Quoted text is +not compiled by this runner. + +[`src/value-mapping.ts`](src/value-mapping.ts) converts WAST values to the raw +runtime shapes and compares results recursively. Export arity comes from the +plan, so a single list result is not confused with multiple results. Floats +compare by bits, with NaN-pattern classes handled separately. Trap messages use +substring matching plus the explicit `TRAP_MESSAGE_EQUIVALENTS` table in the +runner; unmatched wording fails rather than being accepted by fuzzy matching. diff --git a/harness/browser/entry.ts b/harness/browser/entry.ts index df3b713..233da9a 100644 --- a/harness/browser/entry.ts +++ b/harness/browser/entry.ts @@ -2,18 +2,14 @@ // In-page conformance runner (browser lane). // // This module is bundled to `harness/browser/dist/entry.js` and loaded by -// `index.html`. It is deliberately *dumb*: it executes the same command loop -// the Deno conformance test runs (`harness/src/runner.ts` + +// `index.html`. It executes the same command loop as the Deno conformance test +// (`harness/src/runner.ts` + // `RuntimeExecutor`) and streams raw per-command verdicts back to the driver. // ALL classification (xfail, lane overlays, the per-directory table) stays in // Deno, in `tools/browser/run-lane.ts`, so the browser lane and the Deno lane // share one source of truth for expectations. // // See `tools/browser/run-lane.ts` for run instructions. -// -// Platform note: the bundle substitutes `harness/browser/shims/async_hooks.ts` -// for the runtime's `node:async_hooks` import (FINDING M3A-1). Read that -// file's header before interpreting any `async/` result on a browser lane. import type { WastJson } from "../src/schema.ts"; import { RuntimeExecutor } from "../src/runtime-executor.ts"; diff --git a/harness/browser/expectations/chromium.ts b/harness/browser/expectations/chromium.ts index d5bd9b5..aadc40f 100644 --- a/harness/browser/expectations/chromium.ts +++ b/harness/browser/expectations/chromium.ts @@ -1,40 +1,7 @@ -// Chromium lane expectation — the REQUIRED lane. -// -// Chromium and Deno share V8, so the browser lane SHOULD be the Deno lane. -// **It now is, exactly**: zero deltas, and the TOTAL row below is the Deno -// lane's TOTAL row verbatim. -// -// HISTORY — FINDING M3A-1, now closed. This file used to carry 80 -// `expected-fail` entries across 25 `async/` files. Their single root cause -// was a platform leak, not engine variance: `runtime/src/task/scheduler.ts` -// imported `AsyncLocalStorage` from `node:async_hooks` (a Node/Deno builtin -// no browser ships, and whose `AsyncContext` successor no browser ships -// either) to carry the activation ambient across the engine's JSPI -// continuations. The browser bundle had to substitute a synchronous-extent -// stand-in, and 80 commands fell over. -// -// Track M3A-1 removed the dependency instead of tolerating it: the runtime -// now states that ambient explicitly at the sites it owns — see the -// "engine-driven resumptions" section of `runtime/src/task/scheduler.ts`, the -// `Suspending`-import wrapper in `runtime/src/jspi/bridge.ts`, and -// `SuspensionPoint.owner`. Nothing in `runtime/src` imports a platform -// async-context facility any more (docs/architecture.md §4.3), so the browser bundle -// substitutes nothing and the lane has nothing to excuse. -// -// KEEP THIS LIST EMPTY. It is the detector this lane exists for: any entry -// appearing here again means Chromium diverged from Deno, and the first -// question to ask is whether something re-introduced a Node-only dependency -// or a JSPI resumption path that nobody claims the ambient for. -// -// Measured 2026-08-09 on HeadlessChrome/151.0.7922.34 (playwright 1.62.1, -// linux-arm64). JSPI is present and functional (`WebAssembly.Suspending` / -// `promising` + a live suspend/resume round trip). -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// deltas. Corpus grew 1416->1475 commands; see harness/src/xfail.ts and -// harness/shell/expectations/sm-pinned.ts's header for the new-class -// breakdown (engine-independent by construction). +// Chromium is the required browser lane: expect the Deno totals and no +// per-command deltas. Keep this list empty; investigate divergence rather +// than masking it. In particular, check platform-only dependencies and +// unclaimed JSPI resumption context (the closed M3A-1 failure class). import type { LaneExpectation } from "./types.ts"; diff --git a/harness/browser/expectations/firefox.ts b/harness/browser/expectations/firefox.ts index 3fb19c1..0f4fc6c 100644 --- a/harness/browser/expectations/firefox.ts +++ b/harness/browser/expectations/firefox.ts @@ -1,44 +1,11 @@ // Firefox lane expectation — a findings lane (best-effort, non-gating). // -// RESULT (2026-08-09, Firefox/153.0 via playwright 1.62.1, linux-arm64, -// headless, launched with `javascript.options.wasm_js_promise_integration = -// true`): **the lane runs the full corpus, Deno-identical.** All 59 files, -// 1395 commands. -// -// ENGINE FINDINGS -// --------------- -// 1. JSPI WORKS on Firefox behind the pref, end to end. The in-page probe -// (`harness/browser/entry.ts` `probeJspi`) builds a module with a -// `Suspending` import, wraps the export with `WebAssembly.promising`, and -// gets the suspended value back: `{suspending: true, promising: true, -// roundTrip: true}`. docs/architecture.md §12 (Risks) lists "Firefox: flagged" as an accepted -// risk that a pref flip resolves; the flip is sufficient — no -// SpiderMonkey JSPI bug is visible from this corpus. Nothing in the -// empirical pins (a)-(j) misfires here: with the runtime's ambient made -// explicit (M3A-1, below), Firefox reproduces the Deno lane command for -// command. -// 2. Trap wording (SpiderMonkey vs. V8) is no longer a lane delta. The -// runtime passes each engine's raw core-trap text through unmodified -// (`mapCoreException`, runtime/src/exec/boundary.ts); suite-wording -// normalization (per docs/architecture.md §1, the suite's `assert_trap` -// text is de facto wasmtime/V8 wording) now lives harness-side in -// `TRAP_MESSAGE_EQUIVALENTS` (harness/src/runner.ts), which carries -// SpiderMonkey's "unreachable executed" spelling alongside V8's and JSC's -// for `async/builtin-trap-poisons-instance.wast:9`. The trap happens at -// the right place and poisons the instance identically; only the message -// text ever differed. -// -// FINDING M3A-1 IS CLOSED. This file used to carry 80 further entries, -// identical to Chromium's, for the runtime's `node:async_hooks` dependency. -// Track M3A-1 removed that dependency from `runtime/src` (see -// `harness/browser/expectations/chromium.ts` for the summary), so SpiderMonkey -// now reproduces the Deno lane exactly. -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// deltas. Corpus grew 1416->1475 commands; see harness/src/xfail.ts and -// harness/shell/expectations/sm-pinned.ts's header for the new-class -// breakdown (engine-independent by construction). +// The driver enables `javascript.options.wasm_js_promise_integration`; +// these expectations do not describe Firefox's default configuration. +// `probeJspi` in harness/browser/entry.ts checks a live suspend/resume trip. +// The baseline has no per-command deltas from Deno. Core-trap wording is +// normalized by `TRAP_MESSAGE_EQUIVALENTS` in harness/src/runner.ts, not by +// the runtime. The platform async-context failure class M3A-1 is closed. import type { LaneExpectation } from "./types.ts"; diff --git a/harness/browser/expectations/webkit.ts b/harness/browser/expectations/webkit.ts index fe96524..3fdefc4 100644 --- a/harness/browser/expectations/webkit.ts +++ b/harness/browser/expectations/webkit.ts @@ -1,69 +1,26 @@ // WebKit lane expectation — a findings lane (best-effort, non-gating). // -// RESULT (2026-08-09, playwright 1.62.1 webkit build 2336 = -// Safari/WebKit 26.5, linux-arm64 WPE headless): **the lane runs the full -// corpus.** All 59 files, 1395 commands, 10.3 s wall clock — the fastest of -// the three engines. -// -// GETTING IT TO LAUNCH (this host is Ubuntu questing/25.10; playwright's -// WebKit bundle is linked against the Ubuntu 24.04 ABI, so -// `playwright install-deps` cannot satisfy it — libicu74 in particular has no -// questing candidate). Recipe, all outside the repo except the browser cache: -// fetch the noble arm64 .debs for libicu74, libxml2, libavif16 (+ its codec -// deps libdav1d7 libgav1-1 librav1e0 libyuv0 libSvtAv1Enc1d1 libabsl…), -// libenchant-2-2, libevent-2.1-7t64, libflite1, -// libgstreamer-plugins-bad1.0-0, libharfbuzz-icu0, libhyphen0, -// libmanette-0.2-0, libwayland-server0, libevdev2; `dpkg-deb -x` them; copy -// the resulting `usr/lib/aarch64-linux-gnu/*.so*` into BOTH -// `.browser-cache/webkit-/minibrowser-wpe/sys/lib/` and -// `…/minibrowser-gtk/sys/lib/`. Those directories are already on the -// bundle wrapper's `LD_LIBRARY_PATH`, which is the only reliable channel — -// exporting `LD_LIBRARY_PATH` around the driver does NOT reach the browser -// process. Then run with `PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1`. -// // ENGINE FINDINGS // --------------- -// 1. JSPI WORKS on JavaScriptCore, in a stock playwright WebKit build, with -// no flag: `{suspending: true, promising: true, roundTrip: true}` from the -// in-page end-to-end probe. docs/architecture.md §12 (Risks) records "Safari: JSPI in STP only" -// and excludes stable Safari; on this build the API is present and a real -// suspend/resume round trip completes. Worth re-checking against shipping -// Safari before the exclusion in docs/architecture.md §12 (Risks) is relaxed. -// 2. **JSC does not implement multi-memory in this pinned build** — 58 -// commands fail at `WebAssembly.Module` compile time with "there can at -// most be one Memory section for now". This is the substantive WebKit -// finding: the Component Model's canonical ABI routinely needs more than -// one memory in a single core module (transcoding, cross-component -// copies, realloc-into-another-instance), so a JSC lane is capped until -// multi-memory ships there. It is an engine capability gap, not a host -// defect: the same components run on V8 and SpiderMonkey. 111 further -// commands are CASCADE entries from those failed instantiations ("no -// current instance"). -// RESOLVED UPSTREAM (measured 2026-08-09, polyengine#11): webkit-2342 -// (playwright 1.63 alpha roll) ships multi-memory ENABLED BY DEFAULT — -// the lane reaches 1248/0 against it and 173 of this file's 175 deltas -// collapse, leaving only the two wording entries of finding 3. This -// build's `JSC_useWasmMultiMemory` option is an inert stub (verified: -// the env route works — `JSC_useWasm=0` kills wasm — but the flag -// changes nothing). When the playwright pin reaches a webkit-2342+ roll, -// collapse this overlay to EMPTY; the stale-delta detector -// will insist. Details: https://github.com/polymorph-components/polyengine/issues/11 -// 3. Trap wording (JSC vs. V8) is no longer a lane delta. JSC says +// 1. The pinned Playwright WebKit build passes the in-page JSPI round-trip +// probe without flags. This does not establish support in shipping Safari. +// 2. The pinned build rejects multi-memory modules at compile time. FACT +// adapters can require multiple memories for cross-component copies and +// transcoding. Subsequent commands cascade from failed instantiations. +// The gap is tracked in https://github.com/polymorph-components/polyengine/issues/11; +// newer WebKit builds have demonstrated multi-memory support. Re-measure +// when advancing the pin and remove deltas that the stale-delta detector +// identifies, rather than carrying the overlay forward unchanged. +// 3. Trap wording (JSC vs. V8) is normalized harness-side. JSC says // "Unreachable code should not be executed" for the core `unreachable` // trap where the suite expects the wasmtime/V8 wording // (docs/architecture.md §1); the runtime passes each engine's raw text // through unmodified (`mapCoreException`, runtime/src/exec/boundary.ts) -// and the harness now normalizes it (`TRAP_MESSAGE_EQUIVALENTS`, +// and the harness normalizes it (`TRAP_MESSAGE_EQUIVALENTS`, // harness/src/runner.ts). // -// FINDING M3A-1 IS CLOSED, and its 18 entries are gone from this file. The -// runtime no longer depends on a platform async-context facility (see -// `chromium.ts`), so nothing here is attributable to it any more. The -// cascades that used to hang off those entries re-attribute upward to -// finding 2: in this corpus every `async/` file that M3A-1 broke on JSC also -// contains a multi-memory core module, so the earliest failure in the file is -// the compile-time rejection and the rest of the file cascades from THAT. -// Read finding 2 before blaming anything here on the host. +// M3A-1 (platform async-context dependency) is closed. Cascades in this +// overlay are attributed to finding 2's multi-memory compile rejection. import type { LaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/bun-pinned.ts b/harness/shell/expectations/bun-pinned.ts index bcaf1c5..a5737e7 100644 --- a/harness/shell/expectations/bun-pinned.ts +++ b/harness/shell/expectations/bun-pinned.ts @@ -7,13 +7,11 @@ // PIN: bun v1.3.14 (oven-sh/bun GitHub release zip, sha256-verified against // the release's SHASUMS256.txt — tools/shell/pins.json). Both linux arches. // -// TOTALS: seeded as EXACT Deno-lane parity from a local measurement -// (2026-08-11, linux-arm64 dev box, bun 1.3.14's vendored JSC): 1254 passed -// / 0 failed / 95 xfail, zero bun-specific deltas — but ONLY under +// Expect the Deno baseline with no per-command deltas under // `BUN_JSC_useWasmMultiMemory=1`, which the driver sets (run-lane.ts): // -// * Stock bun 1.3.14 ships wasm multi-memory default-OFF and fails 174 -// corpus commands with "there can at most be one Memory section for +// * Stock bun 1.3.14 ships wasm multi-memory default-OFF and rejects +// multi-memory modules with "there can at most be one Memory section for // now" — the CABI routinely needs >1 memory per core module, the same // gap that capped the pinned-WebKit browser lane (issue #11; JSC trunk // and the jsc-pinned shell have it default-on since rev 318852@main). @@ -30,15 +28,8 @@ // releases of Bun and WebKit without notice" for unknown/renamed BUN_JSC_* // options (and silently ignores them otherwise). The pin freezes that risk. // A RE-PIN MUST RE-VERIFY the option name and re-measure: a bun that -// renames or drops `useWasmMultiMemory` regresses to the stock 174-failure -// shape, which this expectation catches loudly (totals mismatch), findings -// lane or not. -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// xfails. Corpus grew 1416->1475 commands; see harness/src/xfail.ts and -// sm-pinned.ts's header for the new-class breakdown (engine-independent by -// construction). +// renames or drops `useWasmMultiMemory` loses multi-memory support, which +// this expectation reports as a totals mismatch even in a findings lane. import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/jsc-pinned.ts b/harness/shell/expectations/jsc-pinned.ts index 7525269..1debbc7 100644 --- a/harness/shell/expectations/jsc-pinned.ts +++ b/harness/shell/expectations/jsc-pinned.ts @@ -1,51 +1,10 @@ -// JSC pinned lane expectation — REQUIRED gate (promoted from the jsc-trunk -// canary; issue #22 follow-up, "promote engine shells to per-push gates"). +// JSC pinned lane: required gate (issue #22), x86_64 only. The verified +// bundle in tools/shell/pins.json is mirrored under the `shell-pins` release +// because the upstream download host retains only a rolling build window. // -// PIN: rev `318852@main`, sha256-verified (tools/shell/pins.json), mirrored -// to a repo-owned GitHub release (`shell-pins` tag) rather than fetched -// from webkitgtk.org/jsc-built-products directly — that upstream host only -// retains a rolling window of ~42 builds, so a pinned URL to an old rev -// rots within weeks (see tools/shell/fetch.ts's fetchJscPinned header). -// -// TOTALS: seeded as EXACT Deno-lane parity — the pinned bytes are, byte for -// byte (sha256-verified), the same bundle rev the jsc-trunk canary measured -// at exact parity on 2026-08-09 (see `./jsc-trunk.ts`'s header: 1250 passed / -// 0 failed / 99 xfail, zero jsc-specific deltas, full capability matrix -// true — JSPI round trip, multi-memory, wasm-GC, EH, memory64, tail-calls, -// relaxed-simd). Same rev, same hash, same corpus => same result; this is -// not a fresh measurement, it's the same one under a `required: true` gate. -// -// CONFIRMED: first post-merge CI run of this lane (ci.yml `core` job, x64 -// leg) reproduced these totals exactly — see that run's log for the -// re-confirmation (this file predates it only in the sense that the pin is -// deterministic: sha256-verified bytes cannot drift between runs). -// -// x86_64 CI only — jsc-built-products (and this mirror of it) never -// published an arm64 channel (tools/shell/fetch.ts's fetchJscPinned refuses -// cleanly with this note on other arches). MACHINERY VALIDATION on -// non-x86_64 hosts: see `./jsc-trunk.ts`'s header for the local-stable-jsc -// recipe — same caveats apply here (jspi=false, multiMemory=false on a -// stable 2.52 build, large deviation report; not a jsc-pinned finding). -// -// A future re-pin (pins.json version bump) that changes these totals is a -// FINDING to triage before the pin bump lands, not silently absorbed here — -// bump this file's totals only after re-measuring against the new pin. -// -// CM#705 pin advance to 2f13265 (polyengine#173): totals bumped to the -// engine-independent Deno-lane baseline (1475/1428/1263 passed/165 xfail/42 -// pending-runtime/5 unsupported-directive — see harness/src/xfail.ts and -// sm-pinned.ts's header for the new-class breakdown) WITHOUT a local -// re-measurement: this lane is x86_64-only and self-skips on this -// aarch64 dev host (see the recipe body in `justfile`'s `shells` target). -// Every OTHER pinned engine (sm-pinned, node-pinned, sm-nightly, -// bun-pinned — all re-measured on aarch64 this round) hit EXACT Deno-lane -// parity with zero per-row deltas, and this lane's own history is "same -// rev, same hash, same corpus => same result" (see the TOTALS note above) — -// so this bump is UNVERIFIED ON AARCH64, MEASURED-BY-CI: the x64 CI leg of -// `core` is what actually re-confirms it against the new pin (as it did for -// the prior bump per the CONFIRMED note above). Any delta the CI leg finds -// there is a finding to triage before merge, not something this dev-host -// pass could have caught. +// Expect the Deno baseline with no per-command deltas. An aarch64 local run +// skips this lane; the x64 CI leg must verify changes to this baseline. +// Re-measure and triage changed totals before accepting a new engine pin. import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/jsc-trunk.ts b/harness/shell/expectations/jsc-trunk.ts index 4b39b2c..5f14d66 100644 --- a/harness/shell/expectations/jsc-trunk.ts +++ b/harness/shell/expectations/jsc-trunk.ts @@ -1,20 +1,12 @@ // JSC trunk lane expectation — a findings lane (best-effort, non-gating; -// issue #22). Seeded from the first successful CI run (2026-08-09, GH -// Actions ubuntu-24.04 x64, bundle rev `318852@main` built 2026-08-08): -// **EXACT Deno-lane parity** — 1254 passed / 0 failed / 95 xfail (the Deno -// lane's own classes, identical classification), zero jsc-specific deltas, -// and the full capability matrix true: JSPI (round trip verified), -// multi-memory, wasm-GC, exception-handling, memory64, tail-calls, -// relaxed-simd. This corroborates the webkit-2342 browser measurement on -// issue #11 (multi-memory default-on in trunk) from an independent channel, -// and is now re-measured weekly by `.github/workflows/canary.yml`. +// issue #22). Expect the Deno baseline with no per-command deltas; +// `.github/workflows/canary.yml` measures trunk on x86_64. // // Trunk moves: a future drift in totals/capabilities is a FINDING to // triage (engine change vs harness assumption), not a failure — the driver // exits 0 either way and reports the diff. // -// EXECUTION MODEL (the two first-run artifacts, fixed in fetch.ts — -// details there): the bundle must stay intact and run via its shipped +// EXECUTION MODEL (see tools/shell/fetch.ts): keep the bundle intact and use its // compiled wrapper (`/jsc`), because `bin/jsc` carries a RELATIVE // PT_INTERP resolved from the bundle root; and the zip's lib/*.so.N names // are symlink entries that must be materialized as real symlinks. @@ -48,19 +40,6 @@ // - unreachable trap wording: `"Unreachable code should not be executed"` // — already a `TRAP_MESSAGE_EQUIVALENTS` row in `harness/src/runner.ts`, // no matcher work needed here. -// -// CM#705 pin advance to 2f13265 (polyengine#173): totals bumped to the -// engine-independent Deno-lane baseline (1475/1428/1263 passed/165 xfail/42 -// pending-runtime/5 unsupported-directive — see harness/src/xfail.ts and -// sm-pinned.ts's header for the new-class breakdown) WITHOUT a local -// re-measurement: this lane is x86_64-only and self-skips on this -// aarch64 dev host. Every OTHER shell lane re-measured on aarch64 this -// round (sm-pinned, node-pinned, sm-nightly, bun-pinned) hit EXACT -// Deno-lane parity with zero per-row deltas, and this lane tracks -// jsc-pinned exactly (same rev, same hash) — so this bump is UNVERIFIED ON -// AARCH64, MEASURED-BY-CI: canary.yml's weekly x64 run is what actually -// re-confirms it (or reports trunk drift, per this file's own findings-only -// discipline). import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/node-pinned.ts b/harness/shell/expectations/node-pinned.ts index 30d042a..97cdf88 100644 --- a/harness/shell/expectations/node-pinned.ts +++ b/harness/shell/expectations/node-pinned.ts @@ -4,14 +4,8 @@ // against the release's SHASUMS256.txt — tools/shell/pins.json). Both linux // arches are published, so unlike jsc-pinned this lane runs on both CI legs. // -// TOTALS: seeded as EXACT Deno-lane parity from a local measurement -// (2026-08-11, linux-arm64 dev box, V8 14.6.202.34-node.28): 1254 passed / -// 0 failed / 95 xfail, zero node-specific deltas, full capability matrix -// true (JSPI round trip, multi-memory, wasm-GC, EH, memory64, tail-calls, -// relaxed-simd) — with NO runtime flags: wasm JSPI is on by default in -// node >= 26, exactly as docs/architecture.md §3's engine table recorded. -// Like jsc-pinned's seeding, determinism does the heavy lifting: pinned -// bytes + pinned corpus + pinned shim flags; confirm on the first CI run. +// Expect the Deno baseline with no per-command deltas or runtime flags. +// JSPI is enabled by default in this pin. // // WHY 26.x AND NOT 24 LTS (measured, same box, same corpus): node 24.18's // V8 13.6 gates JSPI behind `--experimental-wasm-jspi`, and even with the @@ -22,19 +16,9 @@ // node 26 / V8 14.6. Recorded so nobody re-lanes the LTS expecting clean // parity: a node-24 lane needs flag plumbing AND a 2-delta overlay. // -// WHAT THIS LANE ADDS over the Deno lane (same V8 family): the node -// EMBEDDING — ESM loading of the bundle, the node event loop under the -// scheduler, and node's pooled-Buffer I/O (tools/shell/host-node.mjs must -// copy out of the pool before bytes reach WebAssembly APIs; handing the -// pool-backed .buffer to wasm is a classic node-embedder defect this lane -// would catch). Raw-engine coverage was already carried by the shell and -// browser lanes; this pin is about the runtime consumers actually deploy. -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// xfails. Corpus grew 1416->1475 commands; see harness/src/xfail.ts and -// sm-pinned.ts's header for the new-class breakdown (engine-independent by -// construction). +// This lane also exercises Node's embedding: ESM loading, its event loop, +// and pooled-Buffer I/O. tools/shell/host-node.mjs must pass only the intended +// bytes to WebAssembly APIs, not a Buffer's entire backing pool. import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/sm-nightly.ts b/harness/shell/expectations/sm-nightly.ts index 555dc48..bb93295 100644 --- a/harness/shell/expectations/sm-nightly.ts +++ b/harness/shell/expectations/sm-nightly.ts @@ -1,21 +1,9 @@ // SpiderMonkey nightly lane expectation — a findings lane (best-effort, // non-gating; issue #22). // -// RESULT (2026-08-09, SpiderMonkey nightly `js` shell, linux-aarch64, -// jsshell-linux-aarch64.zip fetched from -// archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/): **the -// lane runs the full corpus, Deno-identical.** All 59 files, 1395 commands, -// zero deltas, zero stale xfails, zero unexpected failures. -// -// CAPABILITY MATRIX (this run): jspi = {suspending: true, promising: true, -// roundTrip: true}, multiMemory = true, wasmGc = true, exceptionHandling = -// true, memory64 = true, tailCalls = true, relaxedSimd = true. Every -// proposal this lane's compile-probes (`tools/shell/probes/*.wasm`) check is -// already implemented in this nightly build — consistent with SpiderMonkey -// nightly being the trailing-edge-but-still-ahead-of-stable-Firefox -// reference issue #22 was written to watch (JSPI unflagged here vs. Firefox -// 153's `javascript.options.wasm_js_promise_integration` pref — see -// `harness/browser/expectations/firefox.ts`). +// Expect the Deno baseline with no per-command deltas. The shell's JSPI +// defaults do not establish availability in the shipping browser config; +// see harness/browser/expectations/firefox.ts. // // SHELL-SURFACE FINDINGS (feed the polyfill scope in `tools/shell/polyfill.ts` // and the shell detection in `tools/shell/entry.ts`): @@ -42,12 +30,6 @@ // Track this file the same way `harness/browser/expectations/firefox.ts` // tracks Firefox: any future delta gets a named, dated entry here, never a // blanket overlay. -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// xfails. Corpus grew 1416->1475 commands; see harness/src/xfail.ts and -// sm-pinned.ts's header for the new-class breakdown (engine-independent by -// construction). import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/shell/expectations/sm-pinned.ts b/harness/shell/expectations/sm-pinned.ts index d6e0b9c..0e89624 100644 --- a/harness/shell/expectations/sm-pinned.ts +++ b/harness/shell/expectations/sm-pinned.ts @@ -1,49 +1,12 @@ -// SpiderMonkey pinned lane expectation — REQUIRED gate (promoted from the -// sm-nightly canary; issue #22 follow-up, "promote engine shells to -// per-push gates"). +// SpiderMonkey pinned lane: required gate (issue #22). The release jsshell +// in tools/shell/pins.json is hash-verified on both supported Linux arches. // -// PIN: Firefox release 153.0 jsshell (tools/shell/pins.json), matching the -// browser lane's Firefox version exactly — sha256-verified, both arches -// (archive.mozilla.org release archives are permanent, unlike -// jsc-built-products' rolling window, so no mirror is needed here — see -// tools/shell/fetch.ts's fetchSpiderMonkeyPinned header). -// -// SHELL vs BROWSER CONFIG (important distinction — read before trusting -// this file as a stand-in for "Firefox 153 ships JSPI"): this jsshell has -// JSPI ENABLED BY DEFAULT (verified empirically, see the RESULT below) — -// UNLIKE the Firefox 153 *browser*, which gates JSPI behind the -// `javascript.options.wasm_js_promise_integration` pref (the browser lane's -// driver sets that pref explicitly; see -// `harness/browser/expectations/firefox.ts`). The jsshell is a -// developer/testing build with different default flags than the shipping -// browser config — this lane measures the ENGINE's capability ceiling, not -// what ships to users. The browser lane (post-merge only as of this track) -// remains the check for the actual shipping-config surface. -// -// RESULT (2026-08-09, seed run — this host is aarch64; the release-153 -// jsshell ships an aarch64 build, sha256-verified via pins.json): -// **the lane runs the full corpus, EXACT Deno-lane parity.** All 59 files, -// 1395 commands, 1349 executed, 1254 passed, 0 failed, 95 xfail, 41 -// pending-runtime, 0 pending-capability, 5 unsupported-directive. Zero -// deltas, zero stale xfails, zero unexpected failures. Capability matrix: -// jspi = {suspending: true, promising: true, roundTrip: true}, multiMemory -// = true, wasmGc = true, exceptionHandling = true, memory64 = true, -// tailCalls = true, relaxedSimd = true — multi-memory shipped in Fx125, so -// full parity here was expected; it held exactly, no deltas to record. -// -// Any future re-pin (pins.json version bump) that changes these totals is a -// FINDING to triage before the pin bump lands, not silently absorbed here — -// bump this file's totals only after re-measuring against the new pin. -// -// CM#705 pin advance to 2f13265 (polyengine#173): re-measured (this host is -// aarch64) — full Deno-lane parity holds exactly, zero deltas, zero stale -// xfails. Corpus grew 1416->1475 commands (new reentrance.wast/ -// during-sync-scheduling-candidates.wast/during-sync-call-exclusive-resume. -// wast files, new kebab.wast/max-value-size.wast assert_invalid rows, plus -// a few line-shifted-but-unchanged rows); the new xfail classes -// (cm705-reentrance, cm705-sync-sched, cm707-cancel, name-rules-nyi, -// max-value-size-nyi) are translate-time / our-own-runtime -// drift, engine-independent by construction — see harness/src/xfail.ts. +// This shell enables JSPI by default, unlike the Firefox browser pin, whose +// lane sets `javascript.options.wasm_js_promise_integration` explicitly. +// Neither lane establishes JSPI availability in the default browser config. +// Expect the Deno baseline with no per-command deltas. Re-measure and triage +// changed totals before accepting a new engine pin; shared failure classes +// are recorded in harness/src/xfail.ts. import type { ShellLaneExpectation } from "./types.ts"; diff --git a/harness/src/runner.ts b/harness/src/runner.ts index 5b534bd..69bb320 100644 --- a/harness/src/runner.ts +++ b/harness/src/runner.ts @@ -42,7 +42,7 @@ export function artifactKind(bytes: Uint8Array): Kind { /** * `pending-capability` is a precise, named-in-report subset of * `pending-runtime`: the executor understood the command but a specific - * runtime feature it needs (e.g. async/streams) does not exist yet. + * runtime feature it needs (e.g. shared-everything threads) is unsupported. * `PendingRuntimeError` messages prefixed `pending-capability: ` are * classified this way (see `RuntimeExecutor`). */ diff --git a/harness/src/runtime-executor.ts b/harness/src/runtime-executor.ts index 4cdbe3e..da80a13 100644 --- a/harness/src/runtime-executor.ts +++ b/harness/src/runtime-executor.ts @@ -1,17 +1,10 @@ // The real executor: components run through the plan executor -// (translator shim -> plan v0 -> runtime/src/exec), core modules through the -// JS WebAssembly API (delegated to CoreOnlyExecutor's module path, which is -// exact and needs no reimplementation). +// (translator shim -> plan -> runtime/src/exec), core modules through the +// JS WebAssembly API (CoreOnlyExecutor's module path). // -// Scope note: the runtime is a *sync* executor (docs/architecture.md §6 -// degenerate path); async canonical options / stream|future types raise -// PlanError("... task scheduler ...") from -// runtime/src/exec/boundary.ts createLiftedFunction, or NotImplemented from -// runtime/src/cabi lift/lower for stream/future values. Both are mapped here -// to PendingRuntimeError with a `pending-capability: ` prefix (see -// runner.ts) rather than left to surface as a failure, since the command is -// understood but the capability plainly doesn't exist yet — precise -// hand-off to whichever track builds it. +// Declared capability gaps are mapped to PendingRuntimeError with a +// `pending-capability: ` prefix. Other exceptions retain their failure +// classification; see CAPABILITY_MARKERS and the phase-specific handlers. import { loadPlan, PlanError, TranslateError } from "@polyengine/runtime/plan"; import type { LoadedPlan } from "@polyengine/runtime/plan"; @@ -40,8 +33,7 @@ import { import type { Value } from "./schema.ts"; import { collapseResultsByArity, toComponentValue } from "./value-mapping.ts"; -/** Substrings that indicate the command needs a not-yet-built runtime - * capability (async/streams/etc) rather than a genuine bug. Checked +/** Substrings used to classify declared runtime capability gaps. Checked * against thrown error messages from the shim/plan/executor. Keep narrow — * anything else surfaces as a real failure. */ const CAPABILITY_MARKERS = [ diff --git a/harness/src/summary.ts b/harness/src/summary.ts index d1c83a6..1c970ba 100644 --- a/harness/src/summary.ts +++ b/harness/src/summary.ts @@ -54,8 +54,7 @@ export class Summary { case "passed": stats.executed++; stats.passed++; - // Stale-xfail detection (G7, now a real gate): an entry whose - // command passes is stale and must be pruned. + // Passing commands must also be classified to detect stale xfails. if (isXfail(r)) { this.staleXfails.push({ file: file.source, line: r.line }); } diff --git a/harness/src/xfail.ts b/harness/src/xfail.ts index ef9f581..7cff9f6 100644 --- a/harness/src/xfail.ts +++ b/harness/src/xfail.ts @@ -7,7 +7,7 @@ // // Entries here are removed as the runtime gains the capability that makes // them pass; an xfail entry whose command now PASSES fails the run loudly -// (the stale-xfail detector in tests/conformance_test.ts, a real G7 gate) — +// (the stale-xfail detector in tests/conformance_test.ts) — // prune stale entries rather than accumulating masks. export interface XfailEntry { @@ -157,12 +157,10 @@ export const XFAIL: XfailEntry[] = [ // --- values/post-return.json: post-return.wast:4 ($Tester) declares // every async built-in (task.return, thread.yield/INDEX, waitable-set.*, // subtask.*, stream.*, future.*) to assert they trap from a post-return - // function. The task core shipped; the SURVIVING refusal is - // 'thread-index' — the 🧵 shared-everything-threads class, deferred by - // https://github.com/polymorph-components/polyengine/issues/12 — so the component still declines at instantiation and all 28 - // assert_traps cascade off 'no current instance'. (Reason strings - // rewritten after the task core shipped: they previously named it as - // missing, which would misdirect triage.) + // function. Instantiation requires the unsupported 'thread-index' + // trampoline: deferred-threads, shared-everything threads, + // https://github.com/polymorph-components/polyengine/issues/12. + // The assertions cascade off 'no current instance'. { file: "values/post-return.json", line: 202, @@ -307,84 +305,19 @@ export const XFAIL: XfailEntry[] = [ line: 256, reason: "same 🧵 thread-index (deferred threads, https://github.com/polymorph-components/polyengine/issues/12) dependency as line 202", }, - // post-return.wast:260 uses `context.get`/`context.set`, which wasmtime - // lowers to `CoreDef::UnsafeIntrinsic` — the CoreDef `unsafe-intrinsic` - // encoding (contracts/plan-format.md schema) has no wire form for this yet - // — a known task-scheduler blocker. - // post-return.wast:334 calls `backpressure.inc`/`backpressure.dec` from a - // post-return function. NOTE the observed symptom is a *wrong value* - // ("expected u32 11, got 5"), not an error: the module command fails as a - // capability skip and the invoke then runs against the previously - // instantiated component, which also exports `f`. The value mismatch is an - // artifact of that, not a canonical-ABI bug. - // --- values/variants.json: GREEN. variants.wast:83's component mixes an - // async-lifted export (`mix-ret`) with sync ones, reached through FACT - // adapters; prepare-call / {sync,async}-start-call made it - // instantiate and run — pinned by runtime/tests/integration/ - // e2e_suite_test.ts ("async-lifted exports instantiate and run"). --- - - // ===================================================================== - // async/ — triage as of streams/futures/error-context support. - // - // Streams, futures and error-context are IMPLEMENTED; the entries below no - // longer describe a missing value type. The dominant remaining class is - // JSPI: the *synchronous* form of a stream/future copy, of - // `waitable-set.wait`, and of a cross-component call all block the calling - // wasm frame, which a stackless runtime cannot do. See - // runtime/src/intrinsics/stream_builtins.ts `finishCopy`. - // - // Historic note (early triage) follows. - // - // What now works and is NOT listed here: `trap-on-reenter`, - // `validate-no-async-abi-for-sync-type` and `validate-no-stream-char` are - // fully green, and individual commands pass in eight more files. - // - // What blocks the rest, in order of weight: - // * FACT cross-component async calls (`async-start-call`, - // `sync-start-call`) — 49 commands. This phase implements the async ABI - // at the *host* boundary; the suite almost always drives async through a - // second component, which goes via FACT's adapter intrinsics instead. - // * streams / futures — 41 commands (out of this track). - // * 166 further commands are *cascades*: once a component instance is - // declined at instantiation, every later command against it fails with - // "no current instance". They carry the root cause's reason. - // * 4 genuine one-off gaps, each with its own entry (trap-message - // fidelity, instance poisoning, instantiation-time task context, and one - // shim decoder gap). - // ===================================================================== - // --- async/builtin-trap-poisons-instance.json: root cause: STREAMS --- - // --- async/cancel-stream.json: root cause: STREAMS --- - // --- async/closed-stream.json: root cause: STREAMS --- - // --- async/cross-abi-calls.json: root cause: FACT-ASYNC --- - // --- async/cross-task-future.json: root cause: STREAMS --- - // --- async/drop-cross-task-borrow.json: root cause: FACT-ASYNC --- - // --- async/drop-stream.json: root cause: STREAMS --- - // --- async/drop-waitable-set.json: root cause: FACT-ASYNC --- + // Async failures below distinguish unsupported capabilities from assertions + // cascading after a skipped instantiation. Keep each cascade tied to its + // root cause rather than treating it as an independent runtime failure. // --- async/during-sync-call-*.json + during-sync-scheduling-candidates.json: // all pin 🧵 sync-call-blocking semantics and are built largely from thread // built-ins (thread.new-indirect / resume-later / suspend-then-resume / // suspend / index / yield-then-promote) — the deferred-threads class, - // https://github.com/polymorph-components/polyengine/issues/12. (History: at the prior pin these components - // failed TRANSLATION first, under the now-exited wasmparser/wast - // pin-drift class — see the EXIT note at the top of this file.) At the - // current pin every one of these components TRANSLATES and DEFINES - // fine; the remaining failure is that INSTANTIATING them requires a - // host trampoline for a thread built-in (`thread-index`, - // `thread-new-indirect`, ...) that polyengine's executor does not yet - // implement — a `module`/`module_instance` command reports - // `pending-capability: instantiate: component requires host trampoline - // '...'` and is SKIPPED (not failed; no xfail entry needed for it), and - // every later assert against that instance cascades with "no current - // instance". `async/during-sync-call-exclusive-resume.json` and - // `async/during-sync-scheduling-candidates.json` are BRAND NEW files added by - // the CM#705 pin advance (polyengine#173, third_party/component-model - // 2f13265) — they did not exist pre-advance, so these are new entries, - // not renumbered ones. Predicted class from that dispatch was - // `cm705-sync-sched` (polyengine#249, a semantic scheduling deviation); - // investigation there found the observed failures were translator-level - // at the time (the since-exited pin-drift class); the subsequent re-pin - // resolved that translation gap, uncovering the SAME deferred-threads - // class (#12) one layer down at instantiation. --- + // https://github.com/polymorph-components/polyengine/issues/12. + // Translation and definition succeed, but instantiation requires unsupported + // thread trampolines. `module`/`module_instance` is skipped as + // pending-capability; later assertions cascade with "no current instance". + // These are deferred-threads failures, not the initially suspected + // cm705-sync-sched class (polyengine#249) or the resolved pin-drift class. { file: "async/during-sync-call-may-block-if-other-ready-threads.json", line: 111, @@ -427,14 +360,9 @@ export const XFAIL: XfailEntry[] = [ line: 207, reason: "same cascade as line 206, see that entry", }, - // async/during-sync-call-exclusive-resume.json: BRAND NEW file (CM#705 pin - // advance, polyengine#173; test/async/during-sync-call-exclusive-resume.wast - // is 100% new content, not a renumbering of the deleted - // during-sync-call-no-exclusive-resume.wast). All three of its components - // are built from thread.index/suspend/resume-later; each `module` command - // is itself pending-capability ('thread-index' at line 9, 'thread-suspend' - // at line 65 -- for the third component, whose own preceding module - // command line is not separately listed here) and needs no xfail entry. + // async/during-sync-call-exclusive-resume.json: thread.index/suspend/ + // resume-later make the module commands pending-capability. Only the + // cascading assertions need xfail entries (deferred-threads, #12). { file: "async/during-sync-call-exclusive-resume.json", line: 59, @@ -461,8 +389,7 @@ export const XFAIL: XfailEntry[] = [ line: 103, reason: "same cascade as line 102, see that entry", }, - // async/during-sync-scheduling-candidates.json: BRAND NEW file (CM#705 pin - // advance, polyengine#173). Six components, each built from thread + // async/during-sync-scheduling-candidates.json: components built from thread // built-ins (thread.new-indirect/resume-later/suspend/index/ // yield-then-promote); every component's `module`/`module_instance` // command is pending-capability (deferred thread built-ins, #12) and @@ -526,11 +453,8 @@ export const XFAIL: XfailEntry[] = [ line: 237, reason: "same cascade as line 234, see that entry", }, - // Note: this file's line 245 ("BlockedCallbackTester" component - // definition) itself now TRANSLATES AND DEFINES successfully at the - // 4675ee1 pin (it was stale here pre-cleanup, per the stale-xfail - // detector, and has been pruned); the definition's later use at line 303 - // still needs a host trampoline this executor lacks. + // BlockedCallbackTester defines successfully; instantiating it requires + // an unsupported thread trampoline. { file: "async/during-sync-scheduling-candidates.json", line: 304, @@ -557,9 +481,7 @@ export const XFAIL: XfailEntry[] = [ line: 309, reason: "same cascade as line 308, see that entry", }, - // Same as above for line 315 ("SyncLiftedTester"): the definition itself - // is pruned-stale here (now translates fine); its uses at 403/407 still - // lack a host trampoline. + // SyncLiftedTester has the same instantiation-time thread dependency. { file: "async/during-sync-scheduling-candidates.json", line: 404, diff --git a/justfile b/justfile index 9bc94bd..b31db8e 100644 --- a/justfile +++ b/justfile @@ -1,21 +1,14 @@ -# The orchestration surface: repo-wide recipes here, the CI job bodies in -# .github/justfile (the `gha` module) — each CI job runs exactly one -# `gha::` recipe, so `just ci` is exactly CI. Recipe bodies are the exact -# commands (AGENTS.md "Gates" maps onto them 1:1); comments that used to -# live on workflow steps live on the recipes now. +# Repo-wide recipes; .github/justfile composes the CI job bodies as `gha::` recipes. mod gha '.github' default: @just --list -# The canary lanes are findings-only crons (`gha::canary`, `gha::canary-arm`). -# Exactly the CI jobs: the required `core` matrix + the post-merge `browser` job. +# CI suites: required core matrix and post-merge browser job; excludes canaries. ci: (gha::core) (gha::browser) -# Includes the consumer smokes CI cannot run (they need the polymorph -# checkouts; docs/consumers.md). -# The full pre-commit pass (AGENTS.md "Gates"): everything. +# Full pre-commit gates, including local consumer smokes (docs/consumers.md). gates: version-guard-local fmt-check build test-rust test-protocol test-runtime test-wasi test-sockets-node test-ct-runner test-bundle test-version-guard publish-check test-npm examples test-translate conformance sched-seeds shells browsers smoke-tls smoke-c0 # Fast sanity: builds + native tests + type-checks, no suites. @@ -25,9 +18,7 @@ check: fmt-check build test-rust cd wasi && deno task check cd ct-runner && deno task check -# The runtime package is formatter-clean (`deno fmt`, stock settings; the -# generated bindgen snapshots/envelopes and generator output are excluded in -# runtime/deno.json). Fix with `cd runtime && deno fmt`. +# Runtime formatting; generated files are excluded by runtime/deno.json. fmt-check: cd runtime && deno fmt --check @@ -36,13 +27,8 @@ fmt-check: build: cargo build --workspace -# The translator shim wasm: every Deno suite below loads this artifact. -# Size-tuned (~1.8 MB raw / ~0.5 MB gzip, vs 3.8 MB stock -# release): the shim is a shipped asset (issue #16), so the wasm build opts -# into z/lto/abort via scoped env vars — the workspace [profile.release] -# stays stock so testgen/bindgen keep fast builds and fast corpus runs. -# Semantics are untouched (same crate, same deps); the conformance gate is -# the check that matters and runs on this artifact. +# The workspace release profile stays unchanged for native testgen/bindgen builds. +# Build the tested/shipped translator wasm with size tuning scoped to this build. shim: CARGO_PROFILE_RELEASE_OPT_LEVEL=z \ CARGO_PROFILE_RELEASE_LTO=fat \ @@ -52,30 +38,21 @@ shim: cargo build -p translator-shim --target wasm32-unknown-unknown --release cp target/wasm32-unknown-unknown/release/translator_shim.wasm translator/translator_shim.wasm -# wasmtime CLI is optional in build.sh (smoke run only when present). -# Guest fixture components (examples/guests/build/, gitignored): the -# runtime e2e suites and ct-runner's fixture tests need them. +# Guest components for runtime/ct-runner tests; optional wasmtime smoke when installed. fixtures: ./examples/build.sh -# The consumer-facing embedder examples (examples/README.md): build each -# guest component and run its self-checking host. These double as living -# documentation of the embedder API — CI runs them so they cannot rot. +# Build guest components and run the self-checking embedder examples. examples: shim ./examples/hello-world/run.sh ./examples/kitchen-sink/run.sh -# Build-time translation CLI (tools/translate, contracts/embedder-api.md -# §"Module wiring and instantiation"): translate -# to an envelope, reconstitute artifacts without a translator, verify the -# mismatched-pair refusal. +# Translation CLI/package tests: envelope loading and mismatched-pair refusal. test-translate: shim deno test --allow-read --allow-write=/tmp --allow-run tools/translate/translate_test.ts cd translator && deno task check && deno task test -# Rehearsal finding: 20 runtime e2e tests self-skip when it is absent — -# generation must precede the runtime suite (318/0/3 with; 298/0/23 without). -# The conformance corpus (harness/generated/). +# Generate harness/generated/ before runtime tests, which otherwise skip corpus cases. corpus: cd harness && deno task gen @@ -96,30 +73,19 @@ test-rust: test-runtime: shim fixtures corpus cd runtime && deno task check && deno task test -# The lift/lower CONVENTIONS suite alone (contracts/embedder-api.md -# §"The host-ABI surface and its version"): the executable definition of the host ABI, transcripts compared against -# the committed goldens under `runtime/tests/conventions/golden/`. It lives -# under runtime/tests/, so `test-runtime` already runs it — this is the focused -# lane for working on the host boundary. Updating a golden asserts a host-ABI -# behavior change; runtime/tests/conventions/support.ts's header carries the -# update command and the labelling rule. +# Golden update rules: runtime/tests/conventions/support.ts and contracts/embedder-api.md. +# Focused host-ABI golden suite (also included in test-runtime). test-conventions: shim fixtures cd runtime && deno test --allow-read=..,/tmp --allow-write=/tmp --allow-env=POLYENGINE_SCHED_SEED tests/conventions/ -# The brand vocabulary (contracts/embedder-api.md §"Module identity and -# @polyengine/protocol"): dependency- -# free, so this is the one Deno suite that needs no build artifacts at all. +# Protocol vocabulary tests; no build artifacts required. test-protocol: cd protocol && deno task test test-wasi: cd wasi && deno task test -# The sockets fragment on REAL pinned Node (the whole test-wasi -# suite exercises the same node-builtins backend under Deno's node-compat; -# this lane covers the genuine platform). `deno bundle` resolves the -# workspace imports into one self-contained ESM file; tests/dist/ is -# gitignored. +# Sockets on pinned Node, complementing test-wasi's Deno node-compat coverage. test-sockets-node: deno run -A tools/shell/fetch.ts node-pinned deno bundle --platform browser --format esm -o wasi/tests/dist/node_smoke.mjs wasi/tests/node_smoke.ts @@ -128,98 +94,43 @@ test-sockets-node: test-ct-runner: shim fixtures cd ct-runner && deno task test -# The embedder-bundle release-asset gate (polyengine-embedder.mjs: -# build + shape checks for tools/release-bundle/entry.ts). -# `dual_copy_test.ts` rides here because the bundle IS the second runtime copy -# (contracts/embedder-api.md §"Module identity and @polyengine/protocol"; -# issue #83): it is the only way to get two genuinely distinct -# copies in one process — query-string cache-busting does not, since relative -# imports below the entry resolve to the same cached modules. +# Bundling creates a distinct runtime graph; entry query strings share dependencies. +# Release-bundle shape, execution and cross-copy tests. test-bundle: shim deno test -A tools/release-bundle/ -# The release version guard's own unit tests (tools/version-guard/): semver -# ordering, and every pr/publish/cut check firing and passing against -# injected fixtures — no network, no `gh`, no repository state, because the -# guard's effects are injected. The guard itself runs in CI as gha::core's -# first step (`pr` mode, a no-op outside pull_request runs) and inside -# release.yml (`publish` and `cut` modes); this recipe is what keeps its -# decision logic honest before either of those sees it. +# Version-guard unit tests with injected effects; no network or repository state. test-version-guard: deno test -A tools/version-guard/ -# The release version guard's label-free, event-free pass -# (tools/version-guard/check.ts `local`): lockstep agreement, monotonicity -# against a locally-derivable last cut (git tags, falling back to jsr.io's -# published `runtime` latest — no GitHub API, no PR context), and the -# protocol byte-identity tear check (the #219/#232 incidents) run fatally -# against the WORKING TREE, plus an advisory reminder if committed -# conventions goldens changed. It exists because `pr` mode no-ops the -# instant PR_NUMBER is unset (see below) — so neither a push run nor a -# pre-push `just gates` ever asked "would this tear protocol?" before #232 -# shipped one straight through. `local` is the split: anyone can run it -# with no GitHub context at all, so it goes first in `gates` — cheap, and -# catches a versioning mistake before the expensive suites run at all. -# Same permission shape as `version-guard-pr` (see its comment for why -# --allow-run is not narrowed to `gh,git`) minus --allow-env: `local` reads -# no environment variables at all (no PR_NUMBER/PR_BASE_SHA/GITHUB_*), by -# design — that is what makes it the label-free/event-free half of the -# split. +# Uses git tags or JSR for the last cut, without GitHub/PR context or env access. +# Checks lockstep, monotonicity and published protocol bytes; goldens are advisory. +# Label-free working-tree version checks. version-guard-local: deno run --allow-net=jsr.io --allow-run --allow-read=. tools/version-guard/check.ts local -# The release version guard's early-warning pass (tools/version-guard/check.ts -# `pr`): lockstep agreement, monotonicity against the last cut, breaking/* -# label ↔ minor-bump agreement in both directions, and the protocol-tear -# warning (protocol/src moving without a protocol/deno.json bump — the #219 -# incident). Runs first in `gha::core` so a versioning mistake is the first -# thing a PR hears about, and exits 0 immediately when PR_NUMBER is unset, so -# push runs and a local `just ci` stay green without GitHub. Labels are read -# LIVE from the API (never the event payload) — but label edits deliberately -# do not re-trigger CI, so this is a warning: the enforcement point is -# release.yml's `cut` mode. -# Explicit permissions rather than -A: net is jsr.io only. --allow-run is -# NOT narrowed to `gh,git` — Deno refuses an allowlisted spawn whenever a -# dynamic-linker variable (LD_LIBRARY_PATH) is set in the environment, which -# is exactly the shape a nix-ish shell or a CI runner image can have, so the -# narrow form fails by environment rather than by policy. +# Live labels can change after CI without rerunning it; release checks remain necessary. +# --allow-run stays broad in both guard recipes: Deno rejects allowlisted spawns +# when dynamic-linker environment variables such as LD_LIBRARY_PATH are set. +# PR version/label checks; no-op without PR_NUMBER. version-guard-pr: deno run --allow-net=jsr.io --allow-run --allow-read=. --allow-env tools/version-guard/check.ts pr -# The JSR publish checks (public-API type check, slow types, export and -# import analyzability, config validation) — `deno task check` covers -# none of them, so they only fired at publish time on main before this -# gate. Needs the shim: @polyengine/translator ships translator_shim.wasm -# (statically imported by shim_asset_deno.ts). Registry-side failures -# (scope auth, version conflicts) still only manifest on a real publish. -# --allow-dirty because this is a PRE-commit gate (the dirty check -# protects uploads; there is no upload here). -# JSR publish verification, no upload (`deno publish --dry-run`). +# Auth/version conflicts require real publishing; --allow-dirty allows pre-commit checks. +# JSR package/type/export validation without upload. publish-check: shim deno publish --dry-run --allow-dirty -# The npm distribution of the five JSR packages (tools/npm-build/build.ts). -# Needs the shim: @polyengine/translator carries translator_shim.wasm as a -# packaged asset on the npm side too. Output is gitignored. +# Build npm packages, including the translator wasm asset; output is gitignored. npm-build: shim deno run -A tools/npm-build/build.ts -# The npm distribution's gate: pack the built packages, install them as a -# consumer would, then run the pipeline for real (translate + instantiate a -# guest) and type-check the SHIPPED .d.ts from outside. Catches what -# `publish-check` cannot — npm `exports` subpaths, dependency edges, tarball -# file lists — and above all pins the single-copy property: cross-package -# imports must be npm dependencies, never inlined source (contracts/embedder-api.md -# §"Module identity and @polyengine/protocol"). -# Runs under the PINNED Node (tools/shell/pins.json), like test-sockets-node. +# Cross-package imports must stay dependencies, not inlined copies. +# Test npm package exports, declarations and guest execution on pinned Node. test-npm: npm-build fixtures deno run -A tools/shell/fetch.ts node-pinned .shell-cache/node-pinned/bin/node tools/npm-build/smoke.mjs - # Stamp-path leg: a throwaway build with `--version` proves the - # prerelease stamp (release.yml's pre- path) hits the - # lockstep four exactly while leaving protocol on its own manifest - # version (contracts/embedder-api.md §"Version canonicalization"). No packing/install — fast. Output goes under - # .shell-cache, never the repo's npm/ dir, and is removed after. + # Verify --version stamps only the lockstep four, not protocol; no install. rm -rf .shell-cache/npm-stamp-check deno run -A tools/npm-build/build.ts --version 9.9.9-pre.gtest --out .shell-cache/npm-stamp-check deno run -A tools/npm-build/stamp_check.ts .shell-cache/npm-stamp-check 9.9.9-pre.gtest @@ -249,14 +160,8 @@ shell-lane lane *args: shim corpus deno run -A tools/shell/fetch.ts {{lane}} deno run -A tools/shell/run-lane.ts {{lane}} {{args}} -# JSC has no arm64 channel (jsc-built-products is x86_64-only), so its -# lane guards on the arch and skips cleanly elsewhere. node/bun publish -# both linux arches, so those lanes run everywhere; bun-pinned is -# findings-only (expectation carries `required: false` — deviations print -# and exit 0, only infrastructure failures gate) until it has a track -# record, then promote. -# The per-push pinned shell gates: sm-pinned + node-pinned (+ bun-pinned, -# findings-only) everywhere; jsc-pinned on x64. +# Bun is findings-only (required: false); infrastructure failures still gate. +# Pinned shell lanes: SpiderMonkey/Node on both Linux arches, JSC on x64 only, plus Bun. shells: just shell-lane sm-pinned @if [ "$(uname -m)" = "x86_64" ]; then just shell-lane jsc-pinned; else echo "jsc-pinned: skipped (no arm64 channel)"; fi @@ -267,36 +172,22 @@ shells: deno-canary *args: deno run -A tools/shell/deno-canary.ts {{args}} -# The repo-local cache is what run-lane.ts expects -# (PLAYWRIGHT_BROWSERS_PATH=$PWD/.browser-cache — cache THAT path in CI, -# not ~/.cache/ms-playwright). CI passes --with-deps for system -# libraries; locally a plain `just browsers-install` usually suffices. -# One-time browser provisioning (chromium + firefox) into .browser-cache/. +# CI adds --with-deps for system libraries. +# Provision Chromium/Firefox in .browser-cache/, the lane driver's default. browsers-install *flags: PLAYWRIGHT_BROWSERS_PATH=$PWD/.browser-cache deno run -A npm:playwright@1.62.1 install {{flags}} chromium firefox -# WebKit stays non-blocking until it has a track record (issue #11): the -# lane's expectation overlay encodes JSC's missing multi-memory, and GH's -# ubuntu-24.04 matches the ABI playwright's WebKit wants (no library -# staging expected). +# Provision best-effort WebKit; its bundled Linux libraries target Ubuntu 24.04. browsers-install-webkit *flags: PLAYWRIGHT_BROWSERS_PATH=$PWD/.browser-cache deno run -A npm:playwright@1.62.1 install {{flags}} webkit -# chromium and firefox are required (chromium expects exact Deno-lane -# parity; the firefox driver sets the JSPI pref itself — shipped-channel -# config, unlike the jsshell); webkit is best-effort per -# docs/architecture.md §3/§12 (issue #11). -# One browser lane (chromium / firefox / webkit). +# Expectations: harness/browser/expectations/; launch.ts enables Firefox's JSPI pref. +# One browser lane: Chromium/Firefox required, WebKit best-effort. browser-lane lane *args: shim corpus deno run -A tools/browser/run-lane.ts {{lane}} {{args}} -# The post-merge browser gates. The worker / shared-worker rows (issue -# #129) run the SAME corpus / battery inside those realms, judged against -# the SAME per-engine expectations: any delta at all is a realm leak (a -# Window-only dependency creeping into the runtime), and the failure names -# the realm. The first consumer topology (polyvisor G5) hosts the engine -# in a shared worker, so those rows gate here rather than surfacing as a -# consumer-side mystery. +# All realms use the same per-engine expectations to detect realm-specific failures. +# Browser conformance and OPFS gates across page, worker and shared-worker realms. browsers: just browser-lane chromium just browser-lane chromium --realm worker @@ -311,23 +202,14 @@ browsers: just smoke-opfs firefox --realm worker just smoke-opfs firefox --realm shared-worker -# filesystem-web against the REAL Origin Private File System (the unit -# suite runs an in-memory fake — Deno has no navigator.storage): the -# direct descriptor battery plus the fs-probe guest parking through -# the WASI parking kernel (contracts/embedder-api.md §"The WASI parking -# kernel") over real async storage. tools/browser/opfs-smoke.ts. -# `--realm worker|shared-worker` (issue #129) runs the battery inside that -# realm — the OPFS × JSPI-parking × worker-realm intersection the first -# consumer (polyvisor G5) actually ships. +# --realm worker|shared-worker selects a worker instead of the default page. +# Real OPFS: descriptor tests and fs-probe guest I/O through the JSPI parking kernel. smoke-opfs lane *args: shim fixtures deno run -A tools/browser/opfs-smoke.ts {{lane}} {{args}} # ----- consumer smokes + exams (polymorph checkouts; docs/consumers.md) ------- -# Translate all eight targets, then execute the suites. -# polymorph-tls conformance under polyengine (issue #18). -# (--allow-env: tools/smoke-c0/common.ts reads POLYMORPH_ROOT at module -# scope since the wosh rename; the leg tasks always had it via deno task.) +# Translate and execute polymorph-tls suites from read-only consumer artifacts. smoke-tls: shim deno run --allow-read --allow-env=POLYMORPH_ROOT,WOSH_ROOT tools/smoke-tls/run.ts --exec @@ -335,12 +217,8 @@ smoke-tls: shim smoke-c0: shim cd tools/smoke-c0 && deno task leg1 && deno task leg2 && deno task leg3 && deno task leg4 -# The host-boundary microbench (bench/boundary/README.md): calls/sec per -# ABI shape for the CURRENT tree, on plain node (callback + jspi) and -# deno. Manual instrument, not a gate — numbers are box-relative; the -# committed README carries the baseline and the issues it feeds (#8, -# #54; #17's record). `just bench-boundary with-jco` adds the incumbent -# jco lane (npm tree + transpile, prepared on first use). +# `with-jco` prepares and adds the jco comparison (bench/boundary/README.md). +# Manual host-boundary benchmark, not a gate; results are machine-relative. bench-boundary *jco: shim #!/usr/bin/env bash set -euo pipefail @@ -362,16 +240,7 @@ bench-boundary *jco: shim # ----- release ---------------------------------------------------------------- -# The standard shim (what every suite runs against), the size-tuned -# variant (flags per crates/translator-shim/README.md — reproduces the -# published size figures without editing the workspace manifest), the -# embedder bundle, and SHA256SUMS — all written to the repo root. NOTE: -# the size-tuned build leaves the MIN shim in target/; rerun `just shim` -# before running test suites locally afterwards. -# The release artifacts, exactly as the release workflow publishes them. -# Since the shim recipe adopted the size tuning (#58), the tuned build IS -# the artifact every suite runs against, and the former separate "min" -# variant is redundant — one shim, tested and shipped identically. +# Release artifacts in the repo root: the tested shim, embedder bundle and SHA256SUMS. release-artifacts: shim cp target/wasm32-unknown-unknown/release/translator_shim.wasm polyengine-translator-shim.wasm deno run -A tools/release-bundle/build.ts --out polyengine-embedder.mjs diff --git a/runtime/README.md b/runtime/README.md index 0b8c6ac..76260a0 100644 --- a/runtime/README.md +++ b/runtime/README.md @@ -1,155 +1,88 @@ -# runtime — TS core runtime +# @polyengine/runtime -Platform-neutral TypeScript runtime core (docs/architecture.md §4.3). Two layers -exist today: +TypeScript runtime for WebAssembly components. It loads the translator's plan, +instantiates core modules and FACT adapters, and implements the canonical ABI, +resource lifetimes, task scheduling, cancellation, streams, futures, and +error-contexts. Callback-ABI async execution and JSPI-backed suspension are +implemented. -1. the **canonical-ABI v1 value interpreter** (`src/cabi/`) with the - **definitions.py test ports** (`tests/`): docs/architecture.md §11 row 2, - feeding §7 (canonical-ABI decisions) and §8 (the descriptor-IR interpreter); -2. the **plan executor on the task-model skeleton** (`src/plan/`, `src/task/`, - `src/exec/`, `src/intrinsics/`, `src/shim/`): loads the translator shim's - plan v0 (contracts/plan-format.md), instantiates and links the component, and - routes every lifted-export call through Task/Thread structures (degenerate - sync path, docs/architecture.md §6). +## Embed a component -## Layout +```ts +import { instantiate } from "@polyengine/runtime/embedder"; +import { defaultTranslator } from "@polyengine/translator"; +const translator = await defaultTranslator(); +const componentBytes = await Deno.readFile("hello.component.wasm"); +const component = await instantiate({ componentBytes, translator }, {}); +console.log(await component.exports.greet("component model")); ``` -src/cabi/ v1 reference interpreter, mirroring definitions.py sections - types.ts component value-type model (descriptor-IR in-memory form) - trap.ts Trap vs AssertionError vs NotImplemented - memory.ts MemInst (DataView over Uint8Array), int load/store, i64-safe indices - layout.ts alignment / elem_size / align_to - float.ts NaN canonicalization + f32/f64 bit reinterpretation - context.ts canonical options, LiftLowerContext, borrow-scope interfaces - handles.ts Table, ResourceHandle, own/borrow lift/lower, resource.{new,rep,drop} - strings.ts load_string*/store_string* (all three encodings), char checks - load.ts store.ts memory <-> value dispatchers - lift.ts lower.ts flat core values <-> value dispatchers (CoreValueIter, coercions) - values.ts lift_flat_values / lower_flat_values (max_flat spilling) -src/plan/ plan v0 wire types (format.ts) + loader/validator (loader.ts) -src/task/ Task / Thread / Subtask / ComponentInstanceState + sync - driving loop (the scheduler's spine; docs/architecture.md §6) -src/exec/ plan executor (executor.ts: initializers, CoreDef/arg - resolution, export surface) + host boundary (boundary.ts: - canon_lift/canon_lower sync paths, LiveMemory grow-safe views) -src/intrinsics/ host trampolines (contracts/intrinsics.md): core subset - implemented, milestone-aware instantiate-time failures -src/shim/ client for the translator shim's wasm32 C-ABI + envelope -tests/ deno test ports (table-driven; fixtures where generated) -tests/fixtures/ generate.py + JSON generated by running definitions.py -tests/support/ Heap mock (run_tests.py twin), type DSL parser, test() driver -tests/integration/ full-pipeline e2e: shim wasm32 under Deno -> plan -> - executor -> typed calls (hello gate + values/resources - stretch; needs `cargo build -p translator-shim --release - --target wasm32-unknown-unknown` and `./examples/build.sh`) + +This uses the guest from [hello-world](../examples/hello-world/). The +application must resolve the package specifiers through its import map or +package manager. For imports, resources, and stream/future values, start with +[kitchen-sink](../examples/kitchen-sink/). Known components can be +[translated at build time](../tools/translate/) so deployment needs no +translator. + +The [embedder contract](../contracts/embedder-api.md) defines the host-facing +names, value shapes, errors, ownership, and calling conventions. Host-provider +packages use `@polyengine/protocol` for shared types and brands; runtime +selection belongs to the application. Internal `cabi` and `exec` modules are not +package entry points. + +## Package entry points + +| Import | Purpose | +| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `@polyengine/runtime/embedder` | Instantiate components and construct the host-facing facade; inspect required imports; create stream pairs; obtain explicit synchronous export views | +| `@polyengine/runtime/shim` | Load a translator shim and translate component bytes | +| `@polyengine/runtime/plan` | Read and validate plans and translation envelopes | +| `@polyengine/runtime/cache` | Opt-in translation artifact caching with directory and web backends | +| `@polyengine/runtime/digest` | World-digest support for generated binding checks | + +The digest entry point is unstable support for generated code, not a +hand-written host API; regenerate bindings when updating the runtime. + +The loader accepts only `formatVersion: 5`. Translate with a matching toolchain +rather than reusing an older plan. See the +[plan contract](../contracts/plan-format.md) for the wire format and executor +obligations, and [architecture](../docs/architecture.md) for engine support, +JSPI selection, and semantic scope. + +## Source map + +| Directory | Responsibility | +| ---------------------------------------- | ---------------------------------------------------------------------------- | +| `src/embedder/` | Host conventions, import resolution, resource classes, stream/future handles | +| `src/cabi/` | Value layout, flat and memory lift/lower, strings, handle tables | +| `src/plan/` | Wire types, structural validation, descriptor loading | +| `src/exec/` | Instantiation, canonical call boundaries, host stream endpoints | +| `src/task/` | Tasks, threads, waitables, admission, cancellation, copy protocol | +| `src/jspi/` | Engine suspension and continuation bridging | +| `src/intrinsics/` | Canonical builtins and FACT adapter imports | +| `src/shim/`, `src/cache/`, `src/digest/` | Translation client, artifact caching, binding identity | + +## Development + +From the repository root: + +```sh +just test-runtime # builds shim, guest fixtures, and WAST corpus; checks and tests +just test-conventions # focused host-ABI golden transcripts +just sched-seeds # reruns affected suites with shuffled scheduling ``` -Run tests: `deno task test` (from `runtime/`; it is `deno test ---allow-read=..` -— the integration tests read build artifacts from the repo). Type-check: -`deno task check`. - -Regenerate fixtures: `deno task gen-fixtures` (needs `python3`; imports -`third_party/component-model/design/mvp/canonical-abi/definitions.py` -read-only). Fixtures are deterministic and checked in; regeneration is only -needed when the submodule's definitions.py changes. - -## What is ported (PORT-NOW: pure value/memory semantics) - -| Area | Source (run_tests.py) | Tests | -| ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | -| despecialization, alignment, elem_size | implicit throughout | `layout_flatten_test.ts` (fixture-driven, 57 types × i32/i64) | -| flatten_type / flatten_functype incl. async shapes, MAX_FLAT boundaries | `test_flatten` (+ async cases run_tests only reaches via canon_lower) | `layout_flatten_test.ts` (fixtures) | -| flat lift/lower: records, tuples, fixed lists, flags, variants, options, results, enum; variant-to-supertype lower | `test(...)` table, lines ~148–181 | `flat_test.ts` | -| numeric lane wrapping (u8..s64), bool, i64<->BigInt boundaries | `test_pairs` | `flat_test.ts` | -| variant payload lane coercion (i32/i64/f32/f64 joins, zero-padding) | exercised indirectly | `flat_test.ts` (expectations produced by running definitions.py) | -| NaN canonicalization (deterministic profile) | `test_nan32/64` | `nan_test.ts` | -| char validation (surrogates, > 0x10FFFF trap) | `test_pairs(CharType...)` | `flat_test.ts` | -| strings: full utf8/utf16/latin1+utf16 matrix, both address types, byte-exact incl. realloc traffic | `test_string` matrix | `string_test.ts` (fixture-driven: 168 lift + 150 lower byte-exact checks + 504 roundtrips) | -| USVString lone-surrogate replacement (docs/architecture.md §7) | n/a in Python (strings always well-formed) | `string_test.ts` (TS-authored) | -| lists/records/variants/flags/map over heap memory, misalignment traps, i64 memories | `test_heap` | `heap_test.ts` (35 cases) | -| lift/lower_flat_values spilling (17 params, retp out-param, alignment traps) | reached via `test_roundtrips`/`canon_lower` upstream | `values_test.ts` (TS-authored) | -| handle Table (slab, free list LIFO, traps), resource.new/rep/drop, own transfer, borrow lend counting | `test_handles` (pure parts) | `handles_test.ts` | - -## What is deferred (needs task/thread/waitable machinery) - -Everything from `test_roundtrips` down that drives `canon_lift`/`canon_lower` -through `Store`/`Task`/`Thread`: cross-component realloc, the full -`test_handles` (borrow lifetimes enforced at task exit), all async tests -(callbacks, backpressure, waitable sets), streams/futures (copy state machines), -cancellation, `thread.*`/`context.*` built-ins, and the -error-context/stream/future _value types_ (their layout/flatten is implemented; -their lift/lower throws `NotImplemented`). Each has an ignored placeholder in -`tests/deferred_test.ts` with the reason. Per docs/architecture.md §6 the -scheduler is the core deliverable and gets built as the runtime's spine — these -ports become its acceptance tests, not the other way round. - -## Decisions forced by JS semantics (not already settled by docs/architecture.md §7) - -Recorded here because definitions.py had no opinion (Python erases these -distinctions); flagged for plan review: - -1. **Core lane typing**: i32/f32/f64 lanes are `number`, i64 lanes are `bigint`, - enforced by `CoreValueIter`. Consequences ported carefully: variant lane - widening (`i32 -> i64` payload coercion) must convert number→BigInt, and - variant zero-padding pads `0n` for i64 lanes. -2. **Deterministic NaN profile only**: NaN payloads are not reliably - observable/preservable through JS numbers, so `DETERMINISTIC_PROFILE=False` - (NaN scrambling) is unimplementable; we canonicalize on lift _and_ write - canonical NaN bits explicitly on store/lower (never trusting engine NaN - propagation through DataView). -3. **No string encoding provenance** ("provenance-free lowering"): lifted - strings are plain JS strings, not definitions.py's - `(str, encoding, code_units)` tuple. Lowering always follows the reference's - `src_encoding='utf16'` paths because a JS string _is_ a UTF-16 code-unit - sequence. Observable effect vs the reference: same final bytes for utf8/utf16 - destinations, but different realloc traffic than the same-encoding copy fast - path, and the `store_probably_utf16_to_latin1_or_utf16` path is unreachable. - Rationale: cross-component transcoding (where provenance pays off) belongs to - FACT adapters (docs/architecture.md §4.1), not the host boundary. -4. **latin1 decode is hand-rolled**: WHATWG `TextDecoder`'s - "latin1"/"iso-8859-1" labels alias windows-1252 (0x80–0x9F differ); - ISO-8859-1 lift is a manual identity byte→code-point loop. -5. **Addresses/lengths are JS numbers**; i64-lane values (memory64) are - bounds-checked in BigInt before conversion, trapping before any precision - loss could matter. -6. **`String.prototype.toWellFormed()`** (ES2024) implements the USVString - conversion; present in all JSPI-capable engines (the compatibility floor). -7. **Variant/record/flags value shapes** mirror definitions.py's semantics - (variants as `{kind, value}` objects — contracts/descriptor-ir.md §"Host - value shapes" — despecialized tuple records, label→bool maps); `list` is - `Uint8Array` per docs/architecture.md §7. Final host-facing representations - for bindgen remain open (below). - -Also plan-relevant: docs/architecture.md §7 defers **latin1+utf16** "until a -test forces it" — the ported definitions.py string matrix forces it, so the v1 -interpreter now implements it fully (both directions). If we prefer to keep the -runtime surface minimal, the store path can be re-deferred by ignoring the -fixture subset again. - -## Open questions (types.ts is provisional) - -- Wire format of the descriptor IR: `types.ts` is the in-memory sketch; the - translator shim (docs/architecture.md §4.2) will define the serialized form - and likely intern labels/types by index. -- Host-facing value representations for bindings (docs/architecture.md §9): - tuples as arrays? variants as `{ tag, val }`? `option` as `T | undefined` - with a `Some`/`None` escape hatch for nesting? The interpreter's despecialized - shapes are faithful to the reference but not ergonomic. -- Whether host-provided `char`/integer values get validated (trap) or asserted - at the boundary — definitions.py asserts (guest-side values are trusted by - construction); a host API needs a decision. -- `FuncType` here drops parameter/result names (ABI-irrelevant); confirm the - plan format does the same. - -## Upstream discrepancies found (definitions.py vs CanonicalABI.md) - -- `canon_backpressure_set` existed in definitions.py but CanonicalABI.md - documents only `backpressure.inc`/`backpressure.dec`; the repo's own `diff.py` - flagged it. Vestigial back-compat shim — resolved upstream independently (CM - PR #690 removed it; findings tracker CM-2). The stale `$async?` immediate on - `resource.drop` (CanonicalABI.md ~line 4013, noted in docs/architecture.md §7) - is still open. -- `python3 run_tests.py` passes upstream unmodified (Python 3.13.7), so no - reference-snapshot copy was needed. +From `runtime/`, `deno task check` type-checks source and tests, and +`deno task test` runs against existing artifacts. Missing artifacts can cause +integration tests to skip; use the root recipe for the complete setup. +`deno task gen-fixtures` regenerates the checked-in pure-value fixtures from the +pinned `definitions.py` using Python. Regenerate only when intentionally +updating those expectations. + +Tests include reference value ports, scheduler and JSPI regressions, generated +guest integration, and [host-ABI conventions](tests/conventions/). The remaining +reference-port placeholders are in [deferred_test.ts](tests/deferred_test.ts); +they are not an inventory of unimplemented async behavior. Read the +[harness documentation](../harness/) for xfails, skips, and the limits of a +green conformance run. diff --git a/runtime/src/cabi/async_values.ts b/runtime/src/cabi/async_values.ts index 1d4d9df..c43219e 100644 --- a/runtime/src/cabi/async_values.ts +++ b/runtime/src/cabi/async_values.ts @@ -1,26 +1,11 @@ // Lift and lower for the async value types: `stream`, `future` and -// `error-context` (definitions.py `lift_async_value` line 1530, -// `lower_stream`/`lower_future` line 1828, `lift_error_context` line 1451, -// `lower_error_context` line 1757). +// `error-context` (definitions.py `lift_async_value`, `lower_stream`, +// `lower_future`, `lift_error_context`, `lower_error_context`). // -// =========================================================================== -// HOST SHAPE (interpreter decision — flagged, per contracts/descriptor-ir.md) -// =========================================================================== -// -// Every other `ComponentValue` in this interpreter is plain data: numbers, -// strings, arrays, records. The async value types cannot be, and the reference -// is explicit about why: `lift_async_value` returns `e.shared` — the *shared* -// stream/future object, not a copy and not an index — and `lower_stream` wraps -// that same object in a fresh `ReadableStreamEnd` in the destination -// instance's handle table. The identity of the shared object is the value; two -// components holding ends of one stream must observe each other's copies. -// -// So a lifted `stream`/`future` is a `SharedStreamImpl`/`SharedFutureImpl` -// instance, and a lifted `error-context` is an `ErrorContext` instance. Host -// code that receives one should treat it as an opaque token. This is the same -// concession the reference makes; it is recorded here because it widens what -// `ComponentValue` can hold beyond plain data, which the descriptor-IR -// contract's "host-shaped component values" wording did not anticipate. +// Raw values are shared object identities, not copies or handle indices: +// SharedStreamImpl, SharedFutureImpl, and ErrorContext. Lowering a stream or +// future wraps that identity in a fresh readable end in the destination table. +// The public embedder layer supplies facades over these internal tokens. import { copyCensus, ERROR_CONTEXT, hasBrand } from "@polyengine/protocol"; import { assert_, trapIf } from "./trap.ts"; @@ -44,11 +29,8 @@ import { removeHandleWithUnwind } from "../task/scheduler.ts"; * without being one of THIS copy's `ErrorContext`s (contracts/embedder-api.md * §"Module identity"). * - * A backstop, deliberately: the embedder's lowering site (embedder/values.ts) - * refuses a foreign error-context before it can ever reach a handle table, so - * this branch should be unreachable. It exists because "handle is not an - * error-context" is precisely the misleading generic that made #83 expensive - * to diagnose — if a path ever does get here, it says what happened. + * The embedder lowering site also checks identity; this backstop gives raw + * paths the same specific diagnostic rather than a generic handle error. * * This layer is below `embedder/`, so it composes the census from * `@polyengine/protocol` directly rather than importing `embedder/copy.ts`. @@ -70,10 +52,11 @@ function containsBorrow(t: ValType): boolean { } /** - * definitions.py `lift_async_value` (line 1530). + * definitions.py `lift_async_value`. * * Lifting **removes** the handle: the readable end is transferred out of this - * instance's table, which is why a stream can only be passed on once. + * instance's table. That handle cannot be used again; the shared value can + * later return through another lowering/lifting transfer. */ function liftAsyncValue( cx: LiftLowerContext, @@ -110,18 +93,13 @@ function liftAsyncValue( `cannot lift ${what} while it's in a waitable set`, ); // Remember the driving store so a host wrapper can pump the guest later. - // Single-store only: a shared object crossing into a SECOND store is - // unsupported misuse — fail loudly rather than silently pumping the first - // (review advisory, host-streams round). Class field initializes to null; - // != null covers both sentinels. + // A shared object cannot be driven by two stores. The nullish check also + // admits structural test doubles whose store field is absent. const holder = end.shared as { boundStore?: unknown }; const store = (inst as unknown as { store?: unknown }).store; if (holder.boundStore != null && store != null) { - // module identity: when several runtime copies are loaded, "a second store" is very - // often "a second COPY" — the shared object was minted by one runtime and - // is being driven by another. The two stores are indistinguishable from - // here (stores carry no copy identity), so the census is appended as the - // hypothesis it is, rather than asserted (issue #83). + // Stores carry no runtime-copy identity, so the census is diagnostic + // context, not proof that this is a cross-copy mismatch. const census = copyCensus(); assert_( holder.boundStore === store, @@ -133,7 +111,7 @@ function liftAsyncValue( ); } holder.boundStore ??= store; - // Host-wrapper re-arm hook (#162, contracts/embedder-api.md §"Streams and futures"): the readable + // Host-wrapper re-arm hook (contracts/embedder-api.md §"Streams and futures"): the readable // end just left a guest table, so whoever receives it can act on it again. // See `bindOnLower` in exec/host_streams.ts for the retention rule. (end.shared as { onLifted?: ((i: unknown) => void) | null }).onLifted?.( @@ -159,7 +137,7 @@ export function liftFuture( return liftAsyncValue(cx, i, t, ReadableFutureEnd, t.element, "future"); } -/** definitions.py `lower_stream` (line 1828). */ +/** definitions.py `lower_stream`. */ export function lowerStream( cx: LiftLowerContext, v: SharedBase, @@ -170,14 +148,8 @@ export function lowerStream( "lower_stream expects a shared stream value", ); assert_(!containsBorrow(t), "stream may not contain a borrow"); - // Loud element-type check. A host-created stream carries a hand-passed - // `ValType` (typed derivation is bindgen's job), so this is the first point - // at which a mismatch against the guest's declared `stream` can be - // caught — and a silent mismatch would corrupt every copy, since the - // element type is what sizes and lifts the buffer. `fmtValType`, not - // `JSON.stringify`: the latter throws on resource-bearing element types - // (cabi/types.ts `valTypeEqual` contract note) — and as a template-literal - // argument it was evaluated even when the assertion PASSED. + // Host-precondition check: element types determine buffer size and lifting. + // Diagnostics must use fmtValType, not serialize resource identity cycles. const declared = (t as { element?: ValType | null }).element ?? null; if (!sameElemType(v.t, declared)) { assert_( @@ -194,7 +166,7 @@ export function lowerStream( return inst!.handles.add(new ReadableStreamEnd(v)); } -/** definitions.py `lower_future` (line 1833). */ +/** definitions.py `lower_future`. */ export function lowerFuture( cx: LiftLowerContext, v: SharedBase, @@ -221,7 +193,7 @@ export function lowerFuture( return inst!.handles.add(new ReadableFutureEnd(v)); } -/** definitions.py `lift_error_context` (line 1451). Does NOT remove the handle. */ +/** definitions.py `lift_error_context`. Does not remove the handle. */ export function liftErrorContext( cx: LiftLowerContext, i: number, @@ -236,7 +208,7 @@ export function liftErrorContext( return e as ErrorContext; } -/** definitions.py `lower_error_context` (line 1757). */ +/** definitions.py `lower_error_context`. */ export function lowerErrorContext( cx: LiftLowerContext, v: ErrorContext, diff --git a/runtime/src/cabi/bulk_lists.ts b/runtime/src/cabi/bulk_lists.ts index 07e8905..eb4611a 100644 --- a/runtime/src/cabi/bulk_lists.ts +++ b/runtime/src/cabi/bulk_lists.ts @@ -1,53 +1,22 @@ -// Bulk (TypedArray-backed) list copies for flat element types (issue #67). +// TypedArray-backed list copies for flat element types. // -// The per-element interpreted `load()`/`store()` costs ~13-45 ns/element -// (despecialize + asserts + DataView per element); these helpers replace the -// loop body with one typed-array view per list and a tight per-element pass -// that preserves the interpreted path's EXACT observable semantics: -// -// * integers: the same `assert_` type-shape texts as `storeInt` (`"int -// store"`, `"64-bit store requires bigint"`) but, unlike the scalar path -// in memory.ts (`storeInt`'s range `assert_`s, issue #96), NOT the same -// range check: this bulk path wraps out-of-range values instead of -// raising the host-precondition error (`OverflowError` per -// definitions.py:1568-1569 `int.to_bytes`) that `storeInt` raises. That -// is a deliberate scalar/bulk posture split, not an oversight: -// - the whole point of this file (see the perf numbers above) is an -// allocation-free, branch-minimal per-element loop; an added range -// check is itself a per-element cost, defeating the purpose; -// - values reaching this path from a descriptor-driven lower already -// went through the descriptor layer's own type conversions for the -// cases that matter in practice (see contracts/descriptor-ir.md); -// the wrap here is a defense-in-depth gap only for a raw/buggy -// embedder value, which the scalar path (used for non-bulk-eligible -// kinds, and reachable directly from embedder code) still catches. -// A TypedArray element write coerces exactly like the matching DataView -// setter (wraps mod 2^width), so this is pinned as intentional behavior -// (see bulk_list_test.ts), not merely undocumented; -// * floats: the deterministic profile's NaN canonicalization on BOTH -// directions (float.ts `decodeI32AsFloat` / `encodeFloatAsI32`): every -// lifted NaN becomes the JS canonical NaN, every stored `number` NaN -// writes the canonical bit pattern. Non-NaN values round-trip bit-exactly -// (Float32Array narrowing is the same IEEE round-to-nearest-even as -// `DataView.setFloat32`); -// * bool: store normalizes any value by truthiness to 0/1 (`store()`'s -// `Number(Boolean(v))`), lift maps any nonzero byte to `true` -// (`convertIntToBool` semantics — it never traps for unsigned bytes). +// Integers assert the same host type/shape preconditions as storeInt, but wrap +// out-of-range values modulo the element width rather than asserting range. +// This is a scalar/bulk host-precondition difference, not exact equivalence +// for arbitrary raw inputs (contracts/descriptor-ir.md; bulk_list_test.ts). +// Floats canonicalize NaNs in both directions and use the same narrowing as +// DataView stores. Bool stores use truthiness; loads accept any nonzero byte. // // u8 is NOT here: it has its own, shape-changing fast path (`list` is // `Uint8Array` on the host — load.ts/store.ts). char is NOT here: its lift // validates USVs per element (`convertI32ToChar` traps), which is the cost. // -// NAMED ASSUMPTION (issue #67): wasm linear memory is little-endian by spec; -// JS TypedArrays follow the PLATFORM's endianness. Every engine polyengine -// targets runs little-endian, but rather than bake that in silently, the -// check below gates the fast paths — on a big-endian platform they decline -// and the callers keep the (endianness-correct) DataView per-element loops. +// Wasm memory is little-endian; JS TypedArrays use platform endianness. +// On a big-endian platform numeric views decline and callers use DataView. // // Alignment: the canonical ABI guarantees list pointers are element-aligned, -// and real guest memories sit at byteOffset 0, so the view construction -// below virtually never declines; a misaligned combination (possible for a -// test MemInst wrapping a subarray) falls back the same way. +// but a MemInst subarray's backing byteOffset may be misaligned. Those views +// decline too. The bool byte path needs neither endianness nor alignment. import { assert_ } from "./trap.ts"; import { bytesOf, type MemInst } from "./memory.ts"; @@ -122,7 +91,6 @@ export function tryLoadNumericList( if (intCtor !== undefined) { const view = viewOf(intCtor, mem, ptr, length); if (view === null) return null; - // Manual preallocated loop: measurably faster than Array.from(view). const out = new Array(length); for (let i = 0; i < length; i++) out[i] = view[i]; return out; diff --git a/runtime/src/cabi/context.ts b/runtime/src/cabi/context.ts index 9dad3ae..f82f195 100644 --- a/runtime/src/cabi/context.ts +++ b/runtime/src/cabi/context.ts @@ -45,7 +45,7 @@ export function mkCanonicalOptions( }; } -/** Memory accessor that traps-or-asserts like `cx.opts.memory` derefs. */ +/** Assert the caller supplied the memory option required by this operation. */ export function requireMemory(opts: LiftOptions): MemInst { assert_(opts.memory !== null, "canonical option `memory` required"); return opts.memory; @@ -53,8 +53,7 @@ export function requireMemory(opts: LiftOptions): MemInst { /** * Minimal component-instance stand-in for the value interpreter: a handle - * table plus the `may_leave` gate. The full ComponentInstance (backpressure, - * threads, ...) belongs to the task machinery, which cabi must not import. + * table plus the `may_leave` gate. Scheduling state belongs to the task layer. */ export interface ComponentInstanceLike { handles: Table; @@ -62,14 +61,10 @@ export interface ComponentInstanceLike { } /** - * Brand marking a value as a REAL component instance (task/mod.ts - * `ComponentInstanceState`), as opposed to the many structural - * `ComponentInstanceLike` stand-ins — imported/host resources carry no - * instance at all, and test harnesses supply bare `{handles, mayLeave}` - * doubles. cabi must not depend on task/, so the symbol lives here and - * `ComponentInstanceState` declares it; cabi/handles.ts `isComponentInstance` - * is the only reader. `ComponentInstanceLike` stays deliberately structural: - * the brand is NOT part of it. + * Distinguishes `ComponentInstanceState` from structural test doubles. + * Imported host resources have no implementing instance. `callDtorGated` + * uses this brand to decide whether to enter a real task/lift harness; + * `ComponentInstanceLike` deliberately does not require it. */ export const COMPONENT_INSTANCE: unique symbol = Symbol( "polyengine.ComponentInstance", @@ -79,8 +74,7 @@ export const COMPONENT_INSTANCE: unique symbol = Symbol( * Borrow scopes (definitions.py `LiftLowerContext.borrow_scope`): * - lifting a borrow requires the *subtask* side: `add_lender`. * - lowering a borrow requires the *task* side: `num_borrows`. - * The real Task/Subtask classes are deferred; these are the minimal - * interfaces the value code needs. + * These structural interfaces also admit FACT call scopes. */ export interface SubtaskBorrowScope { addLender(h: import("./handles.ts").ResourceHandle): void; @@ -101,9 +95,8 @@ export class LiftLowerContext { * definitions.py `LiftLowerContext.reallocate`: the guest's realloc runs * with `may_leave` cleared, so a realloc that lowers an import traps * (`canon_lower`'s `trap_if(not ...may_leave)`, implemented here by - * exec/boundary.ts `createLoweredImport`). That bracket is implemented - * below, and it is the whole story: the pinned reference does not route - * the call through `canon_lift` (CM#705). polyengine issue #147. + * exec/boundary.ts `createLoweredImport`). Unlike a resource destructor, + * realloc does not enter a fresh `canon_lift` task. */ reallocate( old: number, @@ -121,10 +114,9 @@ export class LiftLowerContext { } assert_(this.inst.mayLeave, "realloc with may_leave already false"); this.inst.mayLeave = false; - // NO try/finally, deliberately: same bare bracket as the post-return one - // in intrinsics/fact_calls.ts (#91) and the reference's `assert`/restore - // pair. A trapping realloc skips the restore exactly as the reference - // does; the host-boundary unwind and instance poisoning handle the rest. + // No local finally: a trap skips the reference's restore. Host-boundary + // unwind may restore sibling flags; see createLiftedFunction's + // entry-identity rule in exec/boundary.ts. const ptr = realloc!(old, oldByteLength, alignment, newByteLength); this.inst.mayLeave = true; return ptr; diff --git a/runtime/src/cabi/flatten.ts b/runtime/src/cabi/flatten.ts index b251b9e..c246bec 100644 --- a/runtime/src/cabi/flatten.ts +++ b/runtime/src/cabi/flatten.ts @@ -76,26 +76,11 @@ export function flattenTypes(ts: ValType[], opts: LiftOptions): CoreType[] { /** * `flattenTypes(ts, opts).length`, memoized on (ts array identity, ptrType). * - * Every arm of `flattenType` reaches `opts` only through - * `requireMemory(opts).ptrType()` (string/list-without-length read the - * pointer width; every other arm is opts-free or recurses structurally), so - * the flattened element count is a pure function of `(ts, ptrType)` — one - * map per pointer width is the whole of the cache key. `ts` is always a - * plan-owned `ft.params`/`ft.results` array, the same stability argument - * `spillTupleType` (values.ts) relies on, so its identity is a valid key. - * - * This caches the COUNT, not the flattened array: `values.ts`'s only use of - * `flattenTypes` on the per-call path is `.length`, and a cached number has - * no aliasing/mutation hazard to guard (a cached array would need freezing - * plus a lossy `readonly`-to-mutable cast at every read site — the shape - * this replaced). Callers that need the actual flat types (instantiate-time - * `flattenFunctype`) still call `flattenTypes` directly, uncached; that path - * runs once per function, not once per call, so it doesn't need this. - * - * The null-memory path is deliberately NOT cached: `requireMemory` throws - * when `opts.memory` is null (for any `ts` containing a string/unbounded - * list), and that throw must still surface on every call, not just the - * first. + * `flattenType` reads options only for pointer width. Inputs and all reachable + * types must remain immutable, as for plan-owned ft.params/ft.results. + * Only the count is shared; callers of flattenTypes receive fresh arrays. + * Null-memory calls remain uncached so requireMemory still rejects types + * containing strings or variable-length lists, even after a memory-backed hit. */ const flatCountCacheByPtrType = { i32: new WeakMap(), diff --git a/runtime/src/cabi/float.ts b/runtime/src/cabi/float.ts index 31da8db..ab85ce6 100644 --- a/runtime/src/cabi/float.ts +++ b/runtime/src/cabi/float.ts @@ -1,12 +1,9 @@ // Float bit handling (definitions.py NaN canonicalization and // reinterpretation helpers). // -// The reference has a DETERMINISTIC_PROFILE toggle; this port implements only -// the deterministic profile (NaNs canonicalized on lift and on -// store/lower). JS forces this: NaN payloads are not reliably observable or -// preservable through JS numbers, so the "scramble" branch of the reference -// is not implementable — and run_tests.py itself runs the deterministic -// profile. Recorded as a decision in runtime/README.md. +// Use the reference's deterministic NaN policy on lift and store/lower. +// JS numbers do not reliably preserve NaN payloads, so raw payload identity +// is not part of this value boundary's contract. export const CANONICAL_FLOAT32_NAN = 0x7fc00000; export const CANONICAL_FLOAT64_NAN = 0x7ff8000000000000n; diff --git a/runtime/src/cabi/handles.ts b/runtime/src/cabi/handles.ts index 19409d6..0cd8a92 100644 --- a/runtime/src/cabi/handles.ts +++ b/runtime/src/cabi/handles.ts @@ -2,14 +2,9 @@ // `### Resource State`, `canon resource.{new,drop,rep}`, and the // own/borrow lift/lower functions). // -// The Table and ResourceHandle mechanics are pure and ported fully. What is -// simplified here (pending the task machinery): -// - canon_resource_* take the instance explicitly instead of reading -// current_instance() from the running thread; -// - canon_resource_drop routes the dtor through `callDtorGated` below, -// which uses the reference's fresh synchronous lift task/thread. -// Host-initiated drops do NOT come here: they run the dtor through the -// real lift harness (`hostDtorCall`, exec/boundary.ts) — see #160. +// canon_resource_* take the declared instance explicitly. Guest drops use +// `callDtorGated` and a fresh synchronous lift task/thread; host drops use +// `hostDtorCall` (exec/boundary.ts) with host completion policy. import { assert_, trapIf } from "./trap.ts"; import { removeHandleWithUnwind } from "../task/scheduler.ts"; @@ -170,18 +165,8 @@ interface RealComponentInstance { } /** - * Is `x` a REAL component instance (`task/mod.ts` `ComponentInstanceState`), - * as opposed to something that has no instance behind it at all? - * - * Two populations must answer false, and both are load-bearing for - * `callDtorGated`: an imported/host-implemented resource, whose - * `ResourceTypeInfo.impl` is `null` by construction (exec/executor.ts - * `bindImportedResources`), and the bare `{handles, mayLeave}` doubles test - * harnesses supply — neither has an instance to refuse entry into or to - * poison. So this is deliberately NOT a structural match on - * `ComponentInstanceLike`, which those doubles satisfy: it reads the - * `COMPONENT_INSTANCE` brand, declared on `ComponentInstanceState` and - * defined in ./context.ts so that cabi does not have to import task/. + * Only branded component instances have task state and poisoning semantics. + * Host resource impl=null and structural test doubles must not enter that path. */ function isComponentInstance(x: unknown): RealComponentInstance | null { if (x === null || typeof x !== "object") return null; @@ -196,36 +181,17 @@ function isThenable(v: unknown): v is PromiseLike { } /** - * Invoke a resource destructor, as definitions.py `canon_resource_drop` - * (@ 2f13265) does — through `Store.lift`/`Store.lower`: - * - * ```python - * dtor = rt.dtor or (lambda rep: []) - * callee = inst.store.lift(dtor, ft, opts, rt.impl) - * caller = inst.store.lower(callee, ft, opts, inst) - * caller([h.rep]) - * ``` - * - * That lift carries NO gate (CM#705): dropping a handle whose implementing - * instance is mid-execution is VALID, including the dtor-less case. - * - * What this adds is polyengine's per-instance poisoning divergence, and it - * applies to `rt.impl`, not to the dropping instance: a trap out of the dtor - * buries the implementing instance (refusal names the original trap, - * polyengine#145; its live stream/future ends are retired, #66). The - * dropper is poisoned, if at all, by the same trap propagating at its own - * level. `entryRefusal`'s `caller !== callee` guard keeps a component - * dropping a handle to its OWN resource admissible even against a marked - * instance. + * Guest-initiated destructor call. `canon_resource_drop` lifts the dtor with + * a fresh synchronous task/thread, including a no-op dtor when absent. + * Reentrance into a live implementing instance is valid. * - * Capability signals (`NeedsJspi`, `PendingCapability`) are not traps — see - * `isCapabilitySignal` in exec/boundary.ts — so they do not poison. + * The lift harness applies runtime poisoning to `rt.impl` on a trap and + * retires its stream/future ends; capability signals do not poison. The + * dropper's trap propagation is handled separately. `entryRefusal` preserves + * the same-instance exemption for self-drops. * - * SCOPE (#160): this is the **guest-initiated** path only. A guest-initiated - * drop must complete synchronously (the reference lifts the dtor with - * `async_ = False`), so a thenable here is a trap. The host-initiated path - * uses the same lift harness with host completion policy (`hostDtorCall` in - * exec/boundary.ts). Guest entry uses only the reference synchronous drive. + * Guest entry uses the reference's synchronous drive, not host-wide async + * completion; a returned thenable traps. Host drops use `hostDtorCall`. */ export function callDtorGated( rt: ResourceTypeInfo, @@ -234,11 +200,7 @@ export function callDtorGated( ): void { const impl = isComponentInstance(rt.impl); const dtorFn = rt.dtor; - // No component instance behind the resource: an imported (host-implemented) - // resource has `impl === null` by construction (executor.ts - // `bindImportedResources`), so there is no instance to refuse entry into and - // none to poison. Test doubles that supply a bare `{handles, mayLeave}` - // instance land here too — see `isComponentInstance`. + // Host resources and structural test doubles have no task/poisoning state. if (impl === null) { const r = dtorFn?.(rep) as unknown; trapIf( @@ -247,9 +209,7 @@ export function callDtorGated( ); return; } - // The caller is only meaningful when it is a real component instance; a - // host-initiated drop passes null, which is the reference's `caller = None` - // (Store.invoke). It feeds `entryRefusal`'s `caller !== callee` guard below. + // Preserve a real guest caller's identity for the self-drop exemption. const callerInst = isComponentInstance(caller) === null ? null : caller; createDtorEntry({ @@ -272,13 +232,7 @@ export function canonResourceDrop( trapIf(rh.numLends !== 0, "handle still lent out"); if (rh.own) { assert_(rh.borrowScope === null); - // definitions.py line 2326-2333: the dtor runs through the store's - // lift/lower bracket. SCOPE NOTE (#85): the call below is a JS frame - // inside the drop trampoline, so a *guest*-initiated drop whose dtor - // suspends traps under the JSPI frame rule. That is deterministic and - // loud, and routing guest-initiated dtor calls through generated wasm is - // explicitly out of scope for #85 (docs/architecture.md §5/§7 carry the - // known-limitation note). + // Enter a fresh synchronous dtor task, not the dropping task's ambient. callDtorGated(rt, rh.rep, inst); } else { assert_(rh.borrowScope !== null); diff --git a/runtime/src/cabi/layout.ts b/runtime/src/cabi/layout.ts index 6156117..d65f869 100644 --- a/runtime/src/cabi/layout.ts +++ b/runtime/src/cabi/layout.ts @@ -19,12 +19,10 @@ export function alignTo(ptr: number, alignment: number): number { /** * Everything the lift/lower paths need to know about one type's byte layout, - * computed once and shared (issue #261). + * computed once and shared. * - * Deliberately ONE flat interface rather than a discriminated union of - * per-kind layouts: every `Layout` then has the same hidden class, so the hot - * property loads in `load`/`store` stay monomorphic. Fields that do not apply - * to a kind carry null/0 — please do not "improve" this into a union. + * One flat shape keeps property access uniform across kinds. Fields that do + * not apply carry null/0; consumers must interpret them using `d.kind`. */ export interface Layout { /** The despecialized type (definitions.py `despecialize`). */ @@ -42,11 +40,8 @@ export interface Layout { * Layout memo, keyed on type identity — see the despecialization memo in * types.ts for why identity is a sound key. * - * One map PER POINTER WIDTH, because layout is a function of both: `string`, - * variable-length `list` and the handle types all size off `ptrSize(ptrType)`. - * A single map keyed on the type alone would return i32 layouts to a memory64 - * instance — silently, and with no corpus coverage to catch it, which is why - * layout_cache_test.ts pins it explicitly. + * Separate maps per pointer width: string and variable-list descriptors + * contain pointer-sized fields. Handles remain four bytes in either mode. */ const layoutsI32 = new WeakMap(); const layoutsI64 = new WeakMap(); @@ -56,16 +51,12 @@ export function layoutOf(t: ValType, ptrType: PtrType): Layout { const hit = cache.get(t); if (hit !== undefined) return hit; const l = computeLayout(t, ptrType); - // Cached only after the computation returns, so a type that trips - // `elemSizeRecord`'s "empty record" assert keeps tripping it on every call - // instead of leaving a half-built node behind. + // Failed computations must not leave a cache entry. cache.set(t, l); return l; } -/** Frozen for the same reason the despecialized nodes are: the node is now - * shared by every caller, and the engine is a better guarantor of that than - * an audit of today's call sites. */ +/** Freeze shared layout metadata; the input type itself must stay immutable. */ function mkLayout( d: DespecializedValType, align: number, @@ -74,12 +65,8 @@ function mkLayout( discSize: 0 | 1 | 2 | 4 = 0, payloadOffset = 0, ): Layout { - // The premise the whole cache rests on: every alignment is a power of two. - // It is what lets `load`/`store` substitute `ptr % align === 0` for - // `ptr === alignTo(ptr, align)`, and what lets the retained field/payload - // offsets be base-relative (the aligned base factors out of `alignTo`). - // True by induction over `computeLayout` — asserted here so it stays true, - // once per type rather than per value. + // Power-of-two alignments divide the containing type's alignment, allowing + // field/payload offsets to be relative to an aligned base. assert_( align > 0 && (align & (align - 1)) === 0, "layout alignment must be a power of two", @@ -95,10 +82,7 @@ function mkLayout( } /** - * definitions.py `alignment` (line 1201) and `elem_size` (line 1259) fused: - * the two functions switch over the same despecialized kinds, and every hot - * caller wants both. The compound kinds delegate to the per-kind kernels - * below, which stay the line-by-line spec mirror. + * definitions.py `alignment` and `elem_size`, computed together. */ function computeLayout(t: ValType, ptrType: PtrType): Layout { const d = despecialize(t); @@ -149,17 +133,10 @@ function computeLayout(t: ValType, ptrType: PtrType): Layout { } /** - * The field offsets are exactly the intermediate `p` that - * `elemSizeRecord` computes and discards — the same accumulation, retained - * rather than re-derived, so the offsets cannot drift from the spec's own - * arithmetic. Size and alignment still come from the kernels themselves, - * which remain the single source of truth (and the "empty record" assert). - * - * They are BASE-RELATIVE, where the spec realigns a running absolute `p`. - * The two agree because `load`/`store` have already asserted that the base is - * aligned to this record's own alignment, which is the max over its fields', - * and alignments are powers of two: `alignTo(base + p, a) == base + - * alignTo(p, a)` whenever `a` divides `base`. + * Retain the field offsets from the reference's running alignment arithmetic. + * Offsets are byte distances from an aligned record base. This is equivalent + * to realigning absolute pointers because each field alignment divides the + * base: `alignTo(base + p, a) == base + alignTo(p, a)`. */ function recordLayout( d: DespecializedValType, diff --git a/runtime/src/cabi/load.ts b/runtime/src/cabi/load.ts index 4631ff8..4af19f2 100644 --- a/runtime/src/cabi/load.ts +++ b/runtime/src/cabi/load.ts @@ -24,11 +24,9 @@ export function load( t: ValType, ): ComponentValue { const mem = requireMemory(cx.opts); - // One cached layout node per (type, pointer width) — issue #261; it carries - // the despecialized type too, so this is the only map lookup on the path. + // The cached layout also supplies the despecialized type. const L = layoutOf(t, mem.ptrType()); - // Alignments are always powers of two, so `ptr % align === 0` decides - // exactly what `ptr === alignTo(ptr, align)` did, without the float divide. + // Callers establish alignment and bounds before loading a value. assert_(ptr % L.align === 0, "load misaligned"); assert_(ptr + L.size <= mem.length, "load OOB"); const d = L.d; @@ -137,7 +135,7 @@ export function loadListFromValidRange( if (kind === "u8") { return bytesOf(mem, ptr, length).slice(); } - // Other flat element types lift bulk too (issue #67) — same host shapes + // Other flat element types lift in bulk with the same raw shapes // (number[]/bigint[]/boolean[]), same NaN canonicalization; falls through // to the per-element loop for compound types, char (per-element USV // validation is the point), and non-little-endian platforms. @@ -152,12 +150,8 @@ export function loadListFromValidRange( } /** - * `offsets[i]` is field i's byte offset from `ptr`, precomputed on the layout - * node, so the loop is an indexed read rather than the per-field - * `alignTo`/`alignment`/`elemSize` recomputation it used to be (issue #261). - * Taking the offsets rather than the whole `Layout` keeps the function - * self-consistent: its two arguments are the ones the result depends on, and - * there is no unchecked "these came from the same type" invariant to violate. + * `offsets[i]` is field i's byte offset from the aligned base `ptr`. + * Callers must supply offsets computed for these fields and pointer width. */ export function loadRecord( cx: LiftLowerContext, diff --git a/runtime/src/cabi/memory.ts b/runtime/src/cabi/memory.ts index e72c56a..13640b8 100644 --- a/runtime/src/cabi/memory.ts +++ b/runtime/src/cabi/memory.ts @@ -20,14 +20,14 @@ export function ptrSize(ptrType: PtrType): 4 | 8 { /** * `MemInst` caches its `Uint8Array`/`DataView` at construction time and never * re-derives them. That is only sound for a memory that cannot grow within - * this instance's lifetime: `memory.grow` (guest-triggered or host-triggered) - * detaches the backing `ArrayBuffer`, and a cached view over a detached - * buffer reads/writes garbage rather than trapping. + * this instance's lifetime: `memory.grow` can detach the backing ArrayBuffer. + * Cached TypedArray accesses then cease to address memory, while DataView + * accesses throw rather than perform canonical trap checks. * * Production call/lift/lower paths do NOT construct `MemInst` directly against * a live, growable `WebAssembly.Memory` for this reason: the executor * re-derives a fresh view per access via `LiveMemory` - * (`exec/boundary.ts:97-160`), which is grow-safe. `MemInst` is for contexts + * (`exec/boundary.ts`). `MemInst` is for contexts * where the buffer is known fixed for the duration (tests, or a snapshot * already taken). */ @@ -128,7 +128,6 @@ export function loadPtr(mem: MemInst, ptr: number): number | bigint { // never came from validated guest bytes; it is a JS number/bigint an // embedder handed the lowering path). Ported as `assert_`/`AssertionError` // (see cabi/trap.ts's Trap-vs-AssertionError taxonomy), not `Trap`. -// definitions.py:1568-1569 (`store_int`). // // This scalar path is where the check lives; the bulk (TypedArray) path in // bulk_lists.ts intentionally wraps instead — see that file's header for why. @@ -209,13 +208,9 @@ export function writeBytes(mem: MemInst, ptr: number, src: Uint8Array): void { * Exact bounds check usable with i64-lane values: traps when * `ptr + byteLength > len(memory)`, computed without precision loss. * - * `what` selects the trap wording only. Callers that are checking a pointer - * *returned by realloc* pass wasmtime's phrasing for that case - * ("realloc return: beyond end of memory", - * `wasmtime/src/runtime/component/func/options.rs:185`) and the string-lift - * path passes its own ("string pointer/length out of bounds of memory", - * `.../func/typed.rs:1528`), because the official suite matches those texts. - * The default is unchanged. + * Inputs must already be nonnegative canonical addresses/sizes. `what` + * selects diagnostic wording only; realloc and string callers use the + * messages expected by the official suite. */ export function trapIfRangeExceedsMemory( mem: MemInst, diff --git a/runtime/src/cabi/mod.ts b/runtime/src/cabi/mod.ts index bec6ce9..2c59458 100644 --- a/runtime/src/cabi/mod.ts +++ b/runtime/src/cabi/mod.ts @@ -1,10 +1,6 @@ -// Canonical ABI v1 reference interpreter — public surface. -// -// A direct TypeScript port of the value lift/lower machinery of -// third_party/component-model/design/mvp/canonical-abi/definitions.py -// (readable over fast; docs/architecture.md §8 "v1: a generic interpreter"). -// Task/thread/waitable machinery is deliberately absent — see -// runtime/README.md for the port/defer inventory. +// Canonical ABI value interpreter exports (contracts/descriptor-ir.md). +// Semantics follow definitions.py with the host representations documented +// in types.ts. Scheduling and waitables live in ../task/. export * from "./trap.ts"; export * from "./types.ts"; diff --git a/runtime/src/cabi/store.ts b/runtime/src/cabi/store.ts index b95eaf3..5c97378 100644 --- a/runtime/src/cabi/store.ts +++ b/runtime/src/cabi/store.ts @@ -31,10 +31,7 @@ export function store( ptr: number, ): void { const mem = requireMemory(cx.opts); - // The load-side mirror: one cached layout node carrying the despecialized - // type as well (issue #261), and `ptr % align === 0` in place of - // `ptr === alignTo(ptr, align)` — identical for power-of-two alignments, - // minus the float divide. + // The cached layout also supplies the despecialized type. const L = layoutOf(t, mem.ptrType()); assert_(ptr % L.align === 0, "store misaligned"); assert_(ptr + L.size <= mem.length, "store OOB"); @@ -172,20 +169,15 @@ export function storeListIntoValidRange( const mem = requireMemory(cx.opts); const L = layoutOf(elemType, mem.ptrType()); const kind = L.d.kind; - // docs/architecture.md §7: list is Uint8Array-shaped on the host, and - // both directions are bulk copies — this is the store-side mirror of - // load.ts `loadListFromValidRange`'s u8 fast path (issue #54: the - // per-element interpreted store cost ~45 ns/byte, capping async imports - // returning list at ~22 MB/s while the lift ran at memcpy speed). + // list uses Uint8Array at the raw and facade boundaries. if (kind === "u8") { const dst = bytesOf(mem, ptr, v.length); if (v instanceof Uint8Array) { dst.set(v); return; } - // Plain-array sources (raw-layer embedders) keep the exact per-element - // semantics of `storeInt(…, 1)`: assert integer-ness, then mask mod 256 - // (a Uint8Array element write and DataView.setUint8 wrap identically). + // Plain arrays assert integer shape but wrap modulo 256, unlike scalar + // storeInt's range assertion. See bulk_lists.ts for this raw-input split. for (let i = 0; i < v.length; i++) { const x = v[i]; assert_(typeof x === "number" && Number.isInteger(x), "int store"); @@ -193,10 +185,8 @@ export function storeListIntoValidRange( } return; } - // Other flat element types store bulk too (issue #67), preserving the - // per-element semantics exactly (same asserts, same wrap, canonical-NaN - // floats); falls through for compound types, char, and non-little-endian - // platforms. + // Bulk integer stores wrap; floats canonicalize NaNs (bulk_lists.ts). + // Unsupported kinds/platforms/views fall back to scalar stores. if (tryStoreNumericList(mem, v, ptr, kind)) return; const size = L.size; for (let i = 0; i < v.length; i++) { @@ -204,8 +194,7 @@ export function storeListIntoValidRange( } } -/** The store-side mirror of `loadRecord`: indexed offset writes, no per-field - * layout recomputation (issue #261). */ +/** Byte offsets must match these fields and the memory's pointer width. */ export function storeRecord( cx: LiftLowerContext, v: Record, @@ -225,11 +214,8 @@ export function matchCase( cases: CaseType[], ): [number, ComponentValue] { const label = v.kind; - // `caseIndexOf` is the memoized form of the linear scan this used to run on - // every variant stored (issue #261); it maps a duplicated label to -1, so - // the "exactly one match" condition below is unchanged. A missing or - // non-string `kind` cannot be a key of that Map either, so this one assert - // covers a malformed value too. + // Missing labels and duplicates (-1) both violate the exactly-one-match + // host precondition. const i = caseIndexOf(cases).get(label); assert_(i !== undefined && i >= 0, `variant case '${label}' not found`); return [i as number, v.value]; diff --git a/runtime/src/cabi/strings.ts b/runtime/src/cabi/strings.ts index c684aeb..6a65c5f 100644 --- a/runtime/src/cabi/strings.ts +++ b/runtime/src/cabi/strings.ts @@ -2,7 +2,7 @@ // `store_string*`, `convert_i32_to_char`, `char_to_i32`). // // Host-side strings are plain JS strings (docs/architecture.md §7). Two deliberate -// deviations from definitions.py, both recorded in runtime/README.md: +// host-representation choices: // // 1. No encoding provenance. The reference represents a lifted string as // (str, src_encoding, tagged_code_units) so that lowering can pick a @@ -32,10 +32,7 @@ import type { PtrType, StringEncoding } from "./types.ts"; export const REALLOC_I32_MAX = 2 ** 32 - 1; -// Trap wording for realloc-return validation, matching wasmtime -// (`src/runtime/component/func/options.rs:175,185`) so the official suite's -// `values/realloc.wast` expectations match. Semantics are unchanged; only the -// text differs from the earlier hand-written wording. +// Realloc diagnostics match the official suite's `values/realloc.wast` text. export const REALLOC_MISALIGNED = "realloc return: result not aligned"; export const REALLOC_OOB = "realloc return: beyond end of memory"; export const MAX_STRING_BYTE_LENGTH = (1 << 28) - 1; @@ -149,12 +146,8 @@ export function loadStringFromRange( return latin1Decode(bytes); } } catch { - // Message only: the trap condition is unchanged. wasmtime lifts strings - // with `core::str::from_utf8` and surfaces Rust's `Utf8Error`, whose two - // shapes the official suite asserts on separately - // (`values/strings.wast:85` vs `:101`): a byte sequence that can never be - // valid, versus one that is a valid prefix cut short by the end of the - // string. + // `values/strings.wast` distinguishes malformed UTF-8 from a valid prefix + // cut short at the end of input, matching Rust's Utf8Error diagnostics. trap( encoding === "utf-8" ? utf8ErrorMessage(bytes) @@ -232,9 +225,7 @@ export function storeString( ): void { const mem = requireMemory(cx.opts); const [begin, taggedCodeUnits] = storeStringIntoRange(cx, v); - // Write order matches the reference (store_string, definitions.py:1613-1616): - // begin pointer first, then tagged length. Unobservable here (no trap can - // intervene between the two writes), but kept in step for parity. + // `store_string` writes the begin pointer before the tagged code-unit count. storeInt( mem, mem.ptrSize() === 4 ? begin : BigInt(begin), diff --git a/runtime/src/cabi/trap.ts b/runtime/src/cabi/trap.ts index 40c95d8..1c57f61 100644 --- a/runtime/src/cabi/trap.ts +++ b/runtime/src/cabi/trap.ts @@ -1,15 +1,14 @@ // Trap and assertion machinery (definitions.py `Trap`, `trap`, `trap_if`). // // `Trap`'s canonical definition lives in `@polyengine/protocol` -// (contracts/embedder-api.md §"Module identity": it is an embedder-contract value and must be recognizable across -// runtime copies, issue #83); it is re-exported here so every existing -// `from "../cabi/trap.ts"` import path is unchanged. The protocol package is -// dependency-free, so this import introduces no cycle. +// so it is recognizable across runtime copies +// (contracts/embedder-api.md §"Module identity"). // -// `Trap` models a Component Model trap — a deterministic guest-visible fault. -// `AssertionError` models the reference's Python `assert`s: internal -// invariants that callers are supposed to make unviolable. Tests treat only -// `Trap` as an expected outcome. +// `Trap` represents a Component Model trap. `AssertionError` represents +// reference assertions and host-precondition violations, such as an invalid +// value supplied to scalar lowering, not a guest's canonical trap outcome. +// Throwing a JS exception does not itself ensure guest uncatchability; +// see intrinsics/mod.ts `HostTrapState` for that limitation. import { Trap } from "@polyengine/protocol"; @@ -34,7 +33,7 @@ export function assert_(cond: boolean, message?: string): asserts cond { if (!cond) throw new AssertionError(message); } -/** Marks a definitions.py code path this v1 interpreter does not port yet. */ +/** Marker for an unsupported interpreter path, distinct from a guest trap. */ export class NotImplemented extends Error { constructor(what: string) { super(`not implemented in cabi v1: ${what}`); diff --git a/runtime/src/cabi/types.ts b/runtime/src/cabi/types.ts index 4e04322..e3f25e7 100644 --- a/runtime/src/cabi/types.ts +++ b/runtime/src/cabi/types.ts @@ -1,20 +1,7 @@ -// Canonical ABI — provisional component value-type model. -// -// This is the v1 sketch of the "CABI descriptor IR" of docs/architecture.md §8: the type -// information the host-boundary lift/lower interpreter walks. It mirrors the -// type classes of the executable spec -// (third_party/component-model/design/mvp/canonical-abi/definitions.py) as a -// TypeScript discriminated union, holding only what lift/lower needs. -// -// PROVISIONAL — expected to change when the translator shim (docs/architecture.md §4.2) -// defines the real plan format. Open questions, tracked in runtime/README.md: -// - Serialized encoding (this in-memory shape vs the plan's wire format). -// - Whether labels stay strings or become interned indices in the IR. -// - Host-facing value representations for tuple/variant/option/result -// (currently the despecialized definitions.py shapes; bindgen §9 will -// want arrays / tagged unions / undefined-based options). -// - Resource types: here an opaque token; the plan will carry resource-type -// indices + dtor references instead. +// In-memory CABI descriptor model (contracts/descriptor-ir.md). +// The plan loader resolves wire indices into these types and resource identity +// tokens. The lift/lower interpreter follows definitions.py using the raw +// value shapes below; embedder facades translate to their own public shapes. /** Core wasm value types, as strings (mirrors definitions.py flat types). */ export type CoreType = "i32" | "i64" | "f32" | "f64"; @@ -42,9 +29,8 @@ export function coreFuncTypeEquals(a: CoreFuncType, b: CoreFuncType): boolean { export type StringEncoding = "utf8" | "utf16" | "latin1+utf16"; /** - * Opaque token standing in for the implementing component instance of a - * resource type (definitions.py `ResourceType.impl`). The value interpreter - * only ever compares these by identity. + * Minimal shape of a resource's implementing instance + * (definitions.py `ResourceType.impl`); instance comparisons use identity. */ export interface InstanceLike { handles: unknown; // Table — typed loosely here to avoid a cycle; see handles.ts @@ -55,16 +41,10 @@ export interface InstanceLike { * definitions.py `ResourceType`: identity + implementing instance + optional * destructor. Compared by object identity everywhere. * - * `dtorHost` is the **host-initiated**-drop entry (#85, reshaped by #160): - * the dtor built as a fully LIFTED sync function — definitions.py - * `canon_resource_drop` (line 2319) `inst.store.lift(dtor, ft, opts, - * rt.impl)` — so the destructor's activation gets a real Task/Thread, the - * reentrance bracket is released at its first park, and its suspension - * points are resumable by the scheduler. It is wired by exec/executor.ts - * (jspi-`promising` entry only when the dtor is suspension-capable, docs §7) - * and called through `hostDtorCall` (exec/boundary.ts), which also fills it - * in lazily for tokens built directly. It returns `undefined` or a Promise, - * so it is NOT callable from inside a guest activation. + * `dtorHost` is the host-initiated drop entry, wired by exec/executor.ts or + * lazily by `hostDtorCall` in exec/boundary.ts. It lifts the dtor with a fresh + * Task/Thread and lets the scheduler finish an unfinished activation. Its + * undefined-or-Promise return is not a guest-callable core ABI. * * Guest-initiated drops (`callDtorGated`) lift `dtor` through the same * machinery with a fresh synchronous task/thread, the guest caller identity, @@ -198,44 +178,24 @@ export interface FuncType { } // --------------------------------------------------------------------------- -// Component-level values (host-side JS representations, docs/architecture.md §7) +// Component-level values (raw JS representations, contracts/descriptor-ir.md) // --------------------------------------------------------------------------- -/** - * Host-side value representation produced by lifting / consumed by lowering: - * bool -> boolean - * u8..u32, s8..s32, f32, f64 -> number - * u64, s64 -> bigint - * char -> single-code-point string - * string -> string (plain; see README on dropped encoding provenance) - * list -> Uint8Array (copy; docs/architecture.md §7) — other lists -> Array - * record -> { [fieldLabel]: value } - * tuple -> despecialized record { "0": v0, "1": v1, ... } - * variant/enum/option/result -> VariantValue { kind: caseLabel, value: payload|null } - * flags -> { [label]: boolean } - * own/borrow -> number (the resource rep at this layer) - * These mirror definitions.py's Python shapes; final host-facing bindings - * representations are an open question (README). - */ /** * Opaque host token for the async value types. * * `stream`, `future` and `error-context` do not lift to plain data: the - * reference's `lift_async_value` (definitions.py line 1530) yields the + * reference's `lift_async_value` yields the * *shared* stream/future object itself, because its identity is the value — * two components holding ends of one stream must see each other's copies. * Concretely these are `SharedStreamImpl`, `SharedFutureImpl` and - * `ErrorContext` instances (runtime/src/task/streams.ts); they are declared + * `ErrorContext` instances (task/streams.ts); they are declared * opaquely here to keep `cabi/types.ts` free of a dependency on the task * layer. Host code should treat one as a token and pass it back unchanged. * - * LIMITATION: this brand is *structural*, so an all-optional interface admits - * any object — it documents intent, it does not enforce it. The enforcement is - * at the lowering sites, which `assert_` on the concrete class before using a - * value (`lowerStream`/`lowerFuture` in cabi/async_values.ts check - * `instanceof SharedStreamImpl`/`SharedFutureImpl`). A nominal brand would - * need a required property, which the real classes could not satisfy without - * cabi importing the task layer — the cycle this declaration exists to avoid. + * This all-optional interface documents intent but does not enforce identity. + * `lowerStream`/`lowerFuture` in async_values.ts check concrete classes; + * callers must not treat arbitrary objects as valid tokens. */ export interface AsyncValue { readonly __asyncValue?: never; @@ -261,6 +221,16 @@ export type VariantValue = { value: ComponentValue; }; +/** + * Raw values produced by lifting and consumed by lowering, not facade values: + * bool -> boolean; <=32-bit integers/floats -> number; 64-bit integers -> bigint; + * char -> one-scalar string; string -> JS string without encoding provenance; + * list -> copied Uint8Array; other lists -> arrays; + * record/flags -> label-keyed object; tuple -> { "0": v0, "1": v1, ... }; + * variant/enum/option/result -> VariantValue (result error kind is "error"); + * own/borrow -> numeric resource rep; async types -> opaque shared tokens. + * See contracts/embedder-api.md for the distinct facade representations. + */ export type ComponentValue = | AsyncValue | boolean @@ -285,36 +255,11 @@ export type DespecializedValType = Exclude< >; /** - * Despecialization memo, keyed on input identity. - * - * `ValType` objects are built once per plan by plan/loader.ts `loadValType` - * and never mutated, so their identity is stable for the plan's lifetime — - * the same justification embedder/values.ts `checkedLabels` relies on. Issue - * #261: `despecialize` sits at the top of `load`, `store`, `alignment`, - * `elemSize`, `lowerFlat` and `contains`, and allocated a fresh record or - * variant on every one of those calls, per element and per field. - * - * The memo hands every caller the SAME object where each previously got a - * fresh one, so it is sound only as long as no caller mutates the result. - * That invariant is enforced in HALF the cases and merely relied upon in the - * other half, and the difference is worth knowing: - * - * - The five specialized kinds return an object built here, and everything - * built here is frozen. These modules are strict-mode, so a later - * mutation throws instead of silently corrupting every other holder. - * - The default branch returns the plan-owned `t` unchanged — which for a - * plan-declared record/variant/list is MOST types in practice — and - * freezing plan types is out of scope. Nothing enforces immutability - * there; it rests on plan/loader.ts `loadValType`, which builds every - * node bottom-up and never writes to one afterwards. - * - * Violating that second half is now worse than it used to be. Mutating a plan - * record's `fields` was previously self-correcting, because every `load`/ - * `store` recomputed the layout from the current array; with layout.ts's - * cache the mutation instead desynchronizes a `Layout` whose `fieldOffsets` - * were computed from the pre-mutation shape — wrong bytes at wrong offsets, - * no error anywhere. A `ValType` reachable from a plan is immutable; treat - * that as a hard invariant of this layer, not a convention. + * Identity-keyed memo. Input types and all reachable type/field/case arrays + * must remain immutable: mutating them would leave despecialization and byte + * layouts inconsistent. `loadValType` builds plan nodes without later writes. + * Nodes and arrays synthesized here are frozen; existing input nodes returned + * by the default branch, and child types referenced by new nodes, are not. */ const despecializedCache = new WeakMap(); @@ -329,9 +274,8 @@ export function despecialize(t: ValType): DespecializedValType { } /** - * definitions.py `despecialize` (line 1163), line for line. Only the objects - * built HERE are frozen; the default branch returns the plan-owned `t` - * unchanged, and freezing plan types is out of scope. + * definitions.py `despecialize`, using the raw variant/record shapes above. + * Only synthesized nodes are frozen; input nodes are returned unchanged. */ function despecializeUncached(t: ValType): DespecializedValType { switch (t.kind) { @@ -385,15 +329,9 @@ function despecializeUncached(t: ValType): DespecializedValType { // --------------------------------------------------------------------------- /** - * The discriminant WIDTH — the single mirror of definitions.py - * `discriminant_type`'s arithmetic (line 1234), including its `assert(0 < n < - * (1 << 32))`. `math.ceil(log2(n)/8)`: 0|1 -> u8, 2 -> u16, 3 -> u32. - * - * The width is what the layout code actually wanted: on every variant lifted - * or lowered it used to build a `PrimType` purely to hand it to - * `alignment`/`elemSize` (issue #261). u8/u16/u32 each have alignment equal to - * their size, so this one number is both the discriminant's size and its - * alignment — which is why the variant layout kernels need nothing else. + * definitions.py `discriminant_type`: byte width for a nonempty variant with + * fewer than 2^32 cases. u8/u16/u32 have alignment equal to size, so this + * also supplies discriminant alignment to the layout code. */ export function discriminantSize(caseCount: number): 1 | 2 | 4 { const n = caseCount; @@ -404,13 +342,7 @@ export function discriminantSize(caseCount: number): 1 | 2 | 4 { } /** - * Module-level singletons rather than fresh literals per call: `flatten.ts` - * calls `discriminantType` once per variant flattened and passes the result - * to `flattenType`, which despecializes it — a fresh object there is one - * `despecializedCache` insert on immediate garbage every time (the same - * defeat-the-identity-cache pattern `values.ts` `spillTupleType` avoids). - * Frozen for the reason the despecialized nodes are: they are now shared, and - * the engine is a better guarantor of "nobody mutates this" than an audit. + * Frozen shared types let flattening reuse the identity-keyed type caches. */ const U8: PrimType = Object.freeze({ kind: "u8" }); const U16: PrimType = Object.freeze({ kind: "u16" }); @@ -419,8 +351,8 @@ const U32: PrimType = Object.freeze({ kind: "u32" }); /** * definitions.py `discriminant_type`. The table itself lives in * `discriminantSize`; this is only the width -> type mapping, so the bound - * check, the thresholds and the error string exist once. Flattening - * (flatten.ts:157) and layout (layout.ts `alignmentVariant`, + * check and thresholds exist once. Flattening + * (`flattenVariant`) and layout (`alignmentVariant`, * `elemSizeVariant`) therefore cannot drift apart on the discriminant width. */ export function discriminantType(cases: CaseType[]): PrimType { @@ -435,14 +367,8 @@ export function discriminantType(cases: CaseType[]): PrimType { } /** - * Label -> case index for a variant's `cases`, memoized on the array's - * identity (plan-owned and stable, same argument as the despecialization - * memo). Replaces the linear scan `matchCase` ran on every variant stored - * (issue #261). - * - * A label bound to more than one case maps to -1, because the scan this - * replaces asserted on finding exactly ONE match: collapsing duplicates to - * "last wins" would silently accept a variant the old code rejected. + * Label -> case index, keyed on immutable case-array identity. Duplicate + * labels map to -1 so `matchCase` enforces exactly one match, not last-wins. */ const caseIndexCache = new WeakMap>(); @@ -492,20 +418,9 @@ export function contains( // --------------------------------------------------------------------------- /** - * Structural `ValType` equality. - * - * CONTRACT (bugfix, generalized during the #18 tls smoke): naive - * `JSON.stringify(a) === JSON.stringify(b)` recurses into `own`/`borrow`'s - * `ResourceTypeInfo` — a class whose `impl` field is documented "Compared by - * object identity everywhere" (see `ResourceTypeInfo` above) and which cycles - * back to the owning instance state (`impl.handles` holds live resource - * tables that reference their types), so `JSON.stringify` throws - * `TypeError: Converting circular structure to JSON` on ANY type containing - * `own`/`borrow` at any depth. First hit by `task.return` result types - * (polymorph-test's `list>`), then by stream/future - * element types (polymorph-tls streams carrying resource-bearing payloads). - * Object-identity types (`ResourceTypeInfo`) are compared by reference, per - * the documented invariant. + * Structural equality except for resource types, which compare by token + * identity. Do not serialize or recurse into ResourceTypeInfo: its instance + * points back to live handle tables and can form cycles. */ export function valTypesEqual(a: ValType[], b: ValType[]): boolean { return a.length === b.length && a.every((t, i) => valTypeEqual(t, b[i])); @@ -586,9 +501,8 @@ export function valTypeEqual(a: ValType, b: ValType): boolean { } /** - * Cycle-safe display form for diagnostics. `JSON.stringify(t)` is UNSAFE on - * any resource-bearing type (see `valTypeEqual`'s contract note); this prints - * the structural shape and elides `ResourceTypeInfo` identities. + * Diagnostic shape that elides resource identities and their instance cycles. + * The remaining type structure must be acyclic, as in a loaded plan. */ export function fmtValType(t: ValType | null): string { if (t === null) return "_"; diff --git a/runtime/src/cabi/values.ts b/runtime/src/cabi/values.ts index 247eb9c..4bf725f 100644 --- a/runtime/src/cabi/values.ts +++ b/runtime/src/cabi/values.ts @@ -14,15 +14,9 @@ import { asIndex } from "./memory.ts"; import type { ComponentValue, CoreValue, TupleType, ValType } from "./types.ts"; /** - * The spill tuple for a parameter/result list, memoized on the array's - * identity. - * - * Both spill paths used to synthesize `{ kind: "tuple", elements: ts }` on - * every call, which under the identity-keyed layout/despecialization caches - * (issue #261) is a guaranteed miss plus a wasted insert — it would make this - * path slower, not faster. `ts` is always a plan-owned `ft.params`/ - * `ft.results` array (exec/boundary.ts, intrinsics/async_builtins.ts), so its - * identity is as stable as a `ValType`'s. + * Shared spill tuple keyed on parameter/result array identity, allowing both + * paths to reuse type/layout caches. The array and its types must remain + * immutable after first use, as for plan-owned ft.params/ft.results. */ const spillTuples = new WeakMap(); diff --git a/runtime/src/cache/core.ts b/runtime/src/cache/core.ts index 977210a..2ab208c 100644 --- a/runtime/src/cache/core.ts +++ b/runtime/src/cache/core.ts @@ -2,53 +2,19 @@ // only; nothing here depends on the engine's own compiled-module cache, // i.e. layer 2). // -// GOAL: content-address a shim translation by +// Content-address a shim translation by // `(component sha256, translator build hash, feature flags)` so a reload // (same component, same shim build, same features) can skip the // translate-with-the-shim step entirely. // -// CACHE FAILURES ARE NEVER FATAL (issue #196): the cache is a pure -// optimization layered over `Translator.translateRaw` + `loadEnvelope`, both -// of which already succeed/fail on their own terms. A `get`/`put`/internal -// self-heal failure — including a read-only or otherwise unwritable cache -// root — must never turn into a failed translation; at worst it turns into -// a fresh (uncached) translation. `translateCached` swallows `get`/`put` -// failures (surfaced only via the opt-in `onCacheError` callback below); -// the backends (dir.ts, web.ts) swallow their internal self-heal evictions -// and turn any `get`-path I/O failure into a `null` (miss) rather than a -// throw. The one exception, by design, is `TranslateError` from -// `loadEnvelope`: that is a verdict about the *input component*, not a -// cache failure, and keeps propagating uncached. The public `evict()` also -// keeps throwing — an explicit caller asked for that specific effect and -// deserves to know if it didn't happen. +// translateCached treats get/put failures as misses or uncached successes; +// key construction, translation and envelope validation still fail normally. +// Direct backend calls retain their own error contracts, including evict. // -// PERSISTED-ARTIFACT-SET DECISION (governing: contracts/plan-format.md -// "Artifact set" + "No duplicate bytes" — plan-format.md:24-31,63-64): -// we persist `plan.json` (the wire `WirePlan`) and the FACT adapter modules -// only. We do NOT persist the original component bytes. -// -// Evidence this is correct, not merely convenient: -// - plan-format.md is explicit: "The original component binary is the -// third input at instantiation time; the plan never embeds it" and -// "Embedded core modules are referenced as `[offset, len)` byte ranges -// into the original component binary — the executor slices them -// itself" (decision 3). `instantiateComponent`/`Facade.instantiate` -// (runtime/src/embedder/instantiate.ts `ComponentArtifacts`) take -// `componentBytes` as a caller-supplied field *alongside* `plan` and -// `adapters` — never as something the plan or its loader manufacture. -// - Operationally: whoever calls `translateCached` already holds the -// component bytes (that's how they'd have a sha256 to form a cache key -// in the first place, and how `verifyComponent`'s length check runs -// without a bytes store at all). Reload use cases (browser page reload, -// Deno process restart) re-fetch/re-read the *component* from its own -// source of truth (network, disk) every time; only the *translation* -// (the expensive shim call) is worth skipping. Storing componentBytes a -// second time would be pure duplication with no consumer. -// - `get()` therefore verifies integrity using only the requested -// `CacheKey.componentSha256` against the value that was true at `put()` -// time (recorded in the stored metadata and cross-checked against the -// embedded `plan.component.sha256`) — never against fresh bytes, which -// this layer never sees. +// Persist plans and FACT adapters, not original component bytes: the caller +// supplies those again at instantiation (plan-format.md "Artifact set"). +// Backend checks compare metadata and plan.component.sha256 to the requested +// key and validate plan structure; they do not authenticate stored contents. import type { WirePlan } from "../plan/format.ts"; import { loadEnvelope, PlanError } from "../plan/loader.ts"; @@ -64,14 +30,8 @@ export interface TranslatorLike { translateRaw(componentBytes: Uint8Array): string; } -/** Cache layout version. Bumped on any incompatible on-disk/on-Cache-API - * schema change; an unrecognized version is read back as a miss (never a - * crash) so stale caches from an older build self-heal by re-translating — - * and this self-healing holds even when the cache root itself is - * unwritable (issue #196): the eviction attempt that a layout mismatch - * triggers is swallowed internally by the backend, so a read-only - * pre-warmed cache from an older layout degrades to "always miss, always - * re-translate" rather than throwing. +/** Cache layout version. Bump on incompatible storage-schema changes. + * Unknown versions are misses; stale-entry eviction is best effort. * * @internal — on-disk/on-Cache-API schema version, owned by the bundled * cache backends. */ @@ -167,7 +127,7 @@ export async function keyFor( export interface TranslateCachedOptions { features?: string[]; /** - * Opt-in determinism guard (tests only, per dispatch): after a cache miss, + * Opt-in determinism guard: after a cache miss, * translate a second time and assert byte-identical envelope JSON before * trusting/storing the result. Throws `Error` on mismatch. */ @@ -242,10 +202,7 @@ export async function translateCached( ); } } - // loadEnvelope both validates (throws TranslateError for a validation - // verdict — must propagate uncached, per TranslateError's docs: a - // validation verdict is a judgment about the *input component*, not - // something to cache-and-replay) and gives us the split plan/adapters. + // Translation/envelope failures propagate; only successful artifacts cache. const { wire, adapters } = loadEnvelope(first); // A `put` failure (issue #196) is swallowed: the translation already diff --git a/runtime/src/cache/dir.ts b/runtime/src/cache/dir.ts index 0500fb2..6dfe61b 100644 --- a/runtime/src/cache/dir.ts +++ b/runtime/src/cache/dir.ts @@ -4,15 +4,9 @@ // meta.json CacheMeta (layoutVersion, componentSha256, // translatorBuildHash, features) // plan.json the wire WirePlan, JSON-serialized -// adapters/ one file per adapter, name = the tail of the wire -// plan's `modules[].file` (which is already -// `adapters/.wasm` shaped upstream — see -// contracts/plan-format.md "Artifact set" — so this -// backend nests one more `adapters/` level under the -// key directory: `//adapters/.wasm`) +// adapters/ one file per adapter, using modules[].file verbatim // -// Deliberately does NOT store component bytes — see core.ts's -// "PERSISTED-ARTIFACT-SET DECISION" docs for why that's sound, not a gap. +// Original component bytes remain caller-supplied (see core.ts). import type { ArtifactCache, @@ -63,11 +57,7 @@ class DirCache implements ArtifactCache { return `${this.root}/${await keyHex(key)}`; } - /** Internal self-heal eviction (issue #196): the caller is `get`'s own - * recovery path for a poisoned/stale entry, not an explicit caller of - * `evict()` — so a failure here (e.g. an unwritable cache root) must not - * escape and fail what would otherwise be a clean miss. The public - * `evict()` below keeps throwing; only this internal path swallows. */ + /** Best-effort stale-entry eviction; unlike public evict, failure is a miss. */ async #tryEvict(key: CacheKey): Promise { try { await this.evict(key); @@ -118,11 +108,7 @@ class DirCache implements ArtifactCache { return { plan, adapters }; } catch { - // Any parse/read/structural/I-O failure = a poisoned entry (or an - // unreadable/unwritable cache root, issue #196): miss + best-effort - // evict, never trust and never throw out of `get` (dispatch - // requirement). This also covers `exists()`'s rethrow of non- - // `NotFound` stat errors (ENOTDIR, EACCES, ...). + // Parse, validation and I/O failures become misses, even if eviction fails. await this.#tryEvict(key); return null; } @@ -156,11 +142,7 @@ class DirCache implements ArtifactCache { await rmIfExists(dir); await Deno.rename(tmp, dir); } catch (e) { - // A `put` failure is non-fatal at the `translateCached` layer - // (issue #196), but repeated failures must not litter the cache - // root with orphaned `.tmp-` scratch directories. Cleanup - // failures here are themselves swallowed — `put` still throws its - // original error either way. + // Remove partial output best-effort, preserving the original put error. try { await rmIfExists(tmp); } catch { diff --git a/runtime/src/cache/web.ts b/runtime/src/cache/web.ts index 28a3ff1..74b4eb8 100644 --- a/runtime/src/cache/web.ts +++ b/runtime/src/cache/web.ts @@ -1,10 +1,9 @@ // Cache API backend for the artifact cache (docs/architecture.md §10), for browsers // (and other environments implementing the standard `CacheStorage`/`Cache` // interfaces). Feature-detects `globalThis.caches`; throws a named error -// where unavailable (e.g. plain Deno without `--unstable-*` polyfills, or a -// non-secure-context page) rather than silently no-op'ing. +// where unavailable rather than silently no-op'ing. // -// Layout: one synthetic same-origin-ish URL per cache key, stored as a +// Layout: one synthetic absolute URL per cache key, stored as a // single JSON `Response` body containing `{meta, plan, adapters}` (adapters // base64-encoded — `Cache` stores `Response` bodies, not arbitrary trees, // so we can't mirror dirCache's file-per-adapter layout; one blob per entry @@ -51,8 +50,7 @@ function fromBase64(s: string): Uint8Array { return out; } -/** Synthetic request URL an entry is stored under. Same-origin-relative so - * it works under any page origin; the path has no filesystem meaning. */ +/** Synthetic absolute cache key, not a fetched URL or filesystem path. */ function entryUrl(hex: string): string { return `https://artifact-cache.invalid/${hex}`; } @@ -67,11 +65,7 @@ class WebCache implements ArtifactCache { return await globalThis.caches.open(this.cacheName); } - /** Internal self-heal eviction (issue #196): the caller is `get`'s own - * recovery path for a poisoned/stale entry, not an explicit caller of - * `evict()` — so a failure here must not escape and fail what would - * otherwise be a clean miss. The public `evict()` below keeps throwing; - * only this internal path swallows. */ + /** Best-effort stale-entry eviction; unlike public evict, failure is a miss. */ async #tryEvict(key: CacheKey): Promise { try { await this.evict(key); @@ -81,13 +75,9 @@ class WebCache implements ArtifactCache { } async get(key: CacheKey): Promise { - // `open()` failing (no `globalThis.caches` in this environment) is a - // capability/configuration error, not a per-entry I/O failure — it - // propagates uncaught (existing behavior, `WebCacheUnavailableError`) - // so a caller who tries to use this backend somewhere it can't work - // finds out immediately rather than silently always-missing. Once open - // succeeds, every failure below (issue #196: poisoned entry, self-heal - // eviction, ...) is swallowed to a `null` miss. + // Opening errors propagate from direct backend calls. After opening, + // entry read/validation failures become misses. translateCached catches + // either kind of get failure. const cache = await this.open(); try { const hex = await keyHex(key); @@ -119,9 +109,7 @@ class WebCache implements ArtifactCache { } return { plan, adapters }; } catch { - // Poisoned/corrupted entry, or any other I/O failure (issue #196): - // miss + best-effort evict, never trust and never throw out of - // `get`. + // An unreadable/invalid entry is a miss; eviction is best effort. await this.#tryEvict(key); return null; } diff --git a/runtime/src/digest/digest.ts b/runtime/src/digest/digest.ts index 2995523..1932e06 100644 --- a/runtime/src/digest/digest.ts +++ b/runtime/src/digest/digest.ts @@ -3,14 +3,13 @@ // schema — contracts/plan-format.md). Counterpart: // crates/bindgen/src/digest.rs, computed from `wit_parser::Resolve`. Both // must produce byte-identical canonical JSON (and therefore identical -// sha256) for a structurally-equivalent world — this equality IS the -// design validation (see runtime/tests/digest_test.ts's cross-language -// fixture test). The wire plan's own `worldDigest` field is legacy +// sha256) for supported world shapes (runtime/tests/digest_test.ts). +// The wire plan's own `worldDigest` field is legacy // (contracts/plan-format.md), retained for wire compatibility only; this // module computes the normative digest independently. // -// THE NORMALIZATION SPEC IS DOCUMENTED ONCE, in crates/bindgen/src/digest.rs's -// module doc comment (kept in sync with this file) — read it first. Short +// Normalization is governed by contracts/digest.md, with the Rust counterpart +// in crates/bindgen/src/digest.rs. Short // version: sort import/export lists by name; keep everything else // (record fields, variant cases, enum/flags label order, function // parameter order) positional because it's ABI-relevant; drop parameter @@ -18,24 +17,12 @@ // convention) and all docs/stability metadata (not present in the plan at // all); identify resources by qualified name, not by table index. // -// ## Known limitation (plan side only): resourceTables aliasing -// -// The plan's `own`/`borrow` `resource: N` indexes into `resourceTables`, -// and empirically (the `resources` fixture) an `own`/`borrow` occurrence in -// an exported function's type and the `{kind:"type"}` export that *names* -// that same nominal resource can reference two *different* resourceTables -// indices (one per component-linking instance boundary — the type export -// sits at the sub-instance that defines the resource, `own`/`borrow` sites -// reference the root instance's re-export of it). Resolving this in -// general requires walking wasmtime's resource-alias chain, which the plan -// format does not expose directly — this is exactly the "impedance between -// wit-parser's view and the plan's types" flagged for the §9 degraded-mode -// question in the track report. This implementation takes an honest -// shortcut: when a world has exactly one nominal resource type, every -// `own`/`borrow` occurrence (regardless of table index) is identified with -// that resource — correct for every world in the current fixture corpus -// (each has 0 or 1 resource types), but a world with 2+ resources throws a -// clearly-labeled `DigestError` rather than silently guessing. +// Current implementation limits: imported resources are refused. With one +// named exported resource, every table index maps to that name. With multiple +// names, only directly named table indices are resolved; extra table aliases +// cause refusal. Unlike plan/loader.ts, this digest implementation does not +// use concrete tables' ResourceIndex to unify aliases. This is not a missing +// plan capability. Flat import names also do not reconstruct nested interfaces. import type { WireExport, @@ -45,7 +32,7 @@ import type { WireValType, } from "../plan/format.ts"; -// cewd = component-engine world digest, the project's pre-rebrand name; kept as an opaque wire constant. +// Version of the canonical `cewd` document, independent of plan formatVersion. /** @internal */ export const CEWD_VERSION = 1; @@ -106,19 +93,8 @@ function hex(buf: ArrayBuffer): string { // --------------------------------------------------------------------------- function buildResourceNameMap(plan: WirePlan): Map { - // CONTRACT: the `importedResources` field (contracts/plan-format.md - // schema; format.ts:23-33). An imported resource occupies - // `ResourceIndex` slots *before* every defined (own/exported) resource - // (`ResourceIndex = importedResources.length + DefinedResourceIndex`), and - // this implementation has no alias map from those imported-resource - // indices to a qualified name (no plan-format extension exists yet for - // that — see the module-level "Known limitation" comment). Silently - // aliasing an own/borrow reference to an imported resource with the lone - // exported resource's name (the pre-fix single-resource-world shortcut - // below) would produce a digest that matches a WIT world it is NOT - // ABI-compatible with — worse than an unresolved-index throw. Refuse - // conservatively whenever the plan declares any imported resources, full - // stop, regardless of how many named (exported) resources exist. + // This implementation names exported resources only. Refuse imports + // rather than accidentally assign them the single exported name below. if ( plan.importedResources !== undefined && plan.importedResources.length > 0 ) { @@ -136,9 +112,8 @@ function buildResourceNameMap(plan: WirePlan): Map { walkExportsForResourceNames(plan.exports, [], named); if (named.size <= 1) { - // Single-resource (or zero-resource) world: identify EVERY resourceTables - // index with the one named resource, sidestepping the aliasing gap - // documented in the module comment above. + // Single-name fallback: all tables receive that name. This does not + // prove that unnamed tables denote the same nominal resource. const theOne = named.size === 1 ? [...named.values()][0] : undefined; const all = new Map(); if (theOne !== undefined) { @@ -146,8 +121,7 @@ function buildResourceNameMap(plan: WirePlan): Map { } return all; } - // Multi-resource world: table-index aliasing across instance boundaries - // is not resolved by this implementation. Fail loudly. + // Multiple names: this implementation cannot attribute extra table aliases. if (named.size < plan.resourceTables.length) { throw new DigestError( `digest: world has ${named.size} named resource type(s) but ` + @@ -184,17 +158,9 @@ function canonImport( imp: WireImport, resourceNames: Map, ): Canon { - // CONTRACT: the plan's `imports` list is flat (`{name, path, kind, type}`) - // even for interface-qualified imports (`imports[].path`, - // contracts/plan-format.md schema — "Untested: current corpus has no - // imports"); no fixture in this repo's - // sync corpus (hello/values/resources) has any imports, so this path is - // exercised by no test. Best-effort flattened-name treatment, chosen to - // be structurally analogous to the export side's nested naming without - // requiring the executor to reconstruct nested WorldItem::Interface shape - // from a flat list. Revisit when a corpus component actually imports - // something (flagged in the track report for §9's degraded-mode - // question). + // Flatten path segments followed by import name. This does not reconstruct + // bindgen's nested interface items; interface-qualified imports can therefore + // produce a digest mismatch even when their function signatures agree. const name = imp.path.length > 0 ? [...imp.path, imp.name].join("/") : imp.name; @@ -243,16 +209,8 @@ function canonExportItem( if (exp.type.kind === "resource") { return { kind: "resource", name: exp.name }; } - // Non-resource named types (records/variants/etc. declared at - // interface/world scope) do not themselves appear as component exports — - // only functions and resources do — so they are intentionally NOT - // emitted as a top-level item here; they still affect the digest via - // whichever function signatures reference them. Mirrors - // crates/bindgen/src/digest.rs's `canon_interface`/`canon_items` (both - // skip non-resource `TypeDefKind` entries with the identical rationale). - // Caller must filter these out of the containing list (see canonExportItem's - // caller / the `null`-sentinel handling below), since this returns a - // JSON value, not `undefined`, when called directly. + // Non-resource type exports contribute through function signatures, not + // as standalone digest items, matching bindgen's canon_interface/canon_items. return null; } @@ -333,10 +291,7 @@ function canonValType( err: t.err === null ? null : canonValType(t.err, resourceNames), }; case "map": - // Despecialized to list> — matches crates/bindgen/src/ - // digest.rs's treatment of wit_parser's `TypeDefKind::Map`. Fixture- - // only path (descriptor-ir.md "Open items": not emitted by current - // translators), unexercised by the sync corpus. + // Match bindgen's TypeDefKind::Map normalization to list>. return { kind: "list", element: { diff --git a/runtime/src/embedder/copy.ts b/runtime/src/embedder/copy.ts index cef2057..91e1bab 100644 --- a/runtime/src/embedder/copy.ts +++ b/runtime/src/embedder/copy.ts @@ -38,10 +38,7 @@ export const RUNTIME_VERSION = "0.6.8"; * Compose a cross-copy diagnostic: what was foreign, which copy is speaking, * the census of every copy in the graph, and the by-value remediation. * - * Kept to one line but complete — the whole point of module identity's stateful half is - * that "recognized but foreign" is a NAMED failure, never a silent - * adaptation (a foreign `Stream` pumped as an async iterable) and never a - * misleading generic ("handle is not an error-context"). + * Stateful foreign handles must be diagnosed, not silently adapted by value. */ export function describeCrossCopy(what: string, remedy?: string): string { const census = copyCensus(); diff --git a/runtime/src/embedder/errors.ts b/runtime/src/embedder/errors.ts index 0d632ef..0bb924a 100644 --- a/runtime/src/embedder/errors.ts +++ b/runtime/src/embedder/errors.ts @@ -1,11 +1,7 @@ // The embedder-facing error model (contracts/embedder-api.md §"Error model"). // -// The canonical definitions live in `@polyengine/protocol` since §"Module identity and @polyengine/protocol" — -// this module is the unchanged import path for them (every existing -// `from "./errors.ts"` / `@polyengine/runtime/embedder` import keeps working) plus -// the runtime-local `NameCollisionError`, which never crosses a copy boundary -// (it is raised while BUILDING a facade, before any value exists) and -// therefore carries no brand: the module identity table's omissions are deliberate. +// ABI error definitions live in @polyengine/protocol. This internal module +// also defines the unbranded, application-facing NameCollisionError. // // Recognition at the runtime's own boundaries is by BRAND, not class: use the // `is*` predicates re-exported below, never `instanceof`, for any value that @@ -28,9 +24,8 @@ export { /** * Two WIT labels in one scope camelCase to the same JS name. * - * A footgun is a design defect (contract principle 2): silently letting one - * field/flag/function shadow another would corrupt values at the boundary with - * no diagnostic anywhere. Refused at facade build instead. + * Raised during facade construction or value adaptation instead of letting + * one field, flag or function silently shadow another. */ export class NameCollisionError extends Error { constructor(message: string) { diff --git a/runtime/src/embedder/imports.ts b/runtime/src/embedder/imports.ts index 3baf8db..715cd75 100644 --- a/runtime/src/embedder/imports.ts +++ b/runtime/src/embedder/imports.ts @@ -2,10 +2,7 @@ // import leaves (contracts/embedder-api.md §"Module wiring and // instantiation"). // -// `plan.imports` proved the right authority, and every embedder -// that needed it would otherwise hand-roll the same walk. -// Blessing it removes that. It is also this layer's own input: the facade -// builds its import wrappers from exactly this list. +// The facade builds its import wrappers from this same plan.imports walk. import type { WirePlan } from "../plan/format.ts"; import { loadPlan } from "../plan/loader.ts"; diff --git a/runtime/src/embedder/instantiate.ts b/runtime/src/embedder/instantiate.ts index 497e463..41d2fb3 100644 --- a/runtime/src/embedder/instantiate.ts +++ b/runtime/src/embedder/instantiate.ts @@ -3,9 +3,9 @@ // DESIGN: the facade is **runtime-driven**. Every // camelCase name, every resource class and every import wrapper is built here, // at instantiate time, from the loaded plan's type tables — the plan already -// carries names, kinds and function types. Bindgen emits compile-time *types* -// that cast this facade; no generated code participates, so everything works -// fully untyped. +// carries names, kinds and function types. Bindgen emits types and a wrapper +// that verifies the world digest before delegating here; untyped callers can +// instantiate directly. // // Governing contract: contracts/embedder-api.md (all sections). Secondary: // contracts/plan-format.md for the wire shapes read here. @@ -62,16 +62,8 @@ import { type ElemCodec, Future, Stream } from "./streams.ts"; import { markSyncCallable, syncPayloadOf } from "./sync.ts"; /** - * Relay the per-declaration host-import marks from the embedder's function - * onto the wrapper the executor will actually receive, and return the - * wrapper. - * - * Every `#dispatcher` arm re-wraps the embedder's function in a closure, so a - * brand left on the original is INVISIBLE to `buildLoweredImport` — for suspending mark - * that surfaced as a `NeedsJspi`, for cancellation discard (`deferCancel()`) it would be a - * silently discarded commit, which is precisely the failure the brand exists - * to prevent. Both marks are relayed by the same helper so a third one cannot - * be added to one arm and forgotten in the other three. + * Preserve declaration-level suspension/cancellation marks through every + * facade wrapper; the executor sees only the outermost function. */ function relayMarks(from: unknown, to: F): F { if (isSuspending(from)) suspending(to); @@ -109,8 +101,7 @@ export interface ComponentArtifacts { * `translator` accepts the translator-shim wasm bytes (simplest; compiles * the shim per call) or an already-created `Translator` (preferred when * instantiating more than one component, or the same component more than - * once — create it once and reuse; translation itself is sub-millisecond - * warm, the wasm compile is the cost being shared). `requiredImports` + * once, to share the compiled shim and its instance). `requiredImports` * still needs a plan: translate explicitly when you want to inspect the * import surface before instantiating. */ @@ -163,7 +154,7 @@ export async function resolveArtifacts( } export interface EmbedderOptions { - /** Opt in to JSPI-backed suspension (see `InstantiateInput.jspi`). */ + /** Override automatic JSPI selection (see `InstantiateInput.jspi`). */ jspi?: boolean; /** Verify `plan.component.sha256` against the bytes (default true). */ verifyHash?: boolean; @@ -218,11 +209,8 @@ export async function instantiate( imports: facade.rawImports, jspi: opts.jspi, verifyHash: opts.verifyHash, - // THE ordering fix: the facade converted this plan in its constructor and - // wired its import wrappers against those very `ResourceTypeInfo` tokens. - // Host imports fire DURING instantiation (a core module's `start` - // function runs inside `runInitializers`), so the facade cannot wait for - // the handle to learn its own types. + // Share the facade's resource tokens with imports called by core start + // functions, before instantiateComponent returns a handle. loadedPlan: facade.loaded, }); facade.bind(handle); @@ -260,7 +248,7 @@ class Facade { readonly rawImports: HostImports = {}; readonly #resolver: ImportResolver; readonly #bindings = new Map(); - /** ResourceTypeInfo identity -> ResourceIndex (one index, many tokens). */ + /** ResourceTypeInfo identity -> ResourceIndex (many table aliases). */ readonly #tokenIndex = new Map(); /** * The converted plan — owned by the facade and handed to the executor, so @@ -271,10 +259,9 @@ class Facade { readonly loaded: LoadedPlan; readonly #bridge: ValueBridge; /** - * Releases for reps minted while lowering the CURRENT call's arguments. - * Argument lowering is synchronous and uninterrupted (no `await` between - * `#lowerScope = […]` and the reset), so a single slot is race-free even - * with concurrent export calls in flight. + * Releases collected during synchronous argument lowering. #lowerParams + * saves/restores this slot for reentrant lowering; each call retains its + * own release list after the collection window ends. */ #lowerScope: (() => void)[] | null = null; /** ResourceIndex -> registry, for diagnostics (see INTERNAL_HOST_REGISTRIES). */ @@ -288,15 +275,9 @@ class Facade { ) { this.#resolver = new ImportResolver(providers); this.loaded = loadPlan(artifacts.plan); - // `ResourceTypeInfo` identity -> `ResourceIndex`. Both halves are static - // (the tokens are ours; `resourceTables` is wire data), so this map is - // complete before instantiation starts — a host import that fires from a - // guest `start` function can resolve resource types normally. - // - // One resource TYPE can be reached through several resource TABLES - // (plan-format.md "Type exports index into `resourceTables`": a type export's index is a table - // index, and the executor sets impl/dtor on every table whose `resource` - // matches), hence index-keyed bindings with tokens as aliases. + // Resolve identity before core start functions can call imports. Concrete + // tables naming one ResourceIndex share a token; table indices are aliases + // (plan-format.md "Type exports index into `resourceTables`"). artifacts.plan.resourceTables.forEach((table, i) => { if (table.kind !== "concrete") return; const token = this.loaded.resourceTokens[i]; @@ -329,14 +310,8 @@ class Facade { // -- resource-type identity ------------------------------------------------ /** - * Consistency check after instantiation. - * - * The facade no longer *learns* anything here — it handed its own - * `LoadedPlan` to the executor precisely so that nothing about types or - * resource identity depends on instantiation having finished. All this does - * is assert the executor did not silently re-load (which would give it a - * second, disjoint set of `ResourceTypeInfo` tokens and make every - * `own`/`borrow` unresolvable). + * Confirm the executor used the facade's loaded plan, not fresh resource + * tokens that would disagree with the import wrappers. */ bind(handle: ComponentHandle): void { if (handle.loadedPlan !== this.loaded) { @@ -462,12 +437,8 @@ class Facade { else self.#lowerScope.push(release); return rep; } - // Host `own` wrapper lowered as `borrow` (#86): record the lend - // for the duration of this call, so a `drop()` or a GC finalization - // in the window cannot destroy a rep the guest still borrows. - // definitions.py `lift_borrow` -> `Subtask.add_lender` (line 890); - // `#lowerScope` is released where that subtask delivers its - // resolution, i.e. when the call ends. + // Retain the rep until this call ends; explicit/GC drop must not + // destroy it while borrowed (lift_borrow -> Subtask.add_lender). const rep = takeRep(v, t.rt, false, `borrow<${b.name}>`); const release = lendWrapper(v as object); if (self.#lowerScope === null) { @@ -587,11 +558,8 @@ class Facade { ); } const dispatch = this.#dispatcher(leaf, provider); - // The function type is resolved LAZILY, on first call. It must come from - // the *executor's* loaded plan: the `own`/`borrow` types in it carry the - // per-instantiation `ResourceTypeInfo` identity tokens the bridge keys on, - // and those objects do not exist until `instantiateComponent` has run — - // which is after this wrapper has to be handed to it. + // Build the value adapter lazily from the facade's already-loaded types. + // The executor receives this same LoadedPlan, including for start calls. let impl: RawFn | null = null; const wrapper = (...raw: unknown[]) => { if (impl === null) { @@ -603,9 +571,7 @@ class Facade { } return impl(...raw); }; - // suspending mark/cancellation discard brand relay, layer 2 of 2 (see #dispatcher): the executor reads - // the brands off this wrapper, which is what lands in its hostImports - // record. + // The executor reads declaration marks from this outermost wrapper. return relayMarks(dispatch, wrapper); } @@ -660,19 +626,8 @@ class Facade { `${describe(fn)}); expected '${camelCase(m.name)}'`, ); } - // suspending mark/cancellation discard: the `suspending()` and `deferCancel()` brands ride the - // dispatch closure so #wrapLeaf can relay them onto the value the - // executor actually receives. - // - // suspending mark receiver rule: an interface member is invoked with its containing - // object as receiver (matching the static arm's `apply(cls)`), so a - // class INSTANCE is a fully supported spelling of an interface - // provider — methods reading instance state work. A world-level bare - // import has no containing object and stays unbound. (Previously the - // plain arm called extracted functions unbound: a class-instance - // provider type-checked, worked while stateless, and broke with - // `this === undefined` the moment a method touched state — the silent - // liberal-acceptance failure the contract forbids.) + // Interface members keep their provider as receiver; world-level + // functions stay unbound. Relay marks through the dispatch closure. const receiver = leaf.path.length === 0 ? undefined : provider; const dispatch: (args: unknown[]) => unknown = (args) => (fn as RawFn).apply(receiver, args); @@ -697,22 +652,10 @@ class Facade { // deno-lint-ignore no-explicit-any return (args) => new (cls as any)(...args); case "method": { - // suspending mark: the brand authority for an instance method is the CLASS - // PROTOTYPE, read at wrap time — the Suspending-wrap decision is - // per-declaration and taken at instantiation, before any instance - // exists. Instance-level method overrides do not change - // suspendability (marking follows the WIT declaration, not the - // object); the per-call lookup below still dispatches to the - // override's BODY as before. - // - // The probe must not INVOKE accessors: a platform getter (e.g. - // `URLSearchParams.prototype.size`) brand-checks its receiver, and a - // raw `prototype[member]` read runs it with `this` = the prototype — - // an engine TypeError at instantiation, even for guests that never - // call the member. Only a data-property function can carry the suspending mark - // mark (stage-3 method decorators install data properties), so an - // accessor-backed member yields no wrap-time function here and stays - // a call-time concern for the per-call lookup below. + // Declaration marks come from prototype data methods at instantiation; + // per-instance overrides change the body, not the declaration. Do not + // invoke accessors while probing a bare prototype. Actual method + // lookup remains a call-time operation on the receiver. const protoFn = dataMember( (cls as { prototype?: unknown })?.prototype, camelCase(m.member), @@ -752,13 +695,12 @@ class Facade { * The raw (definitions.py-shaped) function the executor lowers, wrapping a * conventions-shaped host implementation. * - * Error model (contract §"Error model"), the inversion of jco's convention: + * Error model (contract §"Error model"): * * a returned value is the ok side; * * `throw new ComponentException(payload)` is the err side of a `result`; * * a `Trap` passes through unchanged; * * **any other throw is a host bug and becomes a trap naming the import** - * — never a guest-visible err. This is what makes the consumers' - * defensive `platformCall`-style wrappers unnecessary by construction. + * — never a guest-visible err. */ #wrapImportFn( leaf: ImportLeaf, @@ -792,15 +734,8 @@ class Facade { value: rt.error === null ? null : fromHost(e.payload, rt.error, o), }; } - // Every remaining branch traps the component. The import's lifted - // stream/future arguments were transferred to the host when the params - // were converted (the guest's ends are gone), and a trapping import is - // a declared host bug — nothing owns them anymore, so drop them here: - // a peer parked on one (a host writer feeding the stream this import - // just received, the #66 E2 shape) settles with the truthful "reader - // went away" instead of hanging forever. The err-VALUE branch above - // deliberately does NOT do this: a fallible import returning err is a - // normal outcome whose implementation may retain the handles. + // Trap paths abandon top-level async arguments transferred to the host. + // A normal result error does not: its implementation may retain them. releaseAsyncArgs(args); if (isTrap(e)) throw e; if (isComponentException(e)) { @@ -809,10 +744,7 @@ class Facade { `only a fallible import may signal an error value`, ); } - // The #83 signature: in a graph with several copies, an UNBRANDED throw - // is usually a pre-module identity copy's `ComponentException` (its brand rode class identity, - // which does not survive the copy boundary). Say so rather than leaving - // the latent puzzle that motivated §"Module identity and @polyengine/protocol". + // Include copy diagnostics for unbranded host failures. const census = copyCensus(); throw new Trap( `${where} threw ${describeThrow(e)}. An unbranded throw from a host ` + @@ -831,15 +763,8 @@ class Facade { const args = ft.params.map((p, i) => toHost(raw[i] as ComponentValue, p, o, scope) ); - // CONTRACT: anything the executor appended PAST the WIT-declared - // params is a runtime-minted extra, not a component value — today - // exactly the `abortable()` signal `createLoweredImport` adds for a - // marked import. It is forwarded verbatim (no `toHost` conversion: it - // has no `ValType` and must reach the host as the platform object it - // is). Without this the facade would silently drop the signal and a - // marked import's `signal` parameter would be forever `undefined` — - // the failure the mark exists to prevent. The slice is empty for every - // unmarked import, so no existing path changes shape. + // Extras beyond WIT params are runtime values, notably abortable()'s + // AbortSignal. Forward without component-value conversion. for (let i = ft.params.length; i < raw.length; i++) args.push(raw[i]); let out: unknown; try { @@ -849,18 +774,9 @@ class Facade { return fail(e, args); } if (isThenable(out)) { - // When the WIT result type is `future`, a thenable - // return IS the future source ("for `future`, a `Promise` or - // `Future`" — §"Streams and futures"), not the call's async - // completion. The import completes immediately with the lowered - // future; the producer settles it on its own schedule. Without this, - // the natural spelling of the wasi:sockets 0.3 TCP `send` shape — - // `func(data: stream) -> future`, an async method whose - // promise resolves when transmission completes — would park the - // call, and a future whose settlement depends on post-return guest - // action (the guest writes `data` AFTER `send` returns) livelocks. - // This branch also covers a returned `Future` handle, which is a - // PromiseLike and would otherwise be adopted and mis-lowered. + // A future-typed result is the source, not async call completion. + // Lower it immediately: settlement may depend on guest work after + // this import returns. This also preserves returned Future handles. if (resultType !== null && resultType.kind === "future") { scope.end(); return ok(out); @@ -1065,9 +981,8 @@ class Facade { * Lower a call's arguments, collecting the releases for anything that was * allocated *for the duration of this call* (see `lowerBorrow`). * - * The collection window is the synchronous argument-lowering phase only — - * `#lowerScope` is set and cleared with no `await` in between — so a single - * slot is correct even with concurrent export calls in flight. + * Save/restore the collection slot for reentrant lowering. Release every + * borrow once, even if another release throws, then report the first error. */ #lowerParams( params: ValType[], @@ -1167,8 +1082,8 @@ class Facade { /** * Wrap one lifted export. * - * Uniformly Promise-shaped (contract §"Functions and async"): a sync - * completion resolves immediately, so there is one calling convention. + * Promise-shaped except for an eager Future handle in future-result + * position (contract §"Functions and async"). * A `result` in *function-result* position resolves `T` or rejects * `ComponentException`; a result nested inside a value is plain `{kind, value}` data * and never throws. @@ -1188,7 +1103,6 @@ class Facade { // PromiseLike, so `await` still yields `T`. const element = resultType.element; wrapper = (...args: unknown[]): Promise => { - // Advisory 9: the generic branch checks arity; so must this one. if (args.length !== ft.params.length) { throw new TypeError( `${where}: expected ${ft.params.length} argument(s), got ` + @@ -1246,9 +1160,7 @@ class Facade { return toHost(raw as ComponentValue, resultType, o); }; } - // sync() brand (contracts/embedder-api.md §"Functions and async"): every - // returned wrapper is branded, additively — the default Promise-shaped - // surface above is unchanged either way. + // sync() selects the synchronous form without changing the default wrapper. if (ft.async === true) { markSyncCallable(wrapper, { kind: "async" }); } else { @@ -1282,11 +1194,8 @@ class Facade { SYNC_ENTRY ] as RawFn | undefined) ?? fn; const unreachableThenable = (raw: unknown): never => { - // Defensive (see the dispatch prompt / sync() failure ladder): a genuine - // park through a plain entry surfaces as a trap, `NeedsJspi`, or - // `SyncEntryBusy` — never a settled thenable VALUE. A silent - // Promise-as-value here would corrupt lifting rather than fail loudly, - // so this is a diagnostic backstop, not a documented outcome. + // A plain entry must refuse suspension, not return a thenable that + // would be mistaken for a component value. void raw; throw new Error( `${where}: the sync entry returned a thenable, which should be ` + @@ -1431,13 +1340,10 @@ function isThenable(v: unknown): boolean { } /** - * Drop the lifted stream/future arguments a trapping import abandoned (#66). - * Top-level parameters only: those are the shapes whose peers park host - * operations; a stream nested inside a record is exotic enough to leave to - * the negligence rules. Uses the teardown drop, not the plain one — the - * calling instance is about to be poisoned by this very trap, and a DROPPED - * notification must not queue a phantom event into its waitables (review - * B2; see task/streams.ts `dropSharedForTeardown`). + * Drop top-level stream/future arguments abandoned by a trapping import. + * Nested handles are not traversed. Teardown suppresses notifications only + * for peers already marked poisoned/retired; it does not anticipate the + * caller's later poisoning (task/streams.ts dropSharedForTeardown). */ function releaseAsyncArgs(args: unknown[]): void { for (const a of args) { diff --git a/runtime/src/embedder/mod.ts b/runtime/src/embedder/mod.ts index 7f947f6..b7d2eb1 100644 --- a/runtime/src/embedder/mod.ts +++ b/runtime/src/embedder/mod.ts @@ -2,9 +2,8 @@ // // The host-facing surface: camelCase facades, resource classes on both sides, // stream/future handles, version-canonical import resolution and the branded -// error model — all built at instantiate time from the plan's type tables, so -// the layer works fully untyped. Bindgen (a separate track) emits compile-time -// types that cast this facade; no generated code participates. +// error model, built from the plan's type tables for typed or untyped callers. +// Generated bindings verify their world digest and delegate to this facade. // Copy registration (contracts/embedder-api.md §"Module identity and // @polyengine/protocol"; issue #83). Runs at module evaluation, so @@ -26,13 +25,8 @@ registerRuntimeCopy({ export { COPY_URL, RUNTIME_VERSION } from "./copy.ts"; -// The host-ABI version (contracts/embedder-api.md §"The host-ABI surface and its -// version"): the runtime's exported surface is application-only. The -// courtesy re-exports (error classes, predicates, brands, `suspending`, -// realm crossing, the copy registry) are removed — host modules import that -// vocabulary from `@polyengine/protocol` directly. The runtime still -// registers its own copy on the census above; it just no longer hands out -// the registry API to callers of this module. +// Application machinery lives here. Host modules import ABI vocabulary +// (errors, brands, suspension marks, realm crossing) from @polyengine/protocol. export { artifactsFromEnvelope, @@ -53,25 +47,20 @@ export { requiredImports, } from "./imports.ts"; -// `NameCollisionError` is the one error class that stays here: it's raised -// while building an instantiation facade, before any handle/value exists — -// application machinery, not host-ABI vocabulary (contracts/embedder-api.md -// §"The host-ABI surface and its version", §"The host-ABI surface and its version"). +// Naming failures belong to facade construction/value adaptation, not host ABI. export { NameCollisionError } from "./errors.ts"; export { type ElemCodec } from "./streams.ts"; -// `createStream()` — the host-ABI version stream-pair factory (contracts/embedder-api.md -// §"The host-ABI surface and its version" / §"Streams and futures"): the -// `Stream.create()` static's application-surface spelling, since the -// concrete `Stream`/`StreamWriter` classes are no longer exported. Handle -// TYPES are spelled against `@polyengine/protocol`'s structural interfaces. +// The application creates pairs; host modules use protocol handle interfaces. import { Stream as InternalStream } from "./streams.ts"; import type { Stream as ProtocolStream, StreamWriter as ProtocolStreamWriter, } from "@polyengine/protocol"; +/** Create a stream/writer pair. Writer operations wait until passing the + * stream to a guest binds its element type. */ export function createStream(): { stream: ProtocolStream; writer: ProtocolStreamWriter; diff --git a/runtime/src/embedder/resources.ts b/runtime/src/embedder/resources.ts index e748ca4..f052408 100644 --- a/runtime/src/embedder/resources.ts +++ b/runtime/src/embedder/resources.ts @@ -1,16 +1,13 @@ // Resources as classes on both sides of the boundary // (contracts/embedder-api.md §"Resources"). // -// The raw boundary represents `own` / `borrow` as bare **reps** -// (cabi/handles.ts `liftOwn` returns `rh.rep`; the host never holds a table -// index). Embedders otherwise turn that into identity tables and -// hand-transcribed `[method]…` keys by hand. Both become runtime obligations -// here. +// The raw boundary uses bare reps, not table indices. This layer maps them +// to guest wrappers or the host's original instances. // // Ownership, per the contract's 2x4 table: // // | position | guest-implemented R | host-implemented R | -// | host receives own | new wrapper (host owns) | instance back, mapping released, NO dispose | +// | host receives own | new wrapper (host owns) | instance back, ownership released, no dispose | // | host receives borrow | wrapper valid for the call | instance, mapping kept | // | host passes own | wrapper invalidated | instance registered | // | host passes borrow | wrapper stays valid | rep reused/allocated | @@ -24,15 +21,8 @@ import { camelCase, pascalCase } from "./casing.ts"; import { markSyncCallable, syncPayloadOf } from "./sync.ts"; /** - * Internal state of a guest-resource wrapper. - * - * The KEY is the process-global `polyengine.resourceState/1` brand - * (contracts/embedder-api.md §"Module identity": it used to be a module-local `Symbol(...)`, on the now-repealed - * assumption that bundle and source runtimes are never mixed in one process — - * issue #83 showed they routinely are). The state SHAPE stays strictly - * runtime-internal, exactly as the module identity brand table notes: another copy may - * RECOGNIZE a wrapper, and must never read or write this object. `copyUrl` is - * what lets this copy tell its own wrappers from a foreign copy's. + * Shared recognition key, runtime-private state. Only `copyUrl` may be read + * to distinguish this copy's wrappers; another copy cannot operate the rep. */ const STATE = RESOURCE_STATE; @@ -47,26 +37,14 @@ interface WrapperState { rt: ResourceTypeInfo; className: string; /** - * Host-side `ResourceHandle.num_lends` (#86). The reference models a - * host-held `own` as a table entry whose `num_lends` is bumped every time - * it is lifted as a `borrow` (definitions.py `Subtask.add_lender`, line - * 890, reached from `lift_borrow`, line 1516) and decremented when the - * borrowing call's subtask delivers its resolution (`deliver_resolve`, - * line 902). `lift_own` and `canon_resource_drop` both trap while it is - * non-zero (lines 1508 / 2325). - * - * Here the host holds bare reps rather than table entries, so the counter - * lives on the wrapper. Its lifecycle point is the *lowering scope* of the - * call the wrapper was passed into (`instantiate.ts` `#lowerParams`), which - * is released exactly when that call ends — the host-side analogue of the - * subtask's resolve delivery. + * Host-side lend count, analogous to ResourceHandle.num_lends. Each + * lowering scope retains the rep until its call ends. Transfer as own is + * forbidden while lent; drop invalidates immediately but defers the dtor. */ lends: number; /** - * A drop (explicit or via the GC backstop) that arrived while `lends > 0`. - * The reference would trap; the host has no frame to trap into by then, so - * the drop is deferred to the last release instead of running the dtor - * under a live guest borrow (which is the use-after-free #86 reports). + * Explicit or GC-backstop drop deferred until the last lend releases. + * Unlike canon_resource_drop's busy trap, this host API queues destruction. */ pendingDrop: boolean; } @@ -77,13 +55,8 @@ export class GuestResource { declare [STATE]: WrapperState; constructor() { - // realm boundary (contracts/embedder-api.md §"Realm boundaries and - // structured-clone-safe forms"; issue #131): guest-resource wrappers are - // realm-local by principle (their machinery lives in the minting - // copy's tables, issue #129's identity rule) — the pill makes a raw - // structuredClone/postMessage of one throw instead of husking. NOTE: - // `makeWrapper` below mints via `Object.create`, which bypasses this - // constructor entirely — it installs the pill itself. + // Raw structured cloning must fail: wrappers depend on this realm's + // runtime state. makeWrapper bypasses this constructor and marks separately. defineRealmLocal(this); } @@ -98,8 +71,8 @@ export class GuestResource { } /** - * Backstop for leaked handles (docs/architecture.md §7). A wrapper that becomes unreachable - * without `drop()` still runs the guest destructor — late, but not never. + * Best-effort backstop for leaked handles (docs/architecture.md §7). + * Finalization is not guaranteed; callers should drop explicitly. */ const runBackstop = (s: WrapperState): void => { // Idempotence: `valid` is the single guard. A wrapper that was dropped, @@ -108,11 +81,8 @@ const runBackstop = (s: WrapperState): void => { if (!s.valid || !s.owns) return; s.valid = false; if (s.lends > 0) { - // A live guest borrow of this rep is outstanding (#86). Running the dtor - // now is exactly the use-after-free the reference forbids - // (definitions.py line 2325, `trap_if(h.num_lends != 0)`); the last - // `releaseLend` runs it instead. The closure held by the lowering scope - // keeps `s` alive, so the deferred drop is not lost with the wrapper. + // The lowering scope retains state until releaseLend can safely destroy + // the rep, even though the wrapper itself is unreachable. s.pendingDrop = true; return; } @@ -122,9 +92,7 @@ const runBackstop = (s: WrapperState): void => { const leaked = new FinalizationRegistry(runBackstop); /** - * Simulate the GC backstop firing for `w` (the FinalizationRegistry callback, - * verbatim). Test seam: real GC finalization is unschedulable, and #86 is - * precisely about what the backstop does in a window a test must control. + * Run the GC callback deterministically for a test of a live borrow window. * * @internal */ @@ -134,22 +102,10 @@ export function simulateFinalizationForTest(w: object): void { } /** - * Run a host-initiated drop of a guest `own` handle. - * - * The host holds a rep, never a table index, so there is nothing to remove - * from a handle table: the observable remainder of definitions.py - * `canon_resource_drop` for an owning handle is the lifted dtor call - * (`hostDtorCall`, exec/boundary.ts), with `caller = None` — a host-initiated - * call, `Store.invoke`'s `caller = None`. - * - * Never throws: the two callers are `drop()`/`[Symbol.dispose]()` — where a - * trap *is* reportable, so it propagates — and the FinalizationRegistry - * callback, where a throw would be swallowed by the engine with no - * diagnostic. `runHostDrop` is the latter's form: a trapping dtor poisons the - * implementing instance (which the lift harness does) and is additionally - * recorded on the store's host-failure channel, so the next driven call - * surfaces it instead of silently continuing on a half-destroyed instance - * (#86, second defect: the former `catch {}`). + * Run a deferred/backstop destructor without throwing into cleanup. Record + * a synchronous failure on the implementing store; hostDtorCall records + * asynchronous failures there too. Immediate explicit drops call it directly + * so their synchronous failures can propagate to the caller. */ function runHostDrop(s: WrapperState): void { try { @@ -179,13 +135,8 @@ export function initWrapper( } /** - * This copy's state for a wrapper, or `undefined`. - * - * A wrapper minted by ANOTHER copy carries the same (process-global) brand key - * but its state belongs to that copy — reading it here would be reading a - * foreign copy's private shape. So it is not a state: it is - * `undefined` here, and `requireLive` turns that into the named cross-copy - * error rather than a misleading "not a resource handle" / "not live". + * Return this copy's wrapper state only. requireLive distinguishes a foreign + * brand from an unbranded object without interpreting foreign resource state. */ export function wrapperState(w: object): WrapperState | undefined { const s = (w as unknown as Record)[STATE]; @@ -236,24 +187,12 @@ function dropWrapper(w: GuestResource): void { leaked.unregister(w); if (!s.owns) return; // a borrow was never ours to drop if (s.lends > 0) { - // Lent out to an in-flight guest call (#86): defer rather than destroy a - // rep the guest still holds a `borrow` of. `drop(): void` stays - // non-blocking either way — the deferred dtor runs from `releaseLend`. + // Invalidate now; destruction waits for the last in-flight borrow. s.pendingDrop = true; return; } - // The dtor runs as an ordinary LIFTED sync call (`hostDtorCall`, #160): - // definitions.py `canon_resource_drop` (line 2319) lifts it with - // `CanonicalOptions(async_ = False)` rather than calling it bare, and that - // is what gives the activation a Task/Thread. A dtor that suspends (a - // `promising`-entered dtor calling a `Suspending` import, - // docs/architecture.md §7) therefore releases the implementing instance's - // entry bracket at its first park, so the scheduler can resume it — the - // old held-bracket form wedged exactly there (#160). - // - // `drop(): void` stays non-blocking: an unfinished dtor's tail is driven - // by the store like any other parked activation, and a failure that has no - // frame to return into is parked on `store.hostFailure`. + // A canonical lifted destructor gets a Task/Thread, not a bare JS call. + // Drop does not await its tail; the store drives it and records late failure. hostDtorCall(s.rt, s.rep); } @@ -261,8 +200,7 @@ function dropWrapper(w: GuestResource): void { * Record that a host-held `own` wrapper was lowered as `borrow` into a * guest call, and return the (idempotent) release for the end of that call. * - * definitions.py: `lift_borrow` -> `Subtask.add_lender` (line 890) on the way - * in, `Subtask.deliver_resolve` (line 902) on the way out. + * Analogue of lift_borrow -> Subtask.add_lender / deliver_resolve. */ export function lendWrapper(w: object): () => void { const s = wrapperState(w); @@ -320,7 +258,7 @@ export function takeRep( `${what}: a borrowed ${s.className} handle cannot be transferred as own`, ); } - // definitions.py `lift_own` (line 1508): `trap_if(h.num_lends != 0)`. A + // definitions.py `lift_own`: `trap_if(h.num_lends != 0)`. A // handle currently lent to an in-flight call cannot be transferred away. if (s.lends > 0) { throw new InvalidHandleError( @@ -360,14 +298,9 @@ export interface GuestResourceSpec { } /** - * Build (once, at class-build time — never per call) the Promise-shaped - * wrapper for one method/static's raw lifted function, exactly as - * `Facade#wrapExportFn` would for a plain export. `buildGuestResourceClass` - * reads the sync() brand off the returned wrapper to install the matching - * `"method"`/`"free"`/`"async"` brand on the class member it builds around - * it — the wrapper itself IS what a per-call closure invokes, so a `self` - * receiver is `wrapper(self, ...args)` for a method the same way a bare - * export is `wrapper(...args)`. + * Wrap a method/static once at class construction, using Facade's export + * conventions. Methods prepend self; the wrapper's sync brand is relayed to + * the class member with a method-specific receiver requirement. */ export type ExportWrapper = ( raw: (...a: unknown[]) => unknown, @@ -380,11 +313,9 @@ export type ExportWrapper = ( /** * Build the class for a guest-implemented resource. * - * The JS constructor is **synchronous**: a JS constructor cannot return a - * Promise, so the contract's "exports are uniformly Promise-shaped" rule has - * one unavoidable exception here. A guest constructor that does not complete - * synchronously is reported as such rather than silently returning a - * half-built object (see the report's contract-friction list). + * Construction must finish synchronously to return a usable resource wrapper. + * A guest constructor returning a thenable is refused; use an async factory + * for asynchronous construction. */ export function buildGuestResourceClass( spec: GuestResourceSpec, @@ -458,10 +389,7 @@ export function buildGuestResourceClass( for (const m of spec.methods) { const js = camelCase(m.member); const where = `${className}.${js}`; - // Built ONCE at class-build time (sync(): "prototype methods and statics - // must carry the brand at class-build time, not per call") — every - // instance's method call goes through this same wrapper, receiver - // (`self`) prepended. + // Share one branded wrapper across instances, prepending self per call. const wrapped = wrapExport(m.raw, m.params, m.results, m.async, where); const methodFn = function (this: GuestResource, ...args: unknown[]) { // params[0] is the `borrow`/`own` self. @@ -469,11 +397,7 @@ export function buildGuestResourceClass( }; const payload = syncPayloadOf(wrapped); if (payload !== undefined) { - // A resource method's sync form takes `self` as its first argument — - // exactly `wrapped`'s own synchronous form (params[0] IS self), so the - // "method" brand's `fn` is `payload.fn` verbatim, just re-tagged so - // `sync()` knows this one needs `sync(instance)` rather than being - // callable bare. + // The sync form already takes self; retag it to require sync(instance). markSyncCallable( methodFn, payload.kind === "free" ? { kind: "method", fn: payload.fn } : payload, // kind "async": pass the brand through unchanged @@ -532,10 +456,9 @@ export function makeWrapper( /** * Runtime-owned instance <-> rep mapping for a host-implemented resource. * - * The rep->instance direction is a **strong** map for exactly as long as the - * guest holds handles: the guest's handle is the only reference keeping a - * host object alive across calls, and a weak map here would let it be - * collected under the guest's feet. + * Strongly retain the instance while guest-owned or borrowed by any host- + * originated call. Returning own releases ownership, not outstanding borrows. + * A guest drop defers disposal until all borrows release. * @internal — runtime-owned instance<->rep mapping; hosts supply a class, not * a registry. */ @@ -629,8 +552,8 @@ export class HostResourceRegistry { } /** - * An `own` arrived from the guest: the host gets its instance back, the - * guest's handle is gone, and **no dispose runs** (the contract's 2x4 table). + * Return the host's instance without disposal. Keep its mapping while any + * host-originated borrow remains, even though guest ownership has ended. */ release(rep: number): object { const inst = this.lookup(rep); @@ -641,8 +564,9 @@ export class HostResourceRegistry { } /** - * The guest dropped its last own handle: run the destructor. This is the - * `HostResourceType` dtor the executor calls from `canon_resource_drop`. + * Guest drop: dispose now or after the last host-originated borrow. + * Pending disposal prevents re-transfer as own; the final release reports + * any disposal failure after removing the mapping. */ dtor(rep: number): void { const entry = this.#byRep.get(rep); @@ -656,7 +580,7 @@ export class HostResourceRegistry { (inst as { [Symbol.dispose]?: () => void })[Symbol.dispose]?.(); } - /** Live handle count — diagnostics and tests. */ + /** Retained mapping count, not handle count; diagnostics and tests. */ get liveCount(): number { return this.#byRep.size; } diff --git a/runtime/src/embedder/streams.ts b/runtime/src/embedder/streams.ts index 91682bf..ca46e59 100644 --- a/runtime/src/embedder/streams.ts +++ b/runtime/src/embedder/streams.ts @@ -48,10 +48,6 @@ import { import { describeCrossCopy } from "./copy.ts"; import { DroppedError, PeerTrappedError } from "./errors.ts"; -// `Chunk` moved to `@polyengine/protocol` (§"The host-ABI surface and its version", §"The host-ABI -// surface and its version"); re-exported here so this module's existing -// export surface (and therefore embedder/mod.ts's, unchanged in this track) -// keeps working. export type { Chunk } from "@polyengine/protocol"; /** @@ -74,38 +70,19 @@ export interface ElemCodec { readonly where?: string; } -// `StreamProducerError`'s canonical definition moved to `@polyengine/protocol` -// with §"Module identity and @polyengine/protocol" (it is an embedder-contract value: recognition must -// survive multiple runtime copies, issue #83). Re-exported here so every -// existing import path is unchanged. export { StreamProducerError } from "@polyengine/protocol"; /** - * Failures recorded against a shared stream object whose driving store could - * not be reached (the stream was never lowered, or the store already carries a - * failure). Surfaced on the next interaction with the handle. + * Producer failures retained per shared object for subsequent handle checks, + * independently of whether a store also received the failure. */ const producerFailures = new WeakMap(); /** - * Report a producer failure on the channel that can actually attribute it. - * - * PRIMARY channel: `store.hostFailure`. This is the runtime's existing - * host-side failure slot — `driveAsync` checks it after every tick and throws - * it out of the driving loop (exec/boundary.ts:468/647/679/692), which is the - * driving loop of *the export call that is consuming this stream*. So the call - * that would otherwise have resolved with truncated data rejects with this - * error instead. It is the same channel `HostActivity.#pumpAsync` already uses - * for a trap raised while pumping between export calls - * (exec/host_streams.ts:284), so the two host-side stream failure paths agree. - * - * The report happens BEFORE the write end is dropped: the drop is what lets - * the guest observe end-of-stream and resolve, and `driveAsync` checks - * `hostFailure` before it checks `done()`. - * - * FALLBACK: no store bound (the stream was never lowered), or the store - * already carries an earlier failure. Then the cause is recorded against the - * shared object and raised on the next interaction with the handle. + * Record on the handle and, if bound, the store's first-failure slot. Return + * whether a store exists. Report before dropping the writer: driveAsync + * checks hostFailure before completion, so truncation cannot hide the cause. + * The slot is store-wide, not an attribution to a particular consuming call. */ function reportProducerFailure( host: HostStream, @@ -159,12 +136,7 @@ export function isU8Element(element: ValType | null): boolean { return element !== null && despecialize(element).kind === "u8"; } -// Re-exported so embedders reach the direct-access byte edge callback shapes from this layer too -// (contracts/embedder-api.md §"Streams and futures"). -// Canonical definitions moved to `@polyengine/protocol` with §"The host-ABI surface and its version" -// (§"The host-ABI surface and its version"); `exec/host_streams.ts` keeps its -// own structurally-identical copies for the low-level seam, so both layers -// agree without either importing the other. +// Protocol callback shapes also match the low-level exec seam structurally. export type { DirectDestination, DirectSource, @@ -202,10 +174,7 @@ export class Stream implements ProtocolStream { private constructor(host: HostStream | null, codec: ElemCodec | null) { this.#host = host; this.#codec = codec; - // realm boundary (contracts/embedder-api.md §"Realm boundaries and - // structured-clone-safe forms"; issue #131): the realm-local pill — - // stateful handles must fail loud (DataCloneError) at a raw - // structuredClone/postMessage instead of husking silently. + // Raw structured cloning must fail rather than lose private handle state. defineRealmLocal(this); } @@ -223,15 +192,9 @@ export class Stream implements ProtocolStream { } /** - * `Stream.create(): { stream, writer }` — the writer-side host end the - * contract names. - * - * The element type is deliberately NOT a parameter: the embedder does not - * have one (a `ValType` is a runtime-internal shape) and the *lowering site* - * always does. So the shared object is created lazily, at the moment the - * stream is passed to a guest, and writer operations issued before that park - * until then. A stream created and written but never passed anywhere simply - * never completes — the same honest hang the low-level layer documents. + * Create a stream/writer pair. The lowering site supplies its runtime + * element type when the stream is passed to a guest. Writer operations + * issued earlier wait for that binding, indefinitely if it never happens. */ static create(): { stream: Stream; writer: StreamWriter } { const stream = new Stream(null, null); @@ -286,15 +249,8 @@ export class Stream implements ProtocolStream { "or use the writer, which parks until then", ); } - // Post-transfer refusal (#162, contracts/embedder-api.md §"Streams and futures"). Lifting - // removes the handle from the source table and lowering installs it in - // the destination's (definitions.py `lift_async_value` line 1530, - // `lower_stream` line 1828): once this handle's shared object has been - // passed to a guest, the guest owns the readable end and a host read here - // would operate a phantom duplicate of it. Refuse loudly instead. - // `StreamWriter` is deliberately unaffected — the host retains the - // writable end, and writing after the pass is the normal stream/future round-trip pattern — - // and `drop()`/`cancelRead()` stay permissive. + // Transfer relinquishes this handle's readable end, not StreamWriter's + // writable end. drop/cancelRead remain available for cleanup. if (this.#consumed) { throw new TypeError( "this Stream handle has already been passed to a guest; the guest " + @@ -340,7 +296,7 @@ export class Stream implements ProtocolStream { * writer's parked operation survives and the stream stays alive. * * Refusals mirror `read`: an unbound `Stream.create()` handle and a handle - * already passed to a guest (the deadlock-verdict suppression transfer guard) both throw, as does a + * already passed to a guest both throw, as does a * non-`u8` element type. */ async readDirect( @@ -352,12 +308,8 @@ export class Stream implements ProtocolStream { requireU8Direct(this.#codec, "readDirect"); const info: DirectSessionInfo = { endedByVerdict: false }; const n = await host.readable.readDirect(consume, info); - // loud component fault precision, `read`'s rule adapted: a session the CONSUMER itself - // ended with `"done"` genuinely completed and keeps its resolution. Any - // other way out (the writer dropped, the session was cancelled, the - // retirement walk settled us) is a settle-path this consumer did not - // cause — so if the peer's instance trapped, reject with the delivered - // count rather than fake a clean end. + // Preserve a callback-completed result. Otherwise report peer poisoning + // with the acknowledged byte count, not a clean session end. if (!info.endedByVerdict) throwIfPeerTrapped(host.value, where, n); return n; } @@ -378,17 +330,9 @@ export class Stream implements ProtocolStream { } /** - * Cancel an in-flight `read` (R-fix review advisory 1). - * - * #97, DELIBERATE AND PINNED: the cancelled `read` resolves with whatever - * had already arrived — typically the empty chunk, which this layer also - * uses as end-of-stream (`read`'s contract, and hence `readable()` and the - * async iterator, which close on it). **A cancelled read is therefore - * indistinguishable from EOS at this layer.** Kept as-is rather than given - * a distinct signal: the caller of `cancelRead()` is the same code that - * observes the read's result, so it already knows which happened, and only - * that caller can reach the state. See exec/host_streams.ts - * `HostReadableEnd.cancelRead` for the mechanism. + * Cancel an in-flight read, resolving with progress so far. An empty + * cancelled chunk is indistinguishable from EOS, so readable() and the + * async iterator close on it. A direct session resolves its byte count. */ cancelRead(): void { this.#host?.readable.cancelRead(); @@ -403,19 +347,9 @@ export class Stream implements ProtocolStream { } /** - * @internal — teardown after a trapping import abandoned this handle - * (#66, instantiate.ts `releaseAsyncArgs`). Unlike `drop()`, this goes - * through `dropSharedForTeardown`, whose parked-side discipline never - * wakes the about-to-be-poisoned caller (review B2: a plain drop queued a - * DROPPED event into the trapping instance's waitables, and a later - * driving loop asserted on the corpse). - * - * The arm is released on this path too (#162, §"Streams and futures"): the wrapper's - * `HostActivity` now closes through the shared object's drop observers, - * which `dropSharedForTeardown` fires unconditionally — so a teardown with - * nothing parked no longer leaves the arm outliving the stream. (This - * paragraph previously recorded that asymmetry as a known, non-blocking - * review advisory.) + * @internal — teardown of an abandoned import argument. Silently retract + * parked ends already marked poisoned/retired; notify healthy peers. + * Shared drop observers close host activity even with nothing parked. */ dropForTeardown(): void { if (this.#dropped) return; @@ -476,7 +410,7 @@ export class StreamWriter implements ProtocolStreamWriter { * (taken as already-lowered bytes), and a plain array of any element type * is lowered per element. u8 chunks travel as `Uint8Array` all the way to * the CABI store's bulk path (issue #54) — which makes a `Uint8Array` - * chunk a BORROW until the returned promise settles; mutating it in that + * chunk a borrow until the returned promise settles; mutating it in that * window is misuse. Plain-array chunks are lowered (copied) up front. */ write(values: Chunk): Promise { @@ -526,8 +460,9 @@ export class StreamWriter implements ProtocolStreamWriter { * Resolves with the session's total byte count. * * `"done"` with zero bytes marked *retracts* (the session ends, the - * reader's operation stays parked, no event — the speculative-park - * correction); `"more"` with zero marked, and a throwing callback, reject. + * reader's operation stays parked, no event); `"more"` with zero marked, + * and a throwing callback, reject. Marks commit only on clean return; + * writes already made through the view are not rolled back on failure. * * Parks until the element type is known, exactly as `write` does — a * `Stream.create()` writer has no element type until the lowering site @@ -543,10 +478,8 @@ export class StreamWriter implements ProtocolStreamWriter { requireU8Direct(this.#stream.codec, "writeDirect"); const info: DirectSessionInfo = { endedByVerdict: false }; const n = await host.writable.writeDirect(produce, info); - // loud component fault precision, `write`'s short-take rule adapted: a session the PRODUCER - // itself ended with `"done"` keeps its resolution; every other way out is - // a settle-path the producer did not cause, so a trapped peer rejects - // here carrying the delivered count. + // Callback completion survives a later peer trap; other endings report + // poisoning with the acknowledged byte count. if (!info.endedByVerdict) throwIfPeerTrapped(host.value, where, n); return n; } @@ -627,19 +560,10 @@ export class Future implements ProtocolFuture { } /** - * A future that is still in flight: the guest call that produces it has not - * resolved yet. - * - * CONTRACT (contracts/embedder-api.md): §"Functions and async" makes every - * export Promise-shaped, and §"Streams and futures" makes `Future` a - * `PromiseLike`. For an export whose *result* is a `future` those two - * collide irreducibly: JS promise resolution unconditionally adopts a - * thenable, so `await someExport()` can never hand back a thenable handle — - * it hands back the value the handle would have yielded. Conservative - * reading, implemented here: the export returns the handle **eagerly** (it - * is itself PromiseLike, so `await` still works and still yields `T`), which - * keeps `drop()`/`cancel()` reachable for a caller that does not await. The - * alternative — resolving a Promise *to* the handle — is not expressible. + * Return a handle before its producing guest call resolves. A Promise + * cannot resolve to Future without adopting its thenable value, so exports + * return this handle eagerly. Callers can drop/cancel it without awaiting; + * awaiting yields T, not the handle. */ static deferred( pending: Promise, @@ -655,14 +579,8 @@ export class Future implements ProtocolFuture { finish?.(false, e); throw e; }); - // Backstop (issue #182): a deferred handle that is never awaited, - // dropped, or cancelled still has `#hostP` sitting there uninspected — if - // the producing export call rejects, that is an unhandled rejection at - // the process level with no handle-level operation to blame. Attach a - // no-op rejection handler to a SEPARATE derived promise; `#read()` above - // still awaits the original `hostP`, so a real failure still surfaces to - // an awaiter (or through `cancel()`/`drop()`'s own swallows) exactly as - // before. + // Observe an unused handle's producing-call rejection without replacing + // hostP: a later await must still receive the original failure. hostP.catch(() => {}); const f: Future = new Future(null, hostP, codec); return f; @@ -690,12 +608,8 @@ export class Future implements ProtocolFuture { } #read(): Promise { - // Post-transfer refusal (#162, §"Streams and futures"), the `Stream.read` mirror: - // once this handle was passed to a guest, the guest owns the readable end - // and a host read would operate a phantom duplicate. A read MEMOIZED - // before the transfer keeps resolving — it genuinely happened while the - // host still owned the end. Rejected rather than thrown: this runs under - // `then()`, where a synchronous throw escapes the promise chain. + // No new reads after transfer; a read memoized before transfer keeps its + // result. Return a rejection to preserve then()'s Promise-shaped failure. if (this.#consumed && this.#settled === null) { return Promise.reject( new TypeError( @@ -739,8 +653,8 @@ export class Future implements ProtocolFuture { } /** - * Release this future handle. Total and idempotent (#90): it never throws, - * and calling it twice — or after `Symbol.dispose` — is a no-op. + * Release this future handle. The embedder contract requires nonthrowing, + * idempotent disposal; repeated calls are no-ops. * * Dropping a future the host never wrote to, once the guest already holds * its readable end, is **abandonment**: the guest's reader can never be @@ -748,7 +662,7 @@ export class Future implements ProtocolFuture { * being handed a value-less completion (exec/host_streams.ts * `HostFuture.drop`, task/streams.ts `abandonSharedFuture`; the spec keeps * that state unreachable by trapping the early writable drop, - * definitions.py:1183-1184). Write-then-drop is the normal path and is + * definitions.py `WritableFutureEnd.drop`). Write-then-drop is the normal path and is * unaffected; a future no guest ever saw is plain cleanup. */ drop(): void { @@ -801,13 +715,8 @@ export class ErrorContext implements ProtocolErrorContext { } } -// module identity brands (contracts/embedder-api.md §"Module identity"): the STATEFUL -// embedder-facing handle classes. Their machinery lives in the copy that -// minted them, so the brand never makes a foreign handle usable — it makes -// it DIAGNOSABLE, at the lowering sites below. `StreamWriter` gains its -// brand with §"The host-ABI surface and its version" (§"The host-ABI surface and its version"): -// writers carried none before because nothing needed to recognize one, and -// `isStreamWriter` now does. +// Brands recognize handles across copies, not share their machinery. +// ErrorContext is the exception: its public message can be copied by value. defineBrand(Stream.prototype, STREAM); defineBrand(StreamWriter.prototype, STREAM_WRITER); defineBrand(Future.prototype, FUTURE); @@ -826,23 +735,18 @@ export type FutureSource = Future | PromiseLike | T; /** * Adapt a producer to a lowered `stream` value, and own the pumping. * - * The driving arm auto-closes on end (the pump drops the write end when the - * producer is exhausted) and on `DROPPED` (host_streams settles the activity - * arm) — R-fix review advisory 2, the deadlock-masking activity-lifetime - * footgun. + * End or drop closes the activity arm; a finished producer cannot keep + * suppressing the store's deadlock verdicts. */ export function lowerStreamSource( src: StreamSource, codec: ElemCodec, ): ComponentValue { - // Order matters (§"Module identity and @polyengine/protocol"). Same-copy handle: the fast path, unchanged. + // Preserve handle identity before considering producer adaptation. if (src instanceof Stream) { return src.takeValue(codec); } - // Branded but not ours: a `Stream` minted by ANOTHER runtime copy. Without - // this check it would fall through to producer adaptation below and be - // pumped by its async iterator — a silent downgrade that quietly voids stream/future round-trip's - // identity guarantees. Refused, loudly, naming both copies (issue #83). + // A foreign handle must not silently become an async-iterator copy. if (hasBrand(src, STREAM)) { throw new TypeError(describeCrossCopy( "this stream handle", @@ -859,11 +763,9 @@ export function lowerStreamSource( /** * Lower one chunk of stream elements. * - * u8 chunks come out as `Uint8Array` — either the caller's own (bytes are - * already canonical component values; issue #54's bulk-store path picks the - * typed array up unchanged at the rendezvous) or packed from a validated - * plain array. A `Uint8Array` offered to a NON-u8 stream keeps the legacy - * behavior: elements are fed through the per-element codec like any array. + * Borrow u8 typed chunks unchanged; validate and pack plain u8 arrays. + * Other element types pass through the per-element codec, including when + * supplied as a Uint8Array. A failed lowering releases the lowered prefix. */ function packChunk( values: readonly T[] | Uint8Array, @@ -899,12 +801,8 @@ async function pump( let failure: unknown; let failed = false; let produced = 0; - // resource stream cancellation companion: the pump learns of the reader dropping - // through short writes, but a producer PARKED on an external event (an - // accept-shaped source holding a live platform resource) offers no write - // to shorten — this notification is its only stop signal. It also fires - // on the loud component fault teardown walk and on our own end-of-pump drop (harmless: the - // loop has exited by then). + // A producer awaiting an external event has no write to shorten. Drop + // notification lets batches cancel its pending pull and release resources. const gone = new Promise((resolve) => host.writable.onDropped(() => resolve(READER_GONE)) ); @@ -953,10 +851,7 @@ async function pump( failure, ); } - // End of production == end of stream. Dropping unconditionally is what keeps - // the activity arm from outliving the data (R-fix advisory 2) and what stops - // a failed producer from hanging the guest forever; the failure has already - // been recorded on the store, so the call fails rather than resolving. + // Always end the stream and release activity after recording any failure. host.writable.drop(); } @@ -982,16 +877,11 @@ function releaseUntaken( } /** - * Normalize every accepted producer shape to an async iterator of batches, - * racing each pull against `gone` (resource stream cancellation): when the stream dies - * with the producer parked, a `ReadableStream` source is `cancel()`ed - * through its reader, and an (async-)iterable source gets its optional - * `cancel()` method invoked — the documented producer-cancellation hook — - * then its pending pull is drained so a straggler element the producer - * already minted still reaches the caller's release path. A source with no - * cancel hook keeps the pre-resource stream behavior: the pump stays parked until the - * producer's next element (or forever — the documented embedder-negligence - * hang class). + * Normalize producers to batches. On reader loss, cancel a ReadableStream + * through its reader, or invoke an async iterable's optional cancel hook. + * Drain an iterable's pending pull so a late element reaches the release + * path before iterator.return(). If cancellation cannot settle that pull, + * this cleanup can remain pending indefinitely. */ async function* batches( src: Exclude, Stream>, @@ -1063,11 +953,7 @@ export function lowerFutureSource( codec: ElemCodec, ): ComponentValue { if (src instanceof Future) return src.takeValue(); - // Branded but not ours (§"Module identity and @polyengine/protocol"). This one is the sharpest edge in the - // family: `Future` is a `PromiseLike`, so a foreign future would otherwise - // be adopted as a plain thenable and appear to work — exactly the silent - // path module identity bans, since the awaited value would ride the OTHER copy's - // machinery with no handle transfer at all. + // Reject foreign handles before thenable adoption can hide a by-value copy. if (hasBrand(src, FUTURE)) { throw new TypeError(describeCrossCopy( "this future handle", @@ -1080,23 +966,9 @@ export function lowerFutureSource( const v = await (src as PromiseLike); await host.write(codec.fromHost(v) as unknown as T); } catch (e) { - // The producer failed. `future` has no error channel of its own, so - // the guest could only ever see a bare drop — the cause goes on the - // store's host-failure channel instead, exactly as for streams, so the - // in-flight call fails with a site-named error. - // - // And then we do NOT drop -- for ATTRIBUTION, not for safety. Dropping - // here is now well-defined (#90: an unwritten, lowered future's drop - // abandons it and the guest reader traps at its rendezvous point, - // exec/host_streams.ts `HostFuture.drop`); the stale version of this - // comment claimed it would trip an internal invariant, which was true - // before the abandonment mechanism existed and is not true now. - // Reporting instead of dropping is still the better outcome: the - // store-level failure names the producer and the site, so the in-flight - // call fails with the real cause rather than with a generic - // "the writable end went away" trap. Only when there is NO store to - // report to (the future was never lowered) do we fall back to dropping, - // so nothing can hang forever. + // Report the producer cause rather than replace it with a generic + // abandonment trap. A bound store receives the failure; only an unbound + // future is dropped here. Reporting does not itself retire the future. const reported = reportProducerFailure( { value: host.value } as unknown as HostStream, codec.where ?? "future producer", diff --git a/runtime/src/embedder/sync.ts b/runtime/src/embedder/sync.ts index deb6ac5..6c50f5a 100644 --- a/runtime/src/embedder/sync.ts +++ b/runtime/src/embedder/sync.ts @@ -1,5 +1,5 @@ // `sync()` — the explicit synchronous view of a WIT-sync export (contracts/ -// embedder-api.md §"Functions and async", §"Functions and async", 2026-08-30). +// embedder-api.md §"Functions and async"). // // Placement: application machinery exported from // `@polyengine/runtime/embedder`, like `createStream` — only an instantiating @@ -187,18 +187,8 @@ function classView(cls: object): unknown { * function or a nested resource class/instance/record maps recursively; * anything else (including an unbranded function) passes through unchanged. * - * CONTRACT (contracts/embedder-api.md §"Functions and async" sync(), the - * `sync(record)` bullet): the bullet says a record's members are "mapped by - * these same rules, recursively" — read most literally, an async-typed - * member nested in a record should behave exactly as `sync(asyncFn)` does at - * top level, i.e. throw. But applying that EAGERLY while building the - * parent's view would make one unrelated async export in a real component's - * exports record (a normal mix — see contracts/embedder-api.md's own async + - * sync export examples) poison `sync(exports)` entirely, defeating the - * per-use adapter's whole purpose. The conservative reading kept here defers - * that failure to the point the caller actually reaches for the async - * member (`throwingMember`), never for members the caller never touches — - * every failure the contract mandates still happens, just lazily. + * Record views invoke this lazily, so an async member fails on access without + * preventing use of unrelated sync members in the same record. */ function mapMember(v: unknown): unknown { if (typeof v === "function") { @@ -243,16 +233,9 @@ function recordView(rec: object): unknown { * async export is not attempted (the contract only requires the runtime * error) — `Sync` stays structural. * - * CONTRACT: the naive `F extends Record` branch (checked - * before this fix) only matches object-LITERAL type aliases — named - * interfaces (generated `*Exports`) and class instance types (generated - * resource classes, e.g. `Counter`) have no implicit index signature and - * are not assignable to it, so they fell through to the `: F` passthrough - * and stayed Promise-shaped. Ordering matters: a non-Promise function type - * (e.g. a resource's `drop(): void`, or `[Symbol.dispose]`) must be checked - * and passed through BEFORE the generic `object` branch, or `{ [K in keyof - * F]: ... }` would try to map over a function's call signature (losing it) - * instead of leaving the function itself alone. */ + * Test non-Promise functions before objects to preserve call signatures. + * `object`, unlike Record, also accepts named interfaces + * and class instance types without requiring an index signature. */ export type Sync = F extends (...a: infer A) => Promise ? (...a: A) => R : F extends (...a: never[]) => unknown ? F // non-Promise functions (e.g. `drop(): void`) pass through unchanged @@ -261,7 +244,7 @@ export type Sync = F extends (...a: infer A) => Promise /** * The synchronous form of a WIT-sync export (contracts/embedder-api.md - * §"Functions and async", §"Functions and async"). + * §"Functions and async"). * * - `sync(fn)` — a lifted export function (plain export, interface member, * or resource static): returns the synchronous form `(...args) => T`. @@ -273,8 +256,9 @@ export type Sync = F extends (...a: infer A) => Promise * with every member mapped by these same rules, recursively; non-branded * members pass through unchanged. * - Views are stable: `sync(x) === sync(x)`. - * - An async-typed export, a bare resource-method function, or anything - * unbranded throws `TypeError`. + * - An async-typed export, bare resource-method function, unbranded top-level + * function, or primitive throws TypeError. In views, unsupported branded + * members fail on access; unbranded functions pass through. */ export function sync Promise>( target: F, diff --git a/runtime/src/embedder/values.ts b/runtime/src/embedder/values.ts index 42aa9c9..ae31716 100644 --- a/runtime/src/embedder/values.ts +++ b/runtime/src/embedder/values.ts @@ -16,9 +16,8 @@ // `Stream`/`Future`/`ErrorContext` handles. // // The adapter is driven entirely by the plan's `ValType`s — no generated code -// participates, which is what lets the same facade serve an untyped embedder -// and a bindgen-typed one (the design ruling: runtime-driven facade, -// compile-time-only bindgen). +// participates in value conversion. Generated wrappers can verify types and +// delegate instantiation; this adapter also serves untyped callers. import type { CaseType, @@ -137,12 +136,8 @@ export function checkNoCollisions( // Per-type adapter tables // --------------------------------------------------------------------------- // -// Everything below is a pure function of the type, so paying for it per -// element is paying for it once per element too many (issue #261). Same -// argument as `checkedLabels` above: `ValType` objects are stable for the -// lifetime of a loaded plan, so a `WeakMap` keyed on the type's identity turns -// a per-call cost into a one-time cost per type, and the table dies with the -// plan. Both directions read the same tables. +// Both directions share immutable type-derived tables, cached by ValType +// identity. Weak keys let the tables die with the loaded plan. interface RecordField { /** The WIT label — the key of the internal record object. */ @@ -181,14 +176,8 @@ function recordTable(t: RecordType): RecordTable { } /** - * Label -> case, memoized on the `cases` array. Replaces the linear - * `t.cases.find(...)` both directions ran per element. - * - * **First wins on a duplicated label** — deliberately NOT `caseIndexOf` from - * cabi/types.ts, which maps a duplicate to -1. The two sites replace scans - * with different pre-existing behavior (`find` takes the first match; the cabi - * scan asserted on exactly one), so each keeps its own; merging them would be - * a behavior change, not a deduplication. + * Label -> first matching case, memoized on the cases array. Unlike the CABI + * caseIndexOf helper, duplicate labels select the first rather than fail. */ const variantTables = new WeakMap>(); @@ -201,7 +190,7 @@ function variantTable(cases: CaseType[]): Map { return m; } -/** The enum's labels as a set, replacing `t.labels.includes(v)`. */ +/** Membership set for enum labels. */ const enumTables = new WeakMap>(); function enumTable(t: EnumType): Set { @@ -470,11 +459,8 @@ export function fromHost( (v as { message: string }).message, ) as unknown as ComponentValue; } - // Branded but no string `message` (§"Module identity and @polyengine/protocol", superseded above only - // for the message-valued case): a genuinely foreign stateful handle — - // it lives in another copy's handle table, so it can never be lowered - // here — but "recognized and named" beats the generic "expected an - // ErrorContext" that sent issue #83 hunting in the wrong direction. + // A foreign brand without a public string message cannot be + // reconstructed by value; report the copy boundary explicitly. if (hasBrand(v, ERROR_CONTEXT)) { throw new TypeError( `${o.where}: ${describeCrossCopy("this error-context")}`, @@ -653,9 +639,7 @@ function int( if (v < lo || v > hi) { throw new TypeError(`${o.where}: ${kind} out of range: ${v}`); } - // The raw boundary takes unsigned lane values for the signed types too? - // No: cabi `lowerFlatSigned32` does the two's-complement fold, so the - // interpreter wants the *signed* number here. Pass it through. + // Keep signed values signed; lowerFlatSigned32 performs lane conversion. return v; } @@ -685,12 +669,8 @@ function elemCodec( where: o.where, toHost: (v) => element === null ? undefined : toHost(v, element, o), fromHost: (v) => element === null ? null : fromHost(v, element, o), - // resource stream: `own` elements a producer lowered but the reader never took - // must be destroyed, not leaked — an un-taken element may hold a live - // platform resource (the tcp `listen` shape: an accepted connection). - // Top-level `own` is the supported element shape; nested owns inside - // composite stream elements remain out of scope until a consumer - // links one. + // Release untaken top-level own elements. This codec does not recursively + // clean up owned resources nested in composite elements. release: element !== null && element.kind === "own" ? (v) => o.bridge.dropOwn(v as number, element) : undefined, diff --git a/runtime/src/embedder/version.ts b/runtime/src/embedder/version.ts index e76f6b6..5965b85 100644 --- a/runtime/src/embedder/version.ts +++ b/runtime/src/embedder/version.ts @@ -1,7 +1,7 @@ // Version-canonical import resolution (contracts/embedder-api.md // §"Version canonicalization"). // -// Authorities, read before writing this: +// Authorities: // * Explainer.md §"canonical interface names" (`canonversion`) — the spec's // compatibility-track split; // * wasmtime-environ `component::names::{NameMap, alternate_lookup_key}`, @@ -255,13 +255,8 @@ export class ImportResolver { return { key: claim.key, value: this.#exact.get(claim.key) }; } } - // CONTRACT: contracts/embedder-api.md §"Version canonicalization" bans - // "unversioned folding": "unversioned keys -> error". Read - // conservatively: an unversioned key is still a legal - // *exact* match for an unversioned import (the ban is about folding - // distinct semver tracks together, and unversioned WIT interfaces - // exist), but it may never serve a *versioned* import. That attempt is - // refused loudly here rather than reported as a plain "not provided". + // Unversioned keys may match exactly, never serve a versioned import. + // Diagnose that registration mistake rather than a generic missing key. const un = this.#unversioned.get(p.base); if (un !== undefined) { throw new ImportResolutionError( diff --git a/runtime/src/exec/boundary.ts b/runtime/src/exec/boundary.ts index e0b6a94..2e146ce 100644 --- a/runtime/src/exec/boundary.ts +++ b/runtime/src/exec/boundary.ts @@ -1,7 +1,6 @@ -// Host-boundary wiring: lifted-export invocation (reference `canon_lift`, -// sync path) and lowered-import bodies (reference `canon_lower`, sync path), -// built on the cabi v1 interpreter (runtime/src/cabi/) driven by the plan's -// canonical options — docs/architecture.md §4.3 items 2 and 5, degenerate sync case. +// Canonical host boundary: sync, callback and stackful lifts, host-import +// lowers, destructor entries, and store drivers. See definitions.py +// `canon_lift` / `canon_lower` and docs/architecture.md §5-7. import { type CanonicalOptions, @@ -181,21 +180,9 @@ export interface ResolvedOptions { postReturn: (() => CoreFn | undefined) | null; callback: (() => CoreFn | undefined) | null; async: boolean; - /** - * `CanonicalOptions.cancellable` (wasmtime-environ 47.0.3 - * `component/info.rs:540`), i.e. whether a built-in reached through these - * options is a *cancellable* block point. - * - * It lives in the options, not in the trampoline: `Trampoline:: - * WaitableSetWait`/`WaitableSetPoll` carry only `{instance, options}` - * (info.rs:815-831). definitions.py takes it as the first parameter of - * `canon_waitable_set_wait` / `canon_waitable_set_poll` (lines 2414/2431), - * which is the same information arriving by a different route. - * - * (`thread.yield` and `subtask.cancel` are the exceptions: wasmtime puts - * their `cancellable` / `async` flags on the *trampoline*, and those - * built-ins read them from the decl.) - */ + /** Cancellability for option-indexed built-ins, including + * `canon_waitable_set_wait` / `canon_waitable_set_poll`. Other built-ins + * such as thread.yield carry their flags on the trampoline declaration. */ cancellable: boolean; coreType: CoreFuncType; instance: ComponentInstanceState; @@ -226,34 +213,15 @@ export function cabiOptions(opts: ResolvedOptions): CanonicalOptions { }, postReturn: null, // post-return handled by the task layer, not cabi async_: opts.async, - // Truthiness only: `flattenFunctype` branches on whether a callback - // exists (async lifts with a callback return a packed i32; stackful ones - // return nothing). Passing the resolver rather than `null` is what makes - // the callback-ABI core type come out right. + // Flattening tests callback presence, not its resolved function value. callback: opts.callback === null ? null : opts.callback, }; } /** - * Call a core function, mapping core-wasm exceptions to canonical-ABI traps - * (reference `call_and_trap_on_throw`). Component traps and internal errors - * of ours propagate unchanged. - */ -/** - * Layering rule: a core-wasm trap's message is engine-specific text (V8, - * SpiderMonkey, JSC each word `unreachable` differently, for instance) and is - * passed through here UNTOUCHED — it is diagnostics only, "engine-flavored" - * and not normalized to any particular host's wording. The runtime never - * emulates another host's (e.g. wasmtime's) message text. - * - * Suite-wording normalization (matching the official test suite's - * `assert_trap` expectations, which are typically worded per wasmtime) lives - * in the harness instead: see `TRAP_MESSAGE_EQUIVALENTS` in - * harness/src/runner.ts, which maps engine-specific spellings to the - * suite-expected forms at comparison time. (The FACT *adapter* traps take a - * different route entirely — they arrive as numeric codes through the `trap` - * trampoline and are runtime-authored text, see `FACT_TRAP_MESSAGES` in - * intrinsics/mod.ts; that table is untouched by this layering rule.) + * Call core wasm and map RuntimeError to a canonical trap (`call_and_trap_on_throw`). + * Preserve engine diagnostic text; suite wording normalization belongs to + * harness/src/runner.ts, not the runtime. Other exceptions pass through. */ export function callCore(fn: CoreFn, args: CoreValue[]): CoreValue[] { @@ -269,19 +237,8 @@ export function callCore(fn: CoreFn, args: CoreValue[]): CoreValue[] { } /** - * The `call_and_trap_on_throw` translation, factored so BOTH routes a core - * trap can take reach it: - * - * * a synchronous throw out of `fn(...args)` (`callCore` above — the plain - * path, and jspi pre-suspension); - * * a **rejection of a `promising` entry's Promise** (jspi pin (e): a trap - * after a resumption arrives as an ordinary rejection). That rejection - * carries the raw `WebAssembly.RuntimeError`, and before this helper was - * applied on the awaited path (`awaitCore` below), a post-suspension - * guest trap escaped to the embedder as `RuntimeError: unreachable` - * instead of the wasmtime-worded `Trap` — every deliberate guest trap - * under detection scored as a harness failure - * (big-interleaving-test.wast:836's assert_trap "unreachable"). + * Shared translation for synchronous core throws and promising-entry + * rejections: post-suspension traps arrive through the Promise path. */ function mapCoreException(e: unknown): unknown { if (e instanceof WebAssembly.RuntimeError) { @@ -336,37 +293,10 @@ function resultsToHost(results: ComponentValue[]): unknown { // Driving the scheduler from the host boundary // --------------------------------------------------------------------------- // -// run_tests.py's `lift_and_run` (line 55) is the reference embedding: -// -// ```python -// func_inst = inst.store.lift(callee, ft, opts, inst) -// _ = inst.store.invoke(func_inst, on_start, on_resolve) -// while inst.store.waiting: -// inst.store.tick() -// ``` -// -// i.e. enter the component, then pump the store until nothing is waiting. -// `drive` below is that loop, with two additions the reference does not need: -// -// 1. **A deadlock verdict.** The reference's `while store.waiting` spins -// forever if no waiting thread is ready, because its host functions run -// on real OS threads and always eventually make progress. Ours cannot -// spin: when no thread is ready and no host promise is outstanding, the -// task can never resolve, which is the same condition `canon_lift`'s -// sync loop traps on (`trap_if(not candidates)`), so we trap too. -// -// 2. **Host promises.** A host import implemented as an `async` JS function -// resolves its subtask on a *microtask turn*, not on a thread. When the -// only way forward is such a promise, `drive` returns a Promise and the -// lifted export's return value becomes a Promise. This needs no JSPI: -// the guest is stackless (callback ABI), so nothing is suspended mid-wasm -// — the guest already returned WAIT and the host merely resumes it later. -// -// Consequence for callers: a lifted export returns `T` when the whole call -// completed synchronously, and `Promise` when a host promise was involved. -// The conformance harness invokes exports synchronously -// (harness/src/runtime-executor.ts) and the official suite has no -// promise-returning host imports, so it only ever sees the synchronous shape. +// Unlike `canon_lift`'s instance-local sync loop, these embedding drivers +// service the whole store, including host promises and JSPI continuations. +// Callback-ABI host waits need no suspended wasm frame; JSPI entry hops also +// make a call asynchronous even when no host import returned a Promise. /** True for thenables, which is what "is this host call asynchronous" means. */ function isPromiseLike(v: unknown): v is PromiseLike { @@ -377,15 +307,8 @@ function isPromiseLike(v: unknown): v is PromiseLike { } // --------------------------------------------------------------------------- -// Handshake probe +// Optional driver tracing // --------------------------------------------------------------------------- -// -// Env-gated tracing of the drive loops. This exists because site 1 is the -// first *lit* suspension site, so the `SuspensionPoint` <-> `Store.tick` <-> -// `driveAsync` handshake had never executed before it; a pure-microtask stall -// there is invisible from the outside (no trap, no rejection -- just an await -// nothing settles). Off unless POLYENGINE_DRIVE_TRACE is set, and the getter is read -// once at module load so normal runs pay a boolean test. const DRIVE_TRACE = (() => { try { return Deno.env.get("POLYENGINE_DRIVE_TRACE") === "1"; @@ -446,44 +369,18 @@ function traceDrive( } /** - * What a driving loop does when it runs out of moves with `done()` still - * false — the reference's empty-candidate-set state. - * - * `"trap"` is definitions.py `canon_lift`'s `trap_if(not candidates)` (line - * 2189) and the default for every driver in this runtime: the sync-lift - * paths, the destructor entry, and the pumps (which cannot reach the verdict - * anyway — see `driveStoreAsync`). - * - * `"exit"` returns instead, with the verdict `"idle"` (`DriveExit`) for the - * caller to act on. - * It exists for ONE caller: an **async-typed** lifted export (#292). The - * reference's driving loop is guarded by `if not ft.async_`, so for such an - * export `canon_lift` returns right after the first `thread.resume()` and the - * driving — with it, the idle verdict — belongs to the embedder's - * `Store.tick`, which never traps. wasmtime draws the same line: - * `run_concurrent` is `poll_until(trap_on_idle=false)` (a `call_concurrent` - * future simply stays pending on idle), and the trapping variant - * `run_concurrent_trap_on_idle` is `pub(super)`, backing only the blocking - * `[Typed]Func::call_async`. Polyengine's Promise-shaped export is - * `call_concurrent` under an always-live `run_concurrent` (docs/architecture.md - * §"Mapping the reference model"), hence "exit". + * No-progress policy while `done()` is false. Sync-typed lifts trap, matching + * definitions.py `canon_lift`'s empty-candidate check. Async-typed lifts exit + * idle and leave their result Promise pending for a later driver; the + * reference does not run its sync loop for them. The harness can opt into + * trapping idle async calls. Pump predicates stop before the idle trap. */ type IdlePolicy = "trap" | "exit"; /** - * WHY a driving loop returned: `"done"` means `done()` held at the exit test, - * `"idle"` means the loop ran out of moves with `done()` still false and - * `idle: "exit"` let it return instead of trapping (only an async-typed lift - * asks for that; see `IdlePolicy`). - * - * The verdict is a RETURN VALUE, not something the caller re-derives, because - * `done` is a predicate over shared, time-varying store state: between a - * driver's exit test and its caller's continuation (one microtask later) - * another driver of the same store can flip it. That is polyengine#310 — a - * lift that exited `done` re-tested `driveDone()` in its `.then`, the - * settlement pump had meanwhile hop-parked a background activation, and the - * lift took the never-completing background path. The exit verdict is a fact - * about the past and cannot rot; the predicate can. + * Snapshot of why the driver exited: its predicate held, or it ran out of + * moves under `idle: "exit"`. Callers must use this verdict, not re-test a + * shared-store predicate after an await; another driver may have changed it. */ type DriveExit = "done" | "idle"; @@ -503,10 +400,7 @@ function drive( try { return driveLoop(store, done, what, idle); } catch (e) { - // EXIT BY EXCEPTION IS STILL AN EXIT. A trap unwinds this call, but the - // sibling work this loop already started — a registered host call, a - // queued tail (which gates `Store.tick`) — does not unwind with it. The - // `done()` path hands both to the settlement pump; so must this one. + // Sibling host calls and activation tails survive this driver's failure. ensureSettlementPump(store); throw e; } @@ -521,16 +415,8 @@ function driveLoop( ): DriveExit | Promise { for (;;) { traceDrive("drive", store, done, "top"); - // The synchronous drain must not run while a thread is parked on a - // Promise: `tick` cannot see those, so a thread that re-parks READY on - // every resume (a callback-ABI guest spinning YIELD) would hold this - // loop forever while the promise-parked thread that would stop the spin - // never gets serviced (drop-subtask.wast:139 under detection: the - // Looper spins YIELD until `return` runs, and `return`'s caller sat - // parked on its activation promise). In jspi mode the lifted export - // returns a Promise anyway, so handing off to `driveAsync` — whose - // drain interleaves fairly — costs nothing; in plain mode `awaiting` - // is always empty and this loop is bit-for-bit what it was. + // Promise-parked threads need microtasks; a synchronous YIELD loop would + // starve them. Hand those stores to the interleaved async drain. while (store.awaiting.size === 0 && store.tick()) { traceDrive("drive", store, done, "ticked"); if (store.hostFailure !== undefined) throw takeHostFailure(store); @@ -538,28 +424,19 @@ function driveLoop( if (store.hostFailure !== undefined) throw takeHostFailure(store); if (done()) { traceDrive("drive", store, done, "EXIT-done"); - // Fully-synchronous completion: no `driveAsync` ran, so its exit hook - // will not fire — arm the settlement pump here for any host calls the - // guest registered fire-and-forget during this drive. + // No async finally will run: hand off any background work here. ensureSettlementPump(store); return "done"; } - // A thread parked on a Promise (jspi) can only progress after a microtask - // turn, exactly like an outstanding host call. So can an outstanding - // pending resumption of THIS store: a suspension has been settled and its - // activation has not run yet (see `Store.tick`). + // Awaiting activations and pending resumptions need an event-loop turn. if (store.awaiting.size > 0 || store.hasPendingResumptions()) { traceDrive("drive", store, done, "->async(awaiting/pending)"); return driveAsync(store, done, what, idle); } if (store.pendingHostCalls.size === 0) { - // The idle verdict. Under "exit" (an async-typed lift, #292) this is - // not a fault at all: the task simply has nothing to run right now and - // the export's Promise stays pending until a later driver finishes it. + // An idle async task remains live for a later driver. if (idle === "exit") { traceDrive("drive", store, done, "EXIT-idle"); - // Same hand-off as the `done()` exit above: work this loop started - // outlives it. ensureSettlementPump(store); return "idle"; } @@ -582,12 +459,7 @@ type AwaitWinner = { awaiting: Promise | null; resumeWith(v: unknown, f?: { error: unknown }): void; }; - /** - * The promise this tag was minted from — i.e. what `t.awaiting` held at - * `tagAwait` time. Carried so a resumption site can check that the thread is - * still parked on THAT promise and not on a later one (see the guard at the - * race's resumption site). - */ + /** Park identity: membership alone cannot distinguish a later re-park. */ p: Promise; value: unknown; failure: { error: unknown } | undefined; @@ -614,22 +486,10 @@ function tagAwait(t: AwaitWinner["t"]): Promise { } /** - * THE asynchronous driving loop, exported for the one other driver in the - * runtime: `HostActivity` in exec/host_streams.ts, which must pump the store - * BETWEEN export calls (when no lifted call is in flight) with exactly these - * semantics — service settled tails, tick to quiescence, then await the race - * of every outstanding promise (parked activations AND `pendingHostCalls`), - * repeat. Reimplementing it there diverged: that copy only drained - * `store.awaiting` and never awaited `pendingHostCalls`, so a guest parked on - * a Promise-returning host import was never resumed and the host's read of - * the stream it was feeding hung (host-pump starvation of `pendingHostCalls`). - * - * Callers that must not hit the deadlock traps below (the host pump: an - * embedder that never does its half is documented to hang, not trap) can - * exclude them entirely — BOTH trap sites require - * `store.pendingHostCalls.size === 0`, and both are reached only through the - * synchronous fall-through from `done()`, so a `done` that returns true - * whenever `pendingHostCalls` is empty provably never traps. + * Shared async driver for host activity and settlement pumps. Service tails, + * tick, then race awaiting activations and host calls. A pump that must stay + * pending rather than trap on idle must make `done()` true whenever + * `pendingHostCalls` is empty; idle traps require the opposite exit decision. */ export async function driveStoreAsync( store: Store, @@ -642,64 +502,15 @@ export async function driveStoreAsync( } /** - * How many `driveAsync` loops are live on a store. - * - * THE INVARIANT is not "only one loop may ever run" — concurrent export calls - * have always produced concurrent loops, and the host-stream pump's stand-down - * below is cooperative, so a *bounded overlap window* remains by construction - * (an export call can start while the pump is parked mid-`await`; the pump - * notices at its next `done()` evaluation, which the driver-arrival one-shot - * below now makes prompt — before issue #239 it was "whenever the host happens - * to answer", i.e. not bounded at all). The invariant is: - * - * **no activation is resumed twice for one settlement, and no activation is - * resumed with a value from a settlement it has already consumed.** + * Live async drivers per store. Concurrent exports may overlap; fallback + * pumps stand down cooperatively when another driver arrives. There is no + * single-driver invariant. * - * Overlap is benign for that invariant because of three mechanisms, in - * decreasing order of how much weight they carry: - * - * (a) LOAD-BEARING — `resumeWith` synchronously deletes the thread from - * `store.awaiting` (task/thread.ts `Thread.resumeWith`), and every - * resumption site here is guarded by an `store.awaiting.has(...)` test - * evaluated synchronously immediately before the call. The loser of a - * race therefore sees the deletion. The ordering that makes this - * airtight is microtask FIFO: both loops' race continuations were - * queued when the *tag* settled, which is strictly before the winner's - * `resumeWith` can run and therefore strictly before any re-park the - * resumed activation performs can queue a new settlement. So the loser - * observes "deleted", never a re-park that restored membership. - * (b) `tagAwait` memoizes per PROMISE (not per thread), so overlapping loops - * racing the same parked thread await the *same* tag object and see one - * settlement, not two independent ones. This is what makes (a)'s - * "queued at tag settlement" premise hold across loops. - * (c) The store's pending-resumption set (`Store.pendingResumptions`) - * serializes the resumption path WITHIN a store: every loop driving - * that store yields at its top while `store.hasPendingResumptions()`, - * so a settled activation runs before anything else is scheduled. - * (Until 2026-08-22 this was a module-global single slot with a - * one-claimant assert; per-store multi-entry replaced it — issues #158 - * mechanism B and #210. Overlapping loops in the sense meant here are - * loops on the SAME store, which is exactly what (c) still covers; - * loops on different stores never shared a settlement to race for.) - * - * (a) is the guarantee; (b) and (c) are what make (a) apply across loops - * rather than only within one. The one corner (a) does NOT cover — a thread - * resumed by the *other* loop's `tick`, re-parked on a NEW promise, whose OLD - * promise then settles late — is closed separately at the resumption site - * below by comparing promise identity, not just membership. - * - * What overlap is NOT benign for is throughput and blame: two loops ticking - * the same store interleave their `serviceSettled`/`tick` phases, and the - * host-stream pump was observed to trip `Trap: table entry empty` out of - * `runCallbackLoop` when it drove unconditionally alongside an export call's - * loop. Export calls own their loops and cannot yield to anyone; the pumps - * are *fallback* drivers — the host-activity pump for embedder operations - * that land BETWEEN export calls, the settlement pump (below) for host-call - * settlements that land between them — so they are the side that stands - * down, using the two accessors below, narrowing the window to the - * cooperative residue described above. When an export call's loop is live it - * already races `pendingHostCalls` and `store.awaiting`, i.e. it pumps host - * activity on the embedder's behalf. + * Each settlement must be delivered once to its original park. `resumeWith` + * deletes awaiting membership synchronously; race winners check both that + * membership and promise identity, then remove queued copies before resuming. + * Per-promise tags share settlement reactions across racers. Per-store + * pending-resumption gates give engine continuations a turn before more ticks. */ const driverDepth = new WeakMap(); const driverIdle = new WeakMap; r: () => void }>(); @@ -722,27 +533,11 @@ export function whenStoreDriverIdle(store: Store): Promise { } // --------------------------------------------------------------------------- -// Driver arrival: closing the overlap window (issue #239) +// Driver arrival // --------------------------------------------------------------------------- // -// The stand-down above ("the pumps are *fallback* drivers") is evaluated only -// at a driver's next `done()`, so the doc's "bounded overlap window" is really -// bounded by whatever the incumbent driver is parked on — and its longest park -// is `Promise.race([...parked tags, ...pendingHostCalls])`, i.e. HOST-CONTROLLED -// time. That is a stall in its own right, and it is fatal in combination with -// the SPECULATIVE resume entry the race holds: `Store.pendingResumptions` is a -// store-wide scheduling gate, so a second driver on the same store spins at -// `driveAsync`'s top and dies at the 10,000-hop internal-bug assert in ~311ms -// (issue #239 — the same-store half of the cross-store stall #210 fixed; see -// `tests/cross_store_driver_test.ts`, whose header describes this gate being -// "held for the entire duration of a guest's wait on a slow host import"). -// -// So drivers announce themselves: every `driveAsync` that finds itself the -// second (or later) loop on a store fires this one-shot, which every driver -// races alongside its parked tags. The incumbent wakes within a microtask, -// drops the speculative entry on its way out of the race, and re-evaluates -// `done()` — which is exactly the stand-down the pumps were always supposed to -// perform, now prompt instead of "whenever the host happens to answer". +// Wake incumbents so they release speculative gates and re-evaluate `done` +// without waiting for a possibly unbounded host call to settle. const driverArrivals = new WeakMap< Store, { p: Promise; r: () => void } @@ -772,45 +567,13 @@ function fireDriverArrival(store: Store): void { } // --------------------------------------------------------------------------- -// Host-call arrival: the other stale snapshot +// Host-call arrival // --------------------------------------------------------------------------- // -// Every park that watches host calls does it by SNAPSHOT — `[... -// store.pendingHostCalls]` is spread once, when the racer parks. A parked -// driver therefore watches the calls that existed at park time and nothing -// else, and the calls a store owns are not a fixed set: guest execution -// registers new ones at the two `pendingHostCalls.add` sites below. -// -// The driver-arrival one-shot above does NOT cover this. It fires when a new -// *driver* starts (`driveAsync` at depth > 1), and the registration that -// opens the hole routinely happens under no new driver at all. Two paths run -// guest code without bumping the driver depth: an export entered through the -// synchronous `drive` path, and `HostActivity.pump()`'s synchronous drain -// (exec/host_streams.ts) while its async half is already parked in -// `driveAsync` — the stream-dom shape, a `readDirect` session live and every -// import awaited from an event handler. Either way the guest lowers a host -// import, the call is registered — and the incumbent parked driver, whose -// snapshot predates it, never hears about it. When that call settles, its -// continuation readies the guest thread and deletes itself from -// `pendingHostCalls`, but nobody drives: the settlement pump is standing down -// because `storeDriverDepth > 0` (the parked driver counts), and the parked -// driver is still waiting on promises that may never settle. The guest is -// resumed by the next unrelated export call. That is the same class as #239 — -// liveness held hostage by whatever an incumbent driver happens to be parked -// on — with the registration, not the arrival of a second loop, as the event -// that goes unheard. -// -// So registrations announce themselves too, on the same one-shot discipline: -// fired by `registerHostCall` below (THE path for both sites) and raced by -// every `driveAsync` park that spreads `pendingHostCalls`, alongside the -// snapshot. A racer wakes within a microtask, re-snapshots (which now includes -// the new call), and re-parks. The settlement pump does not need it: see the -// note at its race. -// -// Kept separate from driver arrival rather than folded into it because the -// two mean different things — "another loop is driving this store, stand -// down" versus "your snapshot is stale, re-take it" — and only the first is -// what `fireDriverArrival`'s callers and doc comment assert. +// Promise races watch snapshots. Synchronous export entry or host-activity +// draining can register a call without starting a new async driver. Announce +// every registration so parked drivers refresh their snapshots independently +// of the driver-arrival stand-down signal. const hostCallArrivals = new WeakMap< Store, { p: Promise; r: () => void } @@ -840,16 +603,9 @@ function fireHostCallArrival(store: Store): void { } /** - * Register an outstanding host call on `store` and announce it to every - * parked racer. THE registration path for real host calls — the two lowering - * sites below go through it, and so does the regression test that pins the - * announcement (tests/parked_driver_host_call_test.ts), because a raw - * `pendingHostCalls.add` is exactly the silent registration this closes. - * - * (`HostActivity`'s arm in exec/host_streams.ts is deliberately NOT a caller: - * it re-arms on every embedder notification and means "the embedder may still - * act", not "the host owes an event" — the same distinction `hasRealHostCall` - * draws.) + * Register real host work and wake parked racers. Use this rather than adding + * directly to `pendingHostCalls`. HostActivity arms are different: they mean + * the embedder may act, not that an external result is outstanding. */ export function registerHostCall( store: Store, @@ -863,52 +619,12 @@ export function registerHostCall( // The settlement pump: liveness between export calls // --------------------------------------------------------------------------- // -// A host-import promise that settles while a driver is live is serviced by -// that driver (`driveAsync` races `store.pendingHostCalls`). One that settles -// while NO driver is live only mutates scheduler state — the registration -// site's continuation delivers results and readies threads, but nothing calls -// `serviceSettled`/`tick`, so the work sits queued until the next export call -// or host stream/future operation happens to drive the store. For a guest -// with genuinely background work — the canonical shape is a task parked WAIT -// on a waitable set whose pending host call is a clock (a componentize-go -// keep-alive ticker, a wasi:clocks `wait-for`) — that turned "the host will -// wake me" into "the embedder's next unrelated call will wake me": a liveness -// gap, not a policy (wasmtime's event loop delivers such wakeups whenever the -// embedder dwells in `run_concurrent`; on a JS host the event loop is always -// dwelling). -// -// The settlement pump closes the gap: whenever a driver exits leaving real -// host calls outstanding (`hasRealHostCall` — activity arms excluded, they -// mean "the embedder may still act", not "the host owes an event"), a -// detached keeper parks on `Promise.race` of those calls and, when one -// settles, drives the store to quiescence with the same loop and the same -// cooperative discipline as the host-activity pump above it in the driver -// hierarchy: -// -// * it stands down whenever an export call's loop is live -// (`storeDriverDepth` / `whenStoreDriverIdle`, plus the `> 1` clause in -// its `done`, exactly as `HostActivity.#pumpAsync`); -// * its `done` returns true whenever `pendingHostCalls` is empty, which is -// the precondition of BOTH deadlock traps in `driveAsync` — the pump can -// therefore never convert the documented embedder-never-acts hang into a -// trap (see the `driveStoreAsync` note above); -// * failures park on `store.hostFailure` for the next embedder call to -// surface, the channel every between-calls driver already uses. -// -// Every real `pendingHostCalls` entry is born during guest execution, i.e. -// inside some driver, so arming at driver exit (`driveAsync`'s finally and -// `drive`'s synchronous completion) observes every registration. A -// HOST-initiated resource dtor (embedder `drop()` between calls) is no -// exception since #160: it is a lifted call like any other, so it brings its -// own driver, and any host call its activation makes is registered inside -// that driver. -// -// STALE SNAPSHOTS: the keeper races the real host calls it saw when it -// parked. A drive it performs can register NEW calls (the keep-alive ticker -// re-arming is the routine case), and `ensureSettlementPump` may be called -// while the keeper is already parked. Both are handled by a nudge promise -// raced alongside the snapshot: arming an already-live pump fires the nudge, -// the keeper wakes, re-snapshots, and re-parks. +// Every driver exit, including exceptions, hands off real host calls, queued +// tails and hop-parked activations. The keeper drives their wakeups without +// requiring another export call. It stands down for live drivers, stops at +// quiescence rather than task completion, and parks failures on hostFailure. +// Activity arms are excluded. Re-arming a live keeper nudges it to refresh its +// snapshot, including calls registered by a drive it performed itself. const settlementPumps = new WeakSet(); const settlementNudges = new WeakMap< @@ -936,11 +652,7 @@ function fireSettlementNudge(store: Store): void { } /** - * What an exiting driver must hand over: real host calls it was watching, a - * queued activation tail (which gates `Store.tick` for every later driver), - * or a hop-parked thread — the #280 rule ("a driver is not done while ANY - * thread of ANY task is hop-parked") applied to the exits that cannot - * evaluate their `done` predicate, i.e. the exception exits. + * Work needing an owner after driver exit, including exception exits. */ function pumpWork(store: Store): boolean { return hasRealHostCall(store) || store.settled.length > 0 || @@ -948,7 +660,7 @@ function pumpWork(store: Store): boolean { } /** - * Ensure a settlement pump is watching `store`'s real outstanding host calls. + * Ensure a settlement pump owns outstanding host calls, tails and entry hops. * Idempotent and cheap; called at every driver exit. Never throws. */ export function ensureSettlementPump(store: Store): void { @@ -978,10 +690,7 @@ async function settlementPumpLoop(store: Store): Promise { // it in a loop. if (store.hostFailure !== undefined) return; const real = realHostCalls(store); - // A queued tail is serviceable RIGHT NOW: drive without parking. A - // hop-parked thread lands on the engine's own schedule, so its promise - // is raced alongside the host calls — that is how an exception exit's - // orphaned hop (F4/#280) gets an owner. + // Queued tails need no await; orphaned entry hops are raced with host work. const hops = store.settled.length > 0 ? [] : entryHopThreads(store).map((t) => t.awaiting).filter(( @@ -990,13 +699,8 @@ async function settlementPumpLoop(store: Store): Promise { if (store.settled.length === 0) { if (real.length === 0 && hops.length === 0) return; const nudge = armSettlementNudge(store); - // The host-call arrival one-shot does NOT ride here, deliberately: a - // registration this snapshot misses always reaches `ensureSettlementPump` - // (which fires the nudge) — `driveAsync`'s finally, `drive`'s synchronous - // completion, and `HostActivity.pump()`'s async half is itself a - // `driveStoreAsync`. Any driver live meanwhile is what we stand down for. - // Rejections are not this pump's to report: the registration site's - // own continuation parks them on `store.hostFailure`. + // Driver exits nudge this snapshot; active drivers own new work meanwhile. + // Registration continuations, not this race, report host rejections. await Promise.race([ ...real.map((p) => p.then(() => {}, () => {})), ...hops.map((p) => p.then(() => {}, () => {})), @@ -1004,19 +708,11 @@ async function settlementPumpLoop(store: Store): Promise { ]); if (storeDriverDepth(store) > 0) continue; } - // Drive unconditionally after a wake: `storeQuiescent` cannot see a - // READY waiting thread (the usual product of a settlement — the - // continuation readied the guest and deleted its own host call), so - // gating the drive on it skips exactly the work this pump exists to - // do. `driveAsync` drains ready threads before consulting `done`, and - // a vacuous round exits on its first `done` evaluation. + // Drain after every wake: storeQuiescent does not count ready waiters + // left by a host settlement that already removed its call registration. await driveStoreAsync( store, - // Quiescence, not completion — and the same three exit clauses as - // the host-activity pump: nothing only an event-loop turn could - // advance; `pendingHostCalls` empty (the deadlock traps' - // precondition, so this pump provably never traps); another driver - // appeared (ours is the 1). + // Stop before idle traps, at quiescence, or when another driver arrives. () => store.pendingHostCalls.size === 0 || storeQuiescent(store) || @@ -1048,50 +744,21 @@ async function driveAsync( ): Promise { const depth = storeDriverDepth(store) + 1; driverDepth.set(store, depth); - // An incumbent driver may be parked in the awaiting-race holding the - // speculative resume entry — a store-wide gate this loop would otherwise - // spin on until the 10,000-hop assert (issue #239). Announce ourselves so it - // stands down within a microtask. + // Wake incumbents to release speculative gates and let fallback pumps stand down. if (depth > 1) fireDriverArrival(store); try { let claimHops = 0; for (;;) { traceDrive("driveAsync", store, done, "top"); - // FIRST: service every settled-but-unserviced activation tail, in settle - // order (`Store.settled` — armed eagerly at park time). A settled - // `awaitValue` is the rest of an activation that already finished its - // wasm; the reference runs that bookkeeping atomically inside - // `Thread.resume`, so nothing may be scheduled past it (`Store.tick` - // refuses while the queue is non-empty). Servicing after ticking let a - // freshly-resumed caller race into an entry gate while a finished - // callee's body had yet to release the exclusive slot — cancellable.wast - // then reported STARTING for an entry the reference admits. + // Complete settled activation bookkeeping before any scheduling decision. store.serviceSettled(); if (store.hostFailure !== undefined) throw takeHostFailure(store); - // A pending resumption of THIS store is an engine-driven resumption in - // flight: its activation has not yet parked again or finished. It will - // die on its own — parking consumes it (`blockCurrentActivation`), - // finishing releases it (`Store.noteAwaiting`'s settle continuation) — so - // yield microtasks until it does. The driver must NOT blanket-clear here: - // an entry may have been taken by a guest built-in settling another - // activation's suspension (`subtask.cancel` delivering a cancellation), - // and clearing it before that activation runs re-opens the - // mis-attribution window the entry exists to close. - // - // PER-STORE (issue #210): read only THIS store's entries. Activations - // never cross stores, so another store's pending resumption is none of - // this loop's business — and a gate shared across stores would spin an - // idle store's driver here, to its death at the hop bound below in - // ~311ms, merely because ANOTHER store's guest was dwelling on a slow - // host import. + // Yield for this store's engine resumptions. Only their execution/park + // or settlement may release them; never clear other owners' entries. if (store.hasPendingResumptions()) { traceDrive("driveAsync", store, done, "yield-pending"); - // Bounded: a pending entry that never dies is an internal bug (every - // path out of a resumed activation releases it — park, finish, trap), - // and a pure-microtask wait would otherwise starve the event loop and - // every stall timer with it. Interleave macrotask hops so timers stay - // alive, and fail loudly rather than spin forever. Scoped per store, - // this is again the internal-bug detector it was meant to be. + // Bound leaked claims, interleaving timer turns to avoid starving + // the event loop while diagnosing an internal scheduling failure. claimHops++; assert_( claimHops < 10_000, @@ -1108,13 +775,8 @@ async function driveAsync( claimHops = 0; while (store.tick()) { if (store.hostFailure !== undefined) throw takeHostFailure(store); - // FAIRNESS between tick-able threads and promise-parked ones. A thread - // that is READY again on every resume (the callback-ABI YIELD spin) - // would otherwise monopolize this drain while a parked thread's - // settled promise waits (the starvation that hung - // drop-subtask.wast:139), and the engine's own continuations (jspi - // pin (j)) only ever land on microtask turns. One hop per tick; bail - // to the top the moment an activation tail lands. + // A READY/YIELD loop must not starve promise settlements. Give engine + // continuations a microtask per tick and service any landed tails first. if (store.awaiting.size > 0) { await Promise.resolve(); if (store.hasServiceableSettled()) break; @@ -1125,54 +787,22 @@ async function driveAsync( traceDrive("driveAsync", store, done, "EXIT-done"); return "done"; } - // Only a SERVICEABLE tail is a reason to loop again: a queue holding - // only tails DEFERRED on a non-enterable instance (issue #156) would - // spin this loop hot — nothing in the cycle awaits. + // Queued tails and pending resumptions take priority over parking. if (store.hasServiceableSettled() || store.hasPendingResumptions()) { continue; } - // Service promise-parked threads (jspi). - // - // This must NOT block on one chosen thread's promise. A thread parked on a - // promising-wrapped nested activation only settles once that activation's - // own suspension points have been resumed -- and resuming those is - // `Store.tick`'s job, i.e. *this loop's* job. Awaiting a single promise - // therefore stops the scheduler while waiting for something that needs the - // scheduler: a pure-microtask stall with no trap and no rejection. - // Observed on `async/async-calls-sync.wast` the moment site 1 became the - // first lit suspension site: turn N serviced a promise that - // never settled while three other parked threads and three ready-able - // suspension points went unexamined. - // - // So: race every outstanding promise (parked threads AND host calls) and - // service whichever settles first, re-ticking each turn. The claim is - // taken in the tagged continuation -- as close to settlement as we can get - // -- so pin (i)'s window (engine-driven wasm resumption running built-ins - // before our continuation) is still covered for the thread that actually - // resumed, without falsely claiming the ambient for threads that did not. + // Race all activations and host calls. Awaiting one chosen activation + // alone could stop the scheduler that its nested suspension needs. if (store.awaiting.size > 0) { - // Is this actually progress, or a deadlock wearing its clothes? - // - // Everything in `store.awaiting` is an INTERNAL promise: a - // promising-wrapped wasm activation. Such a promise settles either on - // its own (the activation ran to completion -- which happens within one - // macrotask turn, since the work is already done and only the microtask - // hop remains) or because WE resume a suspension point it is waiting - // behind. If no thread is ready, no host call is outstanding, and a full - // macrotask turn passes with nothing settling, then nobody can move: the - // awaited promises need us and we need them. That is the deadlock trap - // (definitions.py `canon_lift`'s empty-candidate-set `trap_if`), and - // without this check it presents as a silent stall instead -- which is - // exactly what `tests/jspi/deadlock_test.ts` caught the moment site 2 - // was lit. + // With no external work or pending resumption, allow a timer turn for + // engine hops to settle before declaring idle. Internal activation + // promises alone do not establish that further progress is possible. if ( store.pendingHostCalls.size === 0 && !store.hasPendingResumptions() ) { traceDrive("driveAsync", store, done, "deadlock-probe"); - // Exclude threads whose settle is already QUEUED in `store.settled` - // (issue #156): their promise has settled, so racing them wins - // instantly off the memoized `tagAwait` tag, forever, in an unbounded - // microtask chain — the tail is `serviceSettled`'s to run. + // Queued tails belong to serviceSettled; racing their settled tags + // repeatedly would create an unbounded microtask loop. const queued = new Set(store.settled.map((s) => s.t)); const parked = ([...store.awaiting] as AwaitWinner["t"][]).filter( (t) => !queued.has(t), @@ -1188,20 +818,8 @@ async function driveAsync( `deadlock-probe:progressed=${progressed}`, ); if (!progressed) { - // The race covered a SNAPSHOT of the awaiting set. A thread that - // parked during the macrotask turn (a promising callee's body - // yielding its awaitValue mid-hop — jspi pin (j) makes this - // routine) was not raced, and its promise may already be settled; - // trapping now would declare a deadlock one iteration before the - // loop would have serviced it. Membership change ⇒ re-probe. - // - // `fresh` gets the SAME queued-entry filter `parked` got (issue - // #156), against a RECOMPUTED queued set — the settled queue can - // change across the probe's await. Comparing a filtered snapshot - // against an unfiltered one would read "changed" on every turn in - // the all-deferred wedge state, so the verdict below could never - // be reached and the wedge would present as a silent - // macrotask-paced busy idle instead of a trap. + // Revalidate the snapshot after awaiting. Apply the same queued-tail + // filter to both snapshots, using the current queue for the new one. const freshQueued = new Set(store.settled.map((s) => s.t)); const fresh = ([...store.awaiting] as AwaitWinner["t"][]).filter( (t) => !freshQueued.has(t), @@ -1209,29 +827,8 @@ async function driveAsync( const changed = fresh.length !== parked.length || fresh.some((t, i) => t !== parked[i]); if (changed) continue; - // The probe's precondition can also expire WITHOUT the awaiting - // set changing: the same activation resumes off an engine - // continuation chunk during the probe's macrotask turn (jspi - // pin (j) — a sync-completing Suspending import still defers its - // continuation), runs, and re-parks through the suspending mark arm, which - // registers a fresh `pendingHostCalls` entry. The activation - // promise never settled and `awaiting` membership is unchanged, - // but the park is externally wakeable now — the verdict's own - // precondition (`pendingHostCalls.size === 0`) no longer holds. - // Observed on wasi-shims' stream/future round-trip poll (sync fast path): probe sampled - // hostCalls=0 between a settled park and the next one, then - // trapped a live workload with hostCalls=1. Re-check ⇒ re-probe. - // Likewise a SERVICEABLE settled entry (issue #156): dispatching - // it is progress, so this is not a deadlock verdict — re-probe. - // A deferred-only queue deliberately does NOT re-probe: nothing - // can dispatch it while the lock is held, and if no host call is - // outstanding nothing will ever release that lock, so it falls - // THROUGH to the verdict below — the same loud-wedge treatment the - // servicing race's own all-deferred fallthrough gets. Per the #156 - // analysis that state is unreachable (a lock spanning this loop's - // await always has a `pendingHostCalls` entry, which fails this - // probe's precondition); keeping it loud is what makes it an - // internal-wedge detector rather than dead code. + // Even unchanged awaiting membership can acquire external work, + // pending resumptions or queued tails during the probe. if ( store.pendingHostCalls.size > 0 || store.hasPendingResumptions() || @@ -1252,63 +849,24 @@ async function driveAsync( `and none is ready)`, ); } - // No promise settled, but a thread became READY while we waited -- - // typically a suspension point whose `readyFunc` turned true because - // another activation ran during the macrotask turn. The way forward - // is `Store.tick`, not a promise: go back to the top and resume it. - // Falling through to the servicing block instead would await - // promises that nothing will settle while a runnable thread sits - // there -- the `async/sync-barges-in.wast` stall exactly. + // A thread became ready: tick it rather than awaiting its dependents. continue; } - // Progress IS possible: fall through to the normal servicing below, - // which resumes the settled thread. Returning to the top instead would - // spin -- the memoized tag is already settled, so the race would win - // instantly, forever, without anyone being resumed. + // Consume the settlement, either through the queue or the race below. } - // Re-check membership: the deadlock probe above AWAITS, and everything - // below reads `[...store.awaiting][0]` as if the set were still - // non-empty. A thread resumed during the probe (its settle continuation - // runs `resumeWith`, which deletes it) can empty the set, and the - // snapshot's `parked[0]` is then `undefined` — the exact check-then-act - // shape that made the host pump's copy of this loop throw - // `TypeError: ... (reading 'awaiting')` into `store.hostFailure`, where - // it poisoned a later unrelated call via check-then-act on `store.hostFailure`. Nothing to - // service ⇒ go back to the top and re-evaluate `done`. - // Same re-check for the settled queue, and for the same reason: the - // probe's macrotask turn can land a fresh, SERVICEABLE activation tail - // (that is exactly what "progress IS possible" above usually means). - // The queue owns those threads — the race below deliberately excludes - // them (issue #156) — so the way forward is the top of the loop, where - // `serviceSettled` dispatches them. Without this, filtering the - // just-settled thread out of the race left the loop awaiting promises - // that only its dispatch could settle (observed: tests/jspi/ - // handshake_test.ts stalled, then tripped the claim assert). + // The probe awaited: another driver may have consumed the park, or + // noteAwaiting may have queued its tail. Recheck before selecting one. if (store.awaiting.size === 0 || store.hasServiceableSettled()) { continue; } - // Claim the ambient for ONE parked thread and await its promise -- as - // before, so pin (i)'s window is covered exactly as it was -- but race - // that promise against every other outstanding promise so this loop can - // never be held hostage by it. The claimed thread's promise may only be - // settleable by further scheduler progress (a promising-wrapped nested - // activation whose own suspension points this loop must still resume); - // blocking on it alone is the pure-microtask stall described above. - // Same exclusion as the probe (issue #156): a thread whose tail is - // already queued in `store.settled` must not be raced — its tag is - // settled, so it re-wins instantly and livelocks the event loop, - // starving the very host-call settle that would release the lock. + // Race only parks not already owned by the settled queue. const queued = new Set(store.settled.map((s) => s.t)); const parked = ([...store.awaiting] as AwaitWinner["t"][]).filter( (t) => !queued.has(t), ); if (parked.length === 0) { - // UNREACHABLE BY CONSTRUCTION. `parked` is `store.awaiting` minus - // the threads whose tails are already queued in `store.settled`, and - // we only get here with `awaiting` non-empty and - // `hasServiceableSettled()` false — which means the settled queue is - // EMPTY, so nothing was excluded. Retained as a wedge detector, not - // as expected behavior. + // Defensive fallback: the checks above imply a non-empty awaiting + // set and empty settled queue, so filtering cannot remove all parks. if (store.pendingHostCalls.size > 0) { await Promise.race([ ...store.pendingHostCalls, @@ -1317,10 +875,6 @@ async function driveAsync( ]).catch(() => {}); continue; } - // Per the issue #156 analysis this is unreachable (a spanning lock - // always has a `pendingHostCalls` entry; a synchronous lock cannot - // span this loop's await). An internal-wedge detector, not expected - // behavior. traceDrive("driveAsync", store, done, "DEADLOCK-TRAP-deferred"); trapIf( true, @@ -1337,63 +891,17 @@ async function driveAsync( for (const h of store.pendingHostCalls) { others.push(h.then(() => null, () => null)); } - // A SPECULATIVE entry: the chosen thread is a promising-wrapped - // activation, and the engine may run its wasm during this await (pin - // (i)). It is dropped on the way out — if the activation is genuinely - // mid-resumption its own exact entry (minted by - // `SuspensionPoint.resume`) is what carries it, and dropping an entry - // that names a thread already gone from the set is a no-op. - // - // ONLY ITS OWN ENTRY (issue #158): the `finally` must drop the entry - // THIS loop added and nothing else. A guest-synchronous delivery during - // the await takes a fresh entry of its own, and clearing that one here - // would re-open early the very window it exists to close — which is why - // the gate is a set of entries rather than a single slot. - // - // SOLE DRIVER ONLY, AND ONLY UNTIL ONE ARRIVES (issue #239). The entry - // is a claim over a window this loop cannot bound: the race settles when - // the HOST answers, which may be never. As a store-wide scheduling gate - // (`Store.tick` refuses; every driver yields at its top) that is a wedge - // the moment a second driver exists — it spins at the top of its own - // loop and dies at the 10,000-hop assert in ~311ms, an internal-bug - // detector firing on a perfectly ordinary suspended guest. Two concurrent - // export calls with one slow suspending import were enough; the reported - // shape was a detached guest task cancelling an in-flight import, which - // parks mid-frame with no export call outstanding and leaves the - // settlement pump holding this entry. - // - // What the entry protects — "the engine may run `chosen`'s wasm during - // this await" — it protects by refusing OTHER `Store.tick` callers, and - // this loop is not one of them while it awaits. The tick callers that - // can reach a store mid-race are another `driveAsync` loop and - // `HostActivity.pump`'s synchronous drain (exec/host_streams.ts) — the - // latter is not gated by driver depth, so scoping the entry to "sole - // driver" does hand it a window an unscoped entry would close at - // depth >= 2. - // What holds regardless is the invariant the `driverDepth` note names: - // a genuine resumption is preceded by `SuspensionPoint.resume`'s OWN - // entry (jspi/bridge.ts, minted before the settle), and every - // resumption site here re-checks membership and promise identity - // synchronously — mechanisms (a) and (b), which is where that note - // already puts the weight. - // ONLY IF WE ADDED IT (issue #158, same rule as the `finally` below): - // `pendingResumptions` is a Set by identity, so a genuine entry for - // `chosen` minted meanwhile — or already held — collapses with ours, - // and removing "ours" would drop the genuine one. + // A sole driver may gate ticks speculatively while the engine runs + // chosen's activation. Driver arrival breaks the race and releases the + // gate, preventing an unbounded host wait from blocking a second loop. + // Remove only an identity this loop inserted, never clear the set. + // Genuine SuspensionPoint resumptions establish their own entries. const sole = storeDriverDepth(store) === 1; const added = sole && !store.pendingResumptions.has(chosen); if (added) store.addPendingResumption(chosen); let winner: AwaitWinner | null; try { - // `armDriverArrival` rides the race for every driver, not just the one - // holding the entry: waking on a new arrival is also how a fallback - // pump reaches its next `done()` — i.e. its stand-down — promptly. - // `armHostCallArrival` rides for the sibling reason: the tags below - // are a snapshot of what was parked when we entered the race, so a - // host call registered after that under no new driver (a sync `drive` - // export, `HostActivity.pump()`'s sync drain — neither fires a driver - // arrival) can ready a thread with no racer watching for it. See the - // host-call arrival note above. + // Arrivals trigger stand-down or snapshot refresh without host settlement. winner = await Promise.race([ chosenTag, ...others, @@ -1403,23 +911,9 @@ async function driveAsync( } finally { if (added) store.removePendingResumption(chosen); } - // Resume whichever thread actually settled -- not necessarily the one we - // claimed. Resuming only the claimed thread would spin: its promise may - // never settle, the same thread would be chosen again next turn, and the - // already-settled tags would win the race instantly forever (observed as - // an OOM, not a hang). Our own entry is dropped above before any resumption, - // exactly as on the original single-promise path, so this does not widen - // the ambient window; it only ensures the loop always makes progress. - // Membership is not enough: the corner it misses is a thread the OTHER - // overlapping loop resumed via `tick`, which then re-parked on a NEW - // promise, after which its OLD promise settles late — membership is - // true again but the tag's value belongs to a settlement this thread - // has already consumed. Compare promise identity too. - // ONE SETTLEMENT, ONE DELIVERY (definitions.py `Thread.resume` is - // atomic). `noteAwaiting` records settlements EAGERLY, so this - // promise's `store.settled` entry is already queued; left there, a - // body that re-parks SYNCHRONOUSLY inside `resumeWith` gets the OLD - // value delivered against its NEW park by the next `serviceSettled`. + // Deliver the actual winner only if its park is still current. + // Delete queued copies before resumeWith can synchronously re-park; + // otherwise serviceSettled could deliver this result to the new park. if ( winner !== null && store.awaiting.has(winner.t) && winner.t.awaiting === winner.p @@ -1445,28 +939,11 @@ async function driveAsync( ); } traceDrive("driveAsync", store, done, "await-race"); - // Settlement order among several outstanding host calls is the host's, - // not ours — this is genuine, unavoidable nondeterminism at the boundary - // (the reference has the same freedom in `Store.tick`). Everything - // *inside* the component stays deterministic per scheduler.ts. - // - // The driver-arrival one-shot rides here too. This is the routine park of - // a quiet guest with a real host call outstanding — no speculative entry - // is held, so there is no wedge to break, but a fallback pump parked here - // would otherwise not reach its `done()` (i.e. its stand-down) until the - // HOST answered, leaving two loops interleaving `serviceSettled`/`tick` - // for that whole window. That interleaving is what the `driverDepth` note - // above calls out as bad for throughput and blame. + // Host settlement order is external. Arrivals must also wake this park + // so fallback drivers can stand down and all drivers refresh snapshots. await Promise.race([ ...store.pendingHostCalls, armDriverArrival(store), - // ... and the host-call-arrival one-shot, because the spread above is a - // SNAPSHOT: a host call registered while we are parked here under no - // new driver (a sync `drive` export, `HostActivity.pump()`'s sync drain - // — neither fires a driver arrival) would otherwise be watched by - // nobody at all (the settlement pump stands down while we, the parked - // driver, keep `storeDriverDepth` positive). See the host-call arrival - // note above. armHostCallArrival(store), ]).catch(() => {}); } @@ -1477,9 +954,7 @@ async function driveAsync( const w = driverIdle.get(store); driverIdle.delete(store); w?.r(); - // The store just went driver-idle; if real host calls remain, hand - // liveness to the settlement pump (which stands down again the moment - // any driver starts). + // The last async driver hands off any remaining pump work. ensureSettlementPump(store); } } @@ -1496,64 +971,23 @@ function takeHostFailure(store: Store): unknown { // --------------------------------------------------------------------------- /** - * Build the host-callable function for one lifted export (reference - * `Store.lift` + `canon_lift`, definitions.py lines 578 and 2154). - * - * All three lift shapes go through one `Task` + implicit `Thread`: - * - * * **sync** (`not ft.async`) — call, lift results, `task.return_`, - * post-return, then the sync driving loop until the task resolves; - * * **async + callback** (stackless) — the packed-code loop - * (EXIT / YIELD / WAIT), fully implemented here; - * * **async, no callback** (stackful) — the guest blocks mid-stack, which - * needs genuine wasm-frame suspension: `needsJspi`, at the precise point. - */ -/** - * The plain-entered variant of a **sync-typed** lifted export in jspi mode, - * attached to the promising-wrapped lifted function under this symbol - * (contracts/embedder-api.md §"Functions and async", §"Functions and async"). - * - * In jspi mode every promising-wrapped entry returns a Promise even when the - * activation completes without suspending (jspi pin (e)). Some host contexts - * cannot use a Promise no matter how promptly it resolves, so each sync-typed - * export carries a second lifted function whose ENTRY is plain (unwrapped): - * a guest activation that completes synchronously — the overwhelmingly common - * case for sync-typed WIT — delivers its results synchronously through it. - * - * Two consumers, one mechanism: - * - * * **resource constructors** — a WIT constructor is surfaced as a JS class - * constructor (§"Resources") and a JS constructor cannot await, so the - * embedder layer reads this symbol unconditionally for `[constructor]` - * exports; - * * **the embedder `sync()` adapter** — the explicit per-use - * synchronous view of any sync-typed export. - * - * The cost is confined to genuinely-suspending activations: a blocking - * built-in reached through the plain entry signals `NeedsJspi` (a capability - * error, instance left enterable), and a `Suspending`-wrapped host import - * reached from the unwrapped frame fails as a trap. Both name the export - * rather than silently deadlocking. A call made while the instance has - * hop-parked activations refuses with `SyncEntryBusy` before entering - * (`refuseOnEntryHops` below). + * Plain-entry variant of a sync-typed export, used by resource constructors + * and the embedder's `sync()` adapter (contracts/embedder-api.md §"Functions + * and async"). It avoids the Promise shape of promising entries but cannot + * suspend: blocking capability failures raise `NeedsJspi`, and reaching a + * Suspending import without an eligible stack traps. Pending instance entry + * hops cause a pre-entry, non-poisoning `SyncEntryBusy` refusal. */ export const SYNC_ENTRY: unique symbol = Symbol("polyengine.syncEntry"); // --------------------------------------------------------------------------- -// Pending async-typed lifts, and their poisoning (#292) +// Pending async-typed lift results // --------------------------------------------------------------------------- // -// An async-typed export whose driver exited idle (see `IdlePolicy`) leaves a -// host-visible Promise settled by nothing but the task itself finishing. If -// the task instead dies — a LATER driver runs it and traps — the instance is -// poisoned and that task's threads will never unregister, so the Promise -// would hang forever. That is precisely the failure #66 fixed for parked -// stream/future ends, and it gets the same treatment: a poisoning listener -// that rejects every pending lift of the instance with the poisoning cause. -// -// Registered on the extra-listener seam rather than `setOnInstancePoisoned` -// (which streams.ts owns) — see `addInstancePoisonedListener` for the -// evaluation-order reason both are seams. +// An idle exit transfers result settlement to the task's onResolve callback. +// If a later driver poisons the instance first and async-end retirement returns, +// this listener rejects pending results. A throwing retirement hook prevents +// this notification; recording the poison cause alone does not settle them. const pendingLifts = new WeakMap void>>(); function registerPendingLift(inst: object, reject: (c: unknown) => void): void { @@ -1579,6 +1013,9 @@ addInstancePoisonedListener((inst, cause) => { for (const r of waiters) r(cause); }); +/** Build a `Store.lift` / `canon_lift` entry with a Task and implicit Thread. + * Canonical options select sync result lifting, callback dispatch, or stackful + * execution; the function type separately selects the sync/async idle policy. */ export function createLiftedFunction(input: { name: string; ft: FuncType; @@ -1606,49 +1043,23 @@ export function createLiftedFunction(input: { */ allInstances?: () => Iterable<{ mayLeave: boolean }>; /** - * Opt out of the reference's *synchronous* driving loop (`driveSyncLift`, - * definitions.py `canon_lift` line 2213) for a sync-typed lift whose caller - * does not need a synchronous answer — today only the host-initiated - * resource destructor (#160; `createDtorEntry` below, `drop(): void` is - * documented non-blocking). - * - * This is not a weakening of the deadlock trap: `drive` below enforces the - * same "no ready thread, no pending host call, nothing awaiting" trap, just - * asynchronously — which is exactly the substitution jspi mode already - * makes unconditionally (see the comment at the `driveSyncLift` call). - * It matters only when a *plain*-mode core returns a thenable, i.e. a - * host-supplied JS destructor: the sync loop sees a thread parked on a - * Promise, which it can never advance, and declares a bogus deadlock. + * Let host-initiated destructors complete asynchronously even in plain + * mode. Skip driveSyncLift, which cannot advance a host JS dtor's Promise; + * the store driver still applies the sync-typed idle trap. */ allowAsyncCompletion?: boolean; /** Nested guest destructor only: preserve the caller and use the reference * sync lift drive, not the host's store-wide completion policy. */ guestDtorCaller?: ComponentInstanceState | null; /** - * Refuse — synchronously, before entering — a call made while the instance - * has HOP-parked activations, instead of deferring it (§"Functions and async", - * failure-ladder arm 2). - * - * Set for the `SYNC_ENTRY` variant, which is built with - * `suspensionMode: "plain"` inside a *jspi-mode* instantiation: the - * hop-quiescence gate below is keyed on this function's own mode and so is - * dead for that variant, yet the hazard it exists to prevent is the - * instance's, not the entry's — a hop-parked activation's pending lift - * reads memory a fresh guest turn would mutate (see the gate's comment). - * A synchronous caller cannot be deferred, so it refuses instead. The - * refusal is pre-enter, hence non-poisoning: nothing was entered, so there - * is nothing to poison — the same structural safety as `entryRefusal`. + * Refuse instance entry hops rather than deferring. SYNC_ENTRY uses plain + * mode inside a JSPI instantiation, so its own mode cannot identify this + * instance-wide result-memory hazard. Refusal occurs before entry. */ refuseOnEntryHops?: boolean; /** - * Make **async-typed** exports trap on idle instead of leaving their - * Promise pending (#292). Default false; see `IdlePolicy`. - * - * `InstantiateInput.trapOnIdle`'s only consumer is the conformance harness, - * whose `invoke` directive is a *blocking* call — the wast semantics - * wasmtime serves with `run_concurrent_trap_on_idle` behind - * `[Typed]Func::call_async`, not with `call_concurrent`. It is deliberately - * absent from the embedder layer's options. + * Harness-only blocking-call policy: trap idle async-typed exports rather + * than leaving their Promise pending. Default false; see `IdlePolicy`. */ trapOnIdle?: boolean; }): (...args: ComponentValue[]) => unknown { @@ -1666,10 +1077,7 @@ export function createLiftedFunction(input: { const store = inst.store; const mode: SuspensionMode = input.suspensionMode ?? "plain"; const guestDtor = input.guestDtorCaller !== undefined; - // Entry wrapping, half of jspi/bridge.ts's invariant: a lifted export's core - // function is one of the three activations that can reach a blocking - // built-in, so it is `promising`-wrapped exactly when the imports are - // `Suspending`-wrapped. + // Pair JSPI entries with suspension-capable imports; guest dtors use plain mode. const enteredCore = enterWasm(core, mode); // See the comment at the `drive` call in `invokeNow` and `IdlePolicy`. const idlePolicy: IdlePolicy = ft.async === true && input.trapOnIdle !== true @@ -1709,11 +1117,8 @@ export function createLiftedFunction(input: { // Depth of the sync-call scope stack on entry; see the `finally` below. const syncCallDepth = syncCallStack?.length ?? 0; - // Reference `Store.lift` (@ 2f13265) runs `canon_lift` with NO gate - // (CM#705), so host entry into a live instance is valid. What this adds - // is polyengine's per-instance - // poisoning divergence — a poisoned instance is a corpse, and its refusal - // names the original trap (polyengine#145 ask 1). + // Poison refusal is separate from task admission; preserve the guest + // destructor's caller for same-instance semantics. { const refusal = entryRefusal( inst, @@ -1727,9 +1132,7 @@ export function createLiftedFunction(input: { let resolved: ComponentValue[] | null = null; let resolvedSeen = false; /** - * One-shot: set only by `backgroundCompletion` below, fired here when the - * task resolves. See there for why the resolve callback — not thread - * drain — is the host's answer event (polyengine#313). + * Idle-path result waiter. onResolve, not thread drain, supplies the answer. */ let onResolvedHook: (() => void) | null = null; const task = new Task( @@ -1782,22 +1185,9 @@ export function createLiftedFunction(input: { }; const unwind = (): void => { - // Unwind any FACT sync-call brackets a trap escaped. - // - // A trap thrown inside an adapter skips that adapter's - // `exit-sync-call`, so its `SyncCallScope` (and the `num_lends` it - // holds on the caller's handles) would otherwise survive the call. - // wasmtime does not need this: it poisons the whole store on trap - // (`Store::call_hook`/panic-on-reuse semantics), so no later call can - // observe the stale state. This runtime deliberately supports - // post-trap re-entry — the `trapState.pending` reset above exists for - // exactly that — so the state has to be unwound instead. Leaving it - // would attach the next `transfer-borrow` to a dead scope and leave - // lent handles permanently un-droppable ("while borrowed" forever). + // Failed adapters may skip exit-sync-call; release this task's lenders + // so unaffected instances do not retain abandoned borrows. if (completed) return; - // Per-ACTIVATION now (see `Thread.syncCallStack`): unwind the brackets - // of every activation this task owns, which a trap inside a FACT adapter - // skipped. A task can have several threads, so the loop is over threads. for (const t of task.threads as { syncCallStack: unknown[] }[]) { while (t.syncCallStack.length > 0) { (t.syncCallStack.pop() as LenderScope).releaseLenders(); @@ -1805,20 +1195,9 @@ export function createLiftedFunction(input: { } void syncCallStack; void syncCallDepth; - // FACT clears the callee's / caller's `may_leave` flag around each - // lift and lower (`fact/trampoline.rs`, `set_may_leave_false`) and - // restores it afterwards. A trap in between skips the restore, so an - // instance can be left permanently unable to leave — every later call - // through an adapter then trips FACT's own `CannotLeaveComponent` - // check. With the stack unwound to the host boundary no lift or lower - // is in flight, so `may_leave` is true for every instance by - // definition; assert that resting state rather than leaving the - // component bricked. - // - // The ENTERED instance is excluded: it is poisoned by this trap (see - // `poison` below) and must stay exactly as the trap left it. Restoring - // its `may_leave` would be tidying the state of an instance that is no - // longer allowed to run at all. + // Host-boundary unwind restores sibling mayLeave flags skipped by FACT. + // A guest destructor is nested inside a live caller, so it must not + // restore store-wide flags. The entered instance is excluded in either case. for (const i of guestDtor ? [] : allInstances?.() ?? []) { if (i as unknown as ComponentInstanceState !== inst) { i.mayLeave = true; @@ -1827,29 +1206,11 @@ export function createLiftedFunction(input: { }; /** - * A trap escaped the task: mark the instance poisoned. - * - * polyengine's NAMED DIVERGENCE. definitions.py has no notion of a - * post-trap instance at all — a Trap is the end of the world — and - * wasmtime's answer is to poison the whole store. This runtime keeps the - * component graph alive and buries only the instance that trapped: it is - * not in a known state, so it may never be entered again, and the next - * call reports `cannot enter component instance` with the recorded cause - * appended (polyengine#145 ask 1). - * `test/async/builtin-trap-poisons-instance.wast` asserts exactly this, - * twice; the marker (`notifyInstancePoisoned`) is the whole mechanism. - * - * Only `inst` is affected; sibling instances stay usable. - * - * Poisoned instances can never rendezvous again, so their handle tables' - * live stream/future ends are retired here (#66): parked host operations - * settle (DROPPED) instead of hanging forever, and the recorded failure - * lets the embedder layer reject them loudly. + * Record this instance's failure and retire its async ends/pending lifts. + * Per-instance poisoning is a runtime policy beyond definitions.py; + * sibling instances remain usable. */ const poison = (e: unknown): void => { - // Through the seam (not retireInstanceAsyncEnds directly) so the - // poison marker is recorded too — `Thread.resumeWith` retires this - // instance's late settles against it instead of assert-cascading. notifyInstancePoisoned( inst as unknown as { handles: Iterable }, e, @@ -1857,61 +1218,17 @@ export function createLiftedFunction(input: { }; /** - * Is `e` a *capability* signal rather than a genuine trap? - * - * `NeedsJspi` and `PendingCapability` mean "this runtime is incomplete", - * not "the component faulted". Poisoning on them is wrong on the - * reference's own terms: the operation they stand in for — a synchronous - * stream copy, `waitable-set.wait`, a blocking cross-component call — - * *blocks and then completes* in definitions.py. Every one of those - * executions returns normally there, so the instance stays healthy. - * Poisoning would attribute a permanent fault to a component - * that, on a complete runtime, is perfectly healthy — and it cascades: - * one unsupported operation made every later call on that instance report - * `cannot enter component instance`, which is neither our real behaviour - * nor the reference's. - * - * What unwinding must still do on this path, and what it must not: - * - * - MUST unwind the FACT sync-call scopes and restore `may_leave` - * (`unwind`), for exactly the reasons it does after a trap: a bail-out - * mid-adapter skips `exit-sync-call` and the `may_leave` restore, and - * that state is shared with sibling instances. - * - MUST NOT try to "finish" the abandoned operation. A stream end left - * in `CopyState.COPYING` with its buffer parked in the shared object is - * the honest record of "this copy never happened"; the counterpart has - * not been notified and must not be, because on a complete runtime the - * copy would still be pending. Likewise a `prepare-call` slot consumed - * by a `*-start-call` that then bailed is already cleared by - * `takePrepared`, so nothing leaks there. - * - MUST NOT resolve or cancel the task: the host call fails, and the - * task simply never resolved. - * - * In other words the instance is left exactly as a *pending* operation - * would leave it, which is the truthful state, and the only thing the - * embedder loses is the result of this one call. + * Capability failures unwind adapter bookkeeping without poisoning. + * They do not synthesize operation completion, task resolution or cancellation. */ const isCapabilitySignal = (e: unknown): boolean => e instanceof NeedsJspi || e instanceof PendingCapability; try { thread.resume(); - // definitions.py `canon_lift` (line 2213): the sync driving loop runs - // *inside* the enter/leave bracket, over the callee instance's threads. - // - // It is skipped in jspi mode, and must be. That loop resumes *ready* - // threads and traps when there are none — the reference's deadlock - // trap. A thread parked on a Promise is neither ready nor waiting: only - // a microtask turn can advance it, which a synchronous loop cannot give. - // Running it anyway declared a bogus deadlock the moment a sync-lifted - // export's activation suspended, which then trap-poisoned the instance - // and abandoned the activation mid-bracket — the orphaned - // `exit-sync-call` traced across phases 3h-3j. - // - // `drive` below is the correct driver in that mode: it knows about - // `store.awaiting`, still enforces the deadlock trap (no ready thread, - // no pending host call, nothing awaiting), and returns a Promise, which - // a jspi-mode lifted export returns anyway. + // The reference sync loop drives callee-instance threads. JSPI and + // asynchronous host dtors need the store driver instead so Promise + // continuations can run before the idle verdict. if (!ft.async && mode !== "jspi" && !input.allowAsyncCompletion) { driveSyncLift(task); } @@ -1925,49 +1242,18 @@ export function createLiftedFunction(input: { } /** - * The task outlived its driver (#292): hand the host a Promise settled by - * the task itself. - * - * Reached ONLY when this lift's own driver exited with the verdict - * `"idle"` (`DriveExit`) — it ran out of moves with the task unfinished. - * It is never reached after a `"done"` exit, however the store's state - * may have moved on since (polyengine#310). - * - * Resolution rides `finishHostEntry` unchanged — it already holds - * `completed`/`resultsToHost` — fired from THE TASK'S RESOLVE CALLBACK, - * i.e. `task.return` (polyengine#313). That is the reference's own answer - * event: definitions.py delivers a task's result to its caller through - * `on_resolve`, called from `Task.return_`, and run_tests.py's - * `lift_and_run` keeps ticking the store afterwards for OTHER work, not to - * produce the result. wasmtime's `call_concurrent` is the same shape. - * - * It used to fire on the task's LAST thread unregistering, which is not an - * event a callback-ABI task need ever reach: a guest that keeps spawned - * futures alive for the instance's life (wit-bindgen `spawn_local` — an - * event loop, a driver, an accept loop) leaves `task.threads` non-empty - * forever, so a lift that went idle before `task.return` and was later - * woken by another driver had its results captured and its host Promise - * left hanging. Nor does the answer need deferring to "no wasm call in - * flight": `driveDone`'s `midWasmCall`/`hopParked` clauses keep a DRIVER - * driving under a suspended activation, and on this path there is no lift - * driver left to stop — whichever driver is running when `task.return` - * happens keeps driving the store. + * After an idle exit, settle from onResolve, not the last thread's exit + * (#315 result-settlement rule; definitions.py `Task.return_`). Background + * producers may retain threads indefinitely. Already-captured results + * settle immediately; otherwise register resolution and poison waiters. * - * Rejection has two sources: `finishHostEntry` itself throwing, and the - * instance being poisoned by a later driver that ran this task into a - * trap. Neither calls `unwind()`. A trap on the background path happens - * under ANOTHER driver's `invokeNow`, whose own `catch` already unwinds - * the FACT sync-call scopes and restores `may_leave` — running it twice - * would restore sibling instances' `may_leave` from underneath a lift - * that driver is still mid-flight in. This mirrors what already happens - * to a post-`task.return` producer thread that traps later. + * This callback only shapes lifted values. It does not stop whichever + * driver now owns the task, nor unwind another driver's active FACT scopes. */ const backgroundCompletion = (): Promise => new Promise((resolve, reject) => { - // Already resolved, and only a driver-liveness clause of `driveDone` - // (a foreign hop, this task's own suspended thread, `driveAsync`'s - // idle probe) kept the verdict off `"done"`. The answer is in hand: - // settle now, nothing further will fire. + // An idle verdict can follow resolution while a driver-liveness + // clause remains false. Do not wait for an event that already fired. if (resolvedSeen) { try { resolve(finishHostEntry()); @@ -1997,81 +1283,15 @@ export function createLiftedFunction(input: { let outcome: DriveExit | Promise; try { - // Completion is "the task resolved AND its threads have drained", not - // merely "resolved". `task.return` resolves the task, but the activation - // is not finished until its implicit thread reaches - // `exit_implicit_thread` — for a callback task that means running the - // loop out to EXIT, which releases `inst.exclusiveThread`. - // - // In plain mode the two almost always coincide, because the generator - // runs to completion inside one `resume()`. Under JSPI they do not: the - // guest calls `task.return` while the activation is still suspended, so - // the old predicate let the driver return early and the thread was - // abandoned mid-loop — leaking the exclusive thread and its table slot. - // The lifted call is over when the task has resolved AND this task's - // activation is no longer mid-wasm-call. Those are two different events - // and both matter: - // - // * "task resolved" alone abandons a still-running activation. Under - // JSPI the guest calls `task.return` while suspended, so returning - // there left the callback loop parked forever — leaking the - // exclusive thread and its table slot. - // * "activation finished" alone deadlocks a *producer* guest, which - // legitimately keeps forwarding after `task.return` - // (wit-bindgen `wit_stream::new()` + a spawned loop). - // - // The distinguishing question is *what* the thread is parked on. An - // `awaitValue` park means a wasm call is in flight and will settle on - // its own, so we must keep draining. A park in `store.waiting` means the - // activation is waiting on a scheduler condition only the embedder can - // satisfy — that is a **background activation**: we return to the host - // and leave the thread live, and later `drive`/`pump` calls (host stream - // writes, the next export call) go on servicing it. - // - // AND THE PARK NEED NOT BE THIS TASK'S (issue #280). Scoping the - // wasm-call test to `task.threads` under-approximates the reference - // embedding, whose loop drains the whole store (`while store.waiting: - // store.tick()`, run_tests.py `lift_and_run`). What it missed is an - // activation THIS DRIVER PUT IN FLIGHT: a background task's host import - // settles on a microtask while this driver is live, this driver's - // `tick` resumes that task's callback activation, the promising entry - // hop-parks it (jspi pin (j) — contracts/intrinsics.md §"JSPI - // integration constraints" 4), and then this predicate — blind to - // another task's threads — declared the driver done. Nobody else owned - // that hop: the settlement pump arms only on outstanding real host - // calls, and the call that caused the resumption had already settled - // and left `pendingHostCalls`. Trace: `EXIT-done ... awaiting=1`, then - // an activation nothing services until an unrelated later call happens - // to drive the store. - // - // So the rule: a driver is not done while ANY thread of ANY task is - // hop-parked. A hop settles on the engine's own schedule, so waiting - // for it is bounded — the driver that caused it must see it land. The - // two tests are complementary and both stay: `hopParked` deliberately - // EXCLUDES genuinely JSPI-suspended activations (SuspensionPoint-owned - // parks, which only the embedder can satisfy and which are exactly the - // "background activation" case above), and `midWasmCall` is what still - // covers this task's own suspended thread. + // Driver completion is not thread exhaustion. Require a captured result, + // no awaiting wasm call of this task, and no entry hop anywhere in the + // store. Other tasks' genuine SuspensionPoint parks may remain background + // work, but any engine hop this driver started still needs servicing. + // Callback tasks can retain waiting threads after task.return; awaiting + // those threads' final exit would prevent long-lived producers returning. const midWasmCall = () => task.threads.some((t) => store.awaiting.has(t)); const hopParked = () => entryHopThreads(store).length > 0; const driveDone = () => resolvedSeen && !midWasmCall() && !hopParked(); - // ASYNC-TYPED EXPORTS DO NOT TRAP ON IDLE (#292). definitions.py - // `canon_lift` runs the driving loop — and with it the - // empty-candidate-set `trap_if` — only `if not ft.async_` (line 2189); - // for an async-typed export it returns right after the first - // `thread.resume()` and driving is the embedder's `Store.tick`, which - // never traps. wasmtime splits the same way (`run_concurrent` = - // `poll_until(trap_on_idle=false)` vs the `pub(super)` - // `run_concurrent_trap_on_idle` behind the blocking `call_async`), and - // polyengine's Promise-shaped export is the `call_concurrent` side. - // So the driver EXITS — it does not park and does not trap — and the - // task is left live for whichever driver next runs the store (the next - // export call, the settlement pump, a host stream op): exactly the - // between-calls liveness that already services post-`task.return` - // producer threads. Repro: an async export parked WAITing on an - // intra-component future a later export call writes. - // Sync-typed exports are unchanged: their loop traps on idle in every - // mode, which is what the paragraphs above describe. outcome = drive( store, driveDone, @@ -2083,22 +1303,7 @@ export function createLiftedFunction(input: { throw e; } /** - * Branch on the driver's EXIT VERDICT, never on a fresh `driveDone()` - * (polyengine#310). - * - * `driveDone` is a predicate over store-wide state that other drivers - * mutate. Re-evaluating it here — a microtask after the driver returned, - * on the asynchronous path — reads a different instant than the one the - * driver decided on. Observed: this lift's driver exited `EXIT-done`, - * then the settlement pump (servicing a settled host call belonging to - * another task) resumed a background activation that transiently - * hop-parked, so `hopParked()` read true in the continuation and the lift - * took `backgroundCompletion()` — a wait on an event that had already - * happened for this task (today a harmless detour; when the misroute was - * traced, `backgroundCompletion` waited for the task's LAST thread to - * unregister, i.e. never, for a task holding long-lived spawned futures). - * The verdict cannot rot that way: `"done"` means the driver saw the task - * finished, which stays true. + * Use the captured exit verdict, not a fresh shared-store predicate. */ const finish = (verdict: DriveExit): unknown => verdict === "idle" ? backgroundCompletion() : finishHostEntry(); @@ -2117,11 +1322,7 @@ export function createLiftedFunction(input: { }; return (...hostArgs: ComponentValue[]): unknown => { - // sync() arm 2: the synchronous variant refuses rather than deferring, and - // does so FIRST — before the arity check's sibling logic reaches - // `invokeNow` — because the refusal must be pre-enter to stay - // non-poisoning. See `refuseOnEntryHops` above for why the mode-keyed - // gate below cannot cover this variant. + // Synchronous callers refuse before entry rather than waiting on JSPI hops. if (input.refuseOnEntryHops && entryHopThreads(store, inst).length > 0) { throw new SyncEntryBusy(name); } @@ -2130,26 +1331,10 @@ export function createLiftedFunction(input: { `${name}: expected ${ft.params.length} argument(s), got ${hostArgs.length}`, ); } - // THE HOP-QUIESCENCE GATE (jspi mode only; hop_atomicity_test.ts). - // - // A promising-wrapped entry settles a microtask AFTER the guest's core - // call returns, even when nothing suspended (jspi pin (j)) — so there - // is a hop between core return and the host-side result LIFT, with - // nothing holding the instance against another host entry. In the - // reference no such window exists: `canon_lift` for sync options runs - // core + lift atomically. Admitting another host call into the - // window lets a full guest turn mutate the memory the pending lift - // will read — observed as `Trap: list too long` lifting the wosh - // engine's `tick` (`list>`) after a concurrent `feed-keys` - // turn reused the return area. - // - // The gate: defer this call until the instance has no HOP-parked - // activation. A hop-park is an `awaiting` thread with no owning - // `SuspensionPoint` — the same discriminator `hasRunnableWork` uses; - // genuinely JSPI-suspended activations (SuspensionPoint-owned) keep - // today's documented interleaving (the wasmtime-tracking divergence in - // jspi/bridge.ts), which host-import re-entry patterns rely on. - // Plain mode has no hops and keeps its synchronous fast path exactly. + // Preserve core-return/result-lift ordering across promising-entry hops: + // a new call must not reuse this instance's return memory before lifting. + // Genuine SuspensionPoint parks are excluded, allowing host-import reentry + // (`runtime/tests/jspi/hop_atomicity_test.ts`). This is not a general entry lock. if (mode === "jspi" && entryHopThreads(store, inst).length > 0) { return awaitHopQuiescence(store, inst).then(() => invokeNow(hostArgs)); } @@ -2158,16 +1343,9 @@ export function createLiftedFunction(input: { } /** - * Threads parked on a promising-entry hop: in `store.awaiting` with no - * `SuspensionPoint` owner in `store.waiting` (that would be a genuine JSPI - * suspension). Mirrors `Store.hasRunnableWork`'s (b)/(c) split. - * - * `inst` narrows the result to one component instance — what the - * hop-quiescence gate needs, since the memory a pending lift will read - * belongs to that instance. Omitted, the result is store-wide: what the - * export driver's `done` predicate needs (issue #280), where the question is - * not whose memory is at risk but whether any activation this driver put in - * flight is still mid-hop. + * Awaiting threads without a SuspensionPoint owner are crossing entry hops, + * not genuine scheduler parks. Narrow by instance for result-memory safety; + * omit `inst` for store-wide driver-completion and hand-off checks. */ function entryHopThreads( store: Store, @@ -2192,14 +1370,9 @@ function entryHopThreads( } /** - * Wait until `inst` has no hop-parked activation. Each settled hop is - * serviced synchronously (`serviceSettled` runs the lift segment), after - * which the activation either completed or re-parked; re-derive and - * repeat. Progress is guaranteed: a hop promise settles on the engine's - * own schedule, independent of any other activation of the instance, and - * a settled-but-unserviced hop resolves the race instantly. Multiple - * gated callers re-derive independently (no strict FIFO; starvation-free - * in practice because hops are sub-microtask). + * Await entry hops and service their tails before rechecking. Hops settle on + * the engine's schedule; genuinely blocked activations are excluded. Multiple + * gated callers recheck independently, with no FIFO admission guarantee. */ async function awaitHopQuiescence(store: Store, inst: unknown): Promise { for (;;) { @@ -2218,12 +1391,11 @@ async function awaitHopQuiescence(store: Store, inst: unknown): Promise { } // --------------------------------------------------------------------------- -// Host-initiated resource destructors (#160) +// Resource destructor entries // --------------------------------------------------------------------------- /** - * The canonical function type of a destructor: definitions.py - * `canon_resource_drop` (line 2326) — `FuncType([U32Type()], [], async_ = False)`. + * `canon_resource_drop` uses a sync function taking a u32 rep and no results. */ const DTOR_FT: FuncType = { params: [{ kind: "u32" }], @@ -2232,9 +1404,7 @@ const DTOR_FT: FuncType = { }; /** - * `CanonicalOptions(async_ = False)` (definitions.py line 2325): every field - * at its inert default. A dtor takes one flat `i32` and returns nothing, so - * no memory / realloc / post-return / callback is ever reached. + * `canon_resource_drop`'s sync options: no memory, realloc, post-return or callback. */ function dtorOptions(instance: ComponentInstanceState): ResolvedOptions { return { @@ -2251,27 +1421,14 @@ function dtorOptions(instance: ComponentInstanceState): ResolvedOptions { } /** - * Build the host-callable entry for a resource destructor — a full canonical - * **lift**, exactly as definitions.py `canon_resource_drop` (line 2319) does: - * - * ```python - * opts = CanonicalOptions(async_ = False) - * ft = FuncType([U32Type()], [], async_ = False) - * dtor = rt.dtor or (lambda rep: []) - * callee = inst.store.lift(dtor, ft, opts, rt.impl) - * ``` - * - * The host-initiated paths (embedder `drop()`, the GC backstop, `dropOwn`) - * route through this harness rather than calling the dtor bare, because the - * activation then has a real `Task` + implicit `Thread`: built-ins reached - * inside the dtor are well-attributed (a bare call leaves `currentTask()` - * with no ambient task — `PendingCapability`, or a foreign-task - * misattribution, the #24 class), and settled tails flow through - * `serviceSettled` like any other lifted sync call. + * Build the canonical destructor lift with its own Task and implicit Thread + * (`canon_resource_drop`). Host drops may use promising entry and return a + * Promise; their store driver owns async completion. * - * The returned function takes the rep and returns either `undefined` (the - * activation completed synchronously — the overwhelmingly common case) or a - * Promise, exactly like any lifted sync export in jspi mode. + * A present guestCaller selects a nested, plain sync lift, including when + * its value is null. Preserve the caller for same-instance poison semantics, + * reject thenable dtors, and finish without the host's store-wide hop drain. + * A guest dtor cannot suspend through this JS trampoline frame. */ export function createDtorEntry(input: { /** Diagnostic name; appears in deadlock/trap messages. */ @@ -2295,14 +1452,9 @@ export function createDtorEntry(input: { const guest = input.guestCaller !== undefined; const mode = guest ? "plain" : input.suspensionMode ?? "plain"; const raw: CoreFn = input.dtor ?? (() => undefined); - // A dtor's core type is `(i32) -> ()`, but the *host*-supplied dtors this - // helper also serves (embedder test doubles, `ResourceTypeInfo` built - // directly) are ordinary JS functions whose incidental return value would - // otherwise trip `normalizeCoreValues`' arity check. Discard it — except a - // thenable, which is the activation itself and must reach `awaitCore`'s - // park. Not applied in jspi mode: `WebAssembly.promising` only accepts a - // wasm callable, so the core must be passed through untouched there (and a - // real wasm dtor returns nothing by construction). + // Plain JS dtors may return incidental values; discard them. Preserve host + // thenables for awaitCore, but reject guest ones. Promising requires the raw + // wasm callable, whose result arity already matches `(i32) -> ()`. const core: CoreFn = mode === "jspi" ? raw : ((rep: number) => { const r = raw(rep); trapIf( @@ -2330,24 +1482,14 @@ export function createDtorEntry(input: { } /** - * Run a host-initiated drop of a guest (or host-implemented) resource rep — - * the observable remainder of `canon_resource_drop` for an owning handle when - * the holder is the host (`caller = None`, `Store.invoke`). - * - * A failure that arrives asynchronously has no frame to propagate into, so it - * is parked on the store's host-failure channel (first failure wins), where - * the next driven call surfaces it. The completion promise is deliberately - * NOT registered in `store.pendingHostCalls`: that registration was #160's - * lie — it claims *external* work for a promise whose settlement may need - * this very scheduler. The dtor's genuine external dependencies (its host - * imports) register themselves when they park. Poisoning on a trap now - * happens inside the lift harness (`poison()` in `createLiftedFunction`). + * Drop a host-held resource rep. Async failures go to the store's host-failure + * channel (first failure wins); traps are poisoned by the lifted entry. + * Do not register the dtor completion Promise as external work: it may need + * this scheduler. Its host imports register their own external dependencies. */ export function hostDtorCall(rt: ResourceTypeInfo, rep: number): void { const impl = rt.impl; - // An imported (host-implemented) resource has `impl === null` by - // construction (executor `bindImportedResources`): there is no component - // instance to gate entry into, so the dtor is called directly, as before. + // Imported host resources have no implementing component instance to enter. if (impl === null) { rt.dtor?.(rep); return; @@ -2373,16 +1515,9 @@ export function hostDtorCall(rt: ResourceTypeInfo, rep: number): void { } /** - * Call into wasm and hand back the result, awaiting it only if it is a - * Promise. - * - * This is the whole of the jspi entry seam. In **plain** mode the entry is not - * `promising`-wrapped, `callCore` returns core values, and this returns them - * without yielding — no await, no Promise allocation, the identical - * synchronous path plain mode always used. In **jspi** mode the entry *is* wrapped, so the - * call returns a Promise (jspi pin (e)) and we park the thread on it via the - * `awaitValue` block request; the driving loop resumes us with the values, or - * throws the rejection in (a post-resume trap). + * Enter wasm under a synchronous ambient bracket. Plain results return + * directly; promising results park the generator until a driver delivers + * their value or throws their translated rejection into the body. */ export function* awaitCore( fn: CoreFn, @@ -2390,10 +1525,7 @@ export function* awaitCore( // deno-lint-ignore no-explicit-any thread: any, ): Generator { - // Enter wasm with the activation-attached ambient in scope. In jspi mode the - // engine captures this context when it registers its resumption, so a - // built-in called by the resumed activation can recover its thread even when - // nobody is driving (see `withActivation`). + // The bridge explicitly maintains ambient claims after this bracket unwinds. const raw = withActivation(thread, () => callCore(fn, args)); // `callCore` normalizes a bare value to a one-element array; a promising // entry yields `[Promise]`. @@ -2401,10 +1533,7 @@ export function* awaitCore( const settled = yield { readyFunc: null, cancellable: false, - // A rejection of the promising Promise is a core trap by another route - // (jspi pin (e)); translate it exactly as `callCore` translates a - // synchronous throw, so the embedder sees one `Trap` vocabulary in both - // modes (see `mapCoreException`). + // Map post-resumption RuntimeError just like a synchronous core throw. awaitValue: Promise.resolve(raw[0] as unknown as Promise).then( undefined, (e) => { @@ -2420,7 +1549,7 @@ export function* awaitCore( return raw; } -/** definitions.py `CallbackCode` (line 2220). */ +/** definitions.py `CallbackCode`. */ enum CallbackCode { EXIT = 0, YIELD = 1, @@ -2428,7 +1557,7 @@ enum CallbackCode { } const CALLBACK_CODE_MAX = 2; -/** definitions.py `unpack_callback_result` (line 2226). */ +/** definitions.py `unpack_callback_result`. */ export function unpackCallbackResult( packed: number, ): [code: CallbackCode, waitableSetIndex: number] { @@ -2444,9 +1573,7 @@ export function unpackCallbackResult( } /** - * The body of `canon_lift`'s implicit thread (definitions.py line 2155), - * as a generator so its block points are real suspension points of the - * host-side thread model (see task/scheduler.ts). + * `canon_lift`'s implicit-thread body, with generator block points for the driver. */ function* liftBody(input: { name: string; @@ -2496,17 +1623,8 @@ function* liftBody(input: { } if (opts.callback === null) { - // definitions.py line 2179: `[] = call_and_trap_on_throw(callee, flat_args)` - // — the guest keeps running on its own stack and blocks inside wasm at - // whatever built-in it chooses. There is no return-to-host between the - // call and the block, so the only way to model it is genuine wasm-frame - // suspension. - // - // In jspi mode that is exactly what happens and no special handling is - // needed: the entry is `promising`-wrapped, so the activation suspends on - // whichever blocking built-in it reaches and `awaitCore` parks this thread - // until it finishes. Results arrive through `task.return`, so there is - // nothing to lift here. + // Stackful async execution needs JSPI. Results arrive through task.return, + // not the core function's return value. if (input.mode !== "jspi") { needsJspi( `stackful async lift of export '${name}' (async canonical options ` + @@ -2518,16 +1636,8 @@ function* liftBody(input: { return; } - // --- callback ABI (definitions.py lines 2183-2214) ---------------------- - // - // Stackless by construction: every wasm activation *returns* a packed code, - // and all waiting happens on the host side between activations. This is the - // path wit-bindgen 0.60 emits for every async export, and it needs no JSPI. - // The callback export is the second of the three entries that can reach a - // blocking built-in (jspi/bridge.ts's invariant), so it is wrapped exactly - // like the lifted core. Leaving it plain while the core was promising was a - // *mixed* activation, which pin (c) punishes: the first Suspending import - // it reached would trap. + // Callback ABI waits between invocations without JSPI, but callbacks in + // JSPI mode need the same promising wrapper as the initial core entry. const callback = enterWasm( require(opts.callback, `${name} callback`)!, input.mode, @@ -2546,23 +1656,10 @@ function* liftBody(input: { // --------------------------------------------------------------------------- /** - * Build the core-callable body for one lowered host import (reference - * `canon_lower`, definitions.py line 2242). - * - * Sync and async lowers share one `Subtask` and one pair of - * `on_start`/`on_resolve` closures, exactly as the reference does; the sync - * case is the degenerate one where the callee resolves before returning. - * - * The host callee is a plain JS function. If it returns a **Promise**, the - * subtask resolves when that promise settles: - * - * * async lower — fully supported and JSPI-free. The guest gets a STARTED - * subtask back, joins it to a waitable set, returns WAIT from its - * callback, and the scheduler delivers the SUBTASK event once the promise - * settles. This is the flagship capability of this phase: an ordinary - * `async` JS function is a valid Component Model async import. - * * sync lower — the guest's wasm frame would have to block - * (`thread.wait_until(subtask.resolved)`, line 2286), so: `needsJspi`. + * Build a `canon_lower` host-import body. Async lowers return a subtask handle + * for pending host Promises and deliver results through events. A sync lower + * must wait inside the caller's wasm frame, requiring both JSPI mode and the + * declaration's `suspending()` marker. Both forms share resolution/lender rules. */ export function createLoweredImport(input: { name: string; @@ -2610,7 +1707,7 @@ export function createLoweredImport(input: { ); } - // definitions.py lines 2250-2256. + // `canon_lower`: async results are written indirectly, not returned in lanes. const maxFlatParams = opts.async ? MAX_FLAT_ASYNC_PARAMS : MAX_FLAT_PARAMS; const maxFlatResults = opts.async ? 0 : MAX_FLAT_RESULTS; @@ -2676,42 +1773,12 @@ export function createLoweredImport(input: { subtask.resolve(SubtaskState.RETURNED, flatResults); }; - // --- invoke the host callee (the reference's `callee(...)`, line 2283) -- - // - // definitions.py assigns the callee's `OnCancel` here: - // `subtask.on_cancel = callee(on_start, on_resolve, caller = ...)` - // - // The `OnCancel` is the CALLEE's to supply: `Store.invoke` takes it back - // from the callee it invoked (`on_cancel = f(on_start, on_resolve, caller - // = None)`, definitions.py line 572), i.e. the reference expects the - // embedding to hand back the cancellation behaviour of whatever it is - // hosting. A wasmtime host gets a real one for free — dropping a Rust - // future IS cancellation. A JS Promise has no such channel, so polyengine - // answers on the host's behalf; §"Functions and async" makes the DEFAULT answer the - // reference's prompt-cancel host (`on_cancel = () => on_resolve(None)`), - // installed by the async arm below. - // - // The no-op assigned HERE is only the placeholder for paths where - // `subtask.cancel` is unreachable, so no answer can ever be demanded of - // it: an eagerly-resolving callee never mints a subtask handle (the - // fast-path return below is a bare state), and a sync-typed import's suspending mark - // park never mints one either. It is also the FINAL handler for a - // `deferCancel()`-branded import — accept and ignore, the pre-cancellation discard - // behaviour, now per-declaration. - // - // Leaving `on_cancel` null instead made a *legal* `subtask.cancel` crash - // with an internal AssertionError, which is neither reference behaviour - // nor a sanctioned incompleteness signal. + // The host supplies cancellation policy (`canon_lower`'s on_cancel). + // This no-op is final for deferCancel imports and paths without a subtask + // handle. Pending async calls otherwise install prompt discard below. subtask.onCancel = () => {}; - // abortable() (contracts/embedder-api.md §"Functions and async"): a marked import - // is handed a fresh `AbortSignal` after its WIT-declared parameters. The - // mark controls the SIGNATURE UNCONDITIONALLY — a marked function receives - // a signal on every call, including the paths where it can never fire - // (sync-typed, eager resolve, `deferCancel`) — so the host's arity is a - // property of its declaration, not of how a particular call happened to - // go. `new AbortController()` is evaluated only for marked imports, which - // keeps bare engine shells with no `AbortController` off this path for the - // whole unmarked corpus. + // abortable changes the signature on every call, even when cancellation + // cannot fire. Unmarked imports do not require AbortController support. const controller = abortable ? new AbortController() : null; const args = onStart(); const raw = controller === null @@ -2723,20 +1790,9 @@ export function createLoweredImport(input: { if (isPromiseLike(raw)) { if (!opts.async) { if (mode !== "jspi" || !suspendable) { - // definitions.py line 2286: `thread.wait_until(subtask.resolved)` — - // blocking the calling *wasm frame*. Parking needs BOTH jspi mode - // and the embedder's per-declaration `suspending()` marker: the - // Suspending wrap is applied per-declaration (`importValue`), so an - // unmarked import physically cannot suspend, whatever the mode. - // - // A capability signal is expressly NON-poisoning (the - // trap-unwind/lender-release obligation, contracts/intrinsics.md §A): - // the caller keeps running, so the borrows - // `onStart` lifted into this subtask must be discharged here or - // its lenders stay elevated forever and later `resource.drop`s - // trap "handle still lent out" on a healthy instance (found - // during the #106 closure; same class as the fact_calls.ts #91 - // sites). + // An unmarked import cannot suspend even in JSPI mode. This + // non-poisoning capability exit must release onStart's lenders + // (contracts/intrinsics.md, trap-unwind/lender-release obligation). subtask.unwindLenders(); needsJspi( suspendable @@ -2749,32 +1805,11 @@ export function createLoweredImport(input: { `(contracts/embedder-api.md §"Functions and async")`, ); } - // The park: the reference's plain, NON-cancellable wait — a - // cancel request against the caller stays pending-cancel and is - // delivered at its next cancellable wait, exactly as for any other - // mid-frame block. The instance-entry gate stays HELD across the park - // (the #43 hold rule; see `blockCurrentActivation`'s GATE LIFETIME - // note). - // - // The settle handler only RECORDS the outcome. All CABI work — - // `onResolve`'s result lowering (which may re-enter the guest through - // realloc) and `deliverResolve` — is deferred to `produce`, which - // runs at resume time under the suspension point's ambient claim. - // Lowering from the bare promise continuation instead would execute - // guest code in an unattributed chunk — the issue-#24 class the - // attribution sentinels exist to prevent. + // `canon_lower`'s sync wait is non-cancellable: pending cancellation + // waits for the caller's next cancellable point. Parking does not + // release callback exclusivity. Record the host outcome here, but do + // CABI lowering and lender delivery in produce at scheduler resume. let outcome: { value: unknown } | { error: unknown } | undefined; - // The async arm runs `onResolve` — result lowering, including possible - // realloc re-entry into the guest — in this bare promise continuation, - // where the sync arm above defers all CABI work to `produce` (the - // issue-#24 attribution note). The asymmetry is deliberate (#93): here - // no wasm frame is suspended mid-call — the guest returned BLOCKED and - // is between activations, which is exactly when the reference's - // `on_resolve` runs (the callee's turn), so there is no activation for - // the sentinels to attribute this chunk to. Lowering failures are host - // failures, not guest traps: they land on `store.hostFailure` and the - // driving loop raises them site-named (pinned by - // tests/async_lower_onresolve_failure_test.ts). const promise = Promise.resolve(raw).then( (v) => { store.pendingHostCalls.delete(promise); @@ -2785,37 +1820,10 @@ export function createLoweredImport(input: { outcome = { error: e }; }, ); - // Registered so the driver's deadlock probe counts this park as - // externally-wakeable (driveAsync: `pendingHostCalls.size === 0` is a - // precondition of the deadlock verdict) and so teardown can observe - // the outstanding call, mirroring the async arm below. + // Mark this park externally wakeable for drivers and teardown. registerHostCall(store, promise); - // LENDER DISCHARGE ON EVERY SETTLE PATH (#106, the sibling of the - // fact_calls.ts sync-start park's #102 enumeration): - // - // * produce SUCCESS -> `onResolve` + `deliverResolve` release the - // lenders; the `onSettled` backstop below observes - // `resolveDelivered()` and is a no-op. - // * produce THROW -> exempt under the trap-unwind/lender-release - // obligation (contracts/intrinsics.md §A: release is owed only on exits - // that do NOT poison the caller). Every rejection that reaches - // this park is a poisoning trap in the CALLER's own frame: - // branded `ComponentException`s on fallible imports were already resolved - // into err-shaped VALUES by the conventions layer - // (embedder/instantiate.ts `#wrapImportFn`'s `fail` — they take - // the success arm above), every other conventions-layer throw is - // a `Trap`, and a raw-executor rejection is a declared host bug - // that traps (empirical fact (e)). No capability signal can - // originate inside `produce`: this park only exists once jspi + - // `suspending()` were both granted. The backstop's unwind here is - // belt-and-braces bookkeeping on a poisoned instance, not an - // obligation. - // * abandon -> produce never runs, and an abandoned park - // does NOT poison the caller (pinned by - // resource_lender_park_settle_test.ts) — without the hook the - // subtask's lenders stayed elevated forever and later - // `resource.drop`s trapped "handle still lent out". The hook is - // the fix. + // Success delivers lenders in produce. onSettled is the idempotent + // backstop for produce failure or abandonment, which skips produce. return blockCurrentActivation({ store, task: currentTask(), @@ -2824,14 +1832,8 @@ export function createLoweredImport(input: { produce: () => { const done = outcome as { value: unknown } | { error: unknown }; if ("error" in done) { - // A rejection of a sync-typed import is a host failure: it - // reaches the guest as a rejection of the import's Promise, - // which the engine turns back into a wasm trap (empirical - // fact (e); `SuspensionPoint` routes a produce-throw through - // exactly that path). Branded `ComponentException`s never reach the raw - // boundary — the conventions layer resolves them into - // err-shaped values one layer up (see the settle-path - // enumeration above). + // Reject the import Promise to unwind the guest. The conventions + // layer has already converted fallible ComponentExceptions to values. throw done.error; } onResolve(toResults(done.value)); @@ -2845,19 +1847,13 @@ export function createLoweredImport(input: { onSettled: () => subtask.unwindLenders(), }); } + // Async lowering runs on host settlement, not in a suspended caller's + // produce step. Result-lowering failures use the host-failure channel. const promise = Promise.resolve(raw).then( (v) => { store.pendingHostCalls.delete(promise); - // cancellation discard: the subtask may already be resolved when the host promise - // settles — the discard `onCancel` below resolved it - // CANCELLED_BEFORE_RETURNED (the only pre-settle resolver on this - // arm). The value has no addressee, and `onResolve` would run - // straight into its `state === STARTED` assert ("on_resolve on a - // subtask that never started") and park that AssertionError on - // `store.hostFailure`, poisoning whatever unrelated embedder call - // came next. - // POISONED is the same discard (arch §6 #173): no addressee, and - // lowering would write into the corpse's memory via its `realloc`. + // Discard cancelled or poisoned recipients before lowering can + // write guest memory or re-enter through realloc. if (subtask.resolved() || isInstancePoisoned(opts.instance)) return; try { onResolve(toResults(v)); @@ -2867,61 +1863,25 @@ export function createLoweredImport(input: { }, (e) => { store.pendingHostCalls.delete(promise); - // Same guard, different reason: a rejection of a RENOUNCED call is - // not a host failure. The guest cancelled and was told so; surfacing - // the rejection would fail an unrelated later call with the error of - // an operation nobody is waiting for. POISONED is the same discard - // (arch §6 #173): it would fail a HEALTHY sibling's export call. + // Late rejection of discarded work must not fail an unrelated call. if (subtask.resolved() || isInstancePoisoned(opts.instance)) return; store.hostFailure = e; }, ); registerHostCall(store, promise); if (!deferCancel) { - // cancellation discard DISCARD (contracts/embedder-api.md §"Functions and async"; - // polyengine#241) — the reference's prompt-cancel host, - // `on_cancel = () => on_resolve(None)` (definitions.py canon_lower's - // null branch, line ~2267). - // - // This runs synchronously inside `canon_subtask_cancel`, which already - // set `cancellationRequested` before calling us (the assert in - // `onResolve`'s null branch relies on that ordering). `onResolve(null)` - // arms the SUBTASK event — a delivery-time thunk — and resolves - // CANCELLED_BEFORE_RETURNED, so the built-in's `finish()` tail consumes - // the event, `deliverResolve` releases the lenders (the #106 class, - // discharged exactly as a RETURNED delivery would), and BOTH cancel - // forms return the state without blocking. The null path lowers - // nothing, so there is no realloc re-entry from inside a built-in. - // - // The renounced call can no longer wake the guest, so it must stop - // counting as externally-wakeable for the driver's deadlock probe: - // deregister it NOW. (The settle continuation above also deletes; - // `Set.delete` is idempotent.) + // Prompt-cancel host policy (`canon_lower`'s on_resolve(None)). + // canon_subtask_cancel sets cancellationRequested before calling us. + // Deregister external work, resolve cancellation, then let event delivery + // discharge lenders. The null result path performs no realloc. subtask.onCancel = () => { store.pendingHostCalls.delete(promise); onResolve(null); if (controller !== null) { - // abortable(): tell the host its result was discarded, so it can stop the - // underlying operation — clear a timer, abort a fetch, close a - // dial. Reachable only from this arm by construction: a - // `deferCancel()` import never discards, so its signal never - // fires. - // - // Deferred one microtask. This closure runs SYNCHRONOUSLY inside - // `canon_subtask_cancel`, i.e. inside a live guest activation, and - // host abort listeners must not execute there — that is the - // issue-#24 attribution class, plus arbitrary re-entrancy into a - // guest mid-built-in. `Promise.resolve().then`, not - // `queueMicrotask`: the latter does not exist in bare engine - // shells (see jspi/bridge.ts's SENTINEL_TICK note). - // - // The resulting order is: the guest observes - // CANCELLED_BEFORE_RETURNED first, the host observes the abort a - // tick later. Any settlement the abort provokes (typically an - // `AbortError` rejection) arrives at the settle continuation above - // with the subtask already resolved, so it lands on the cancellation discard - // resolved-subtask guards and is discarded like any other late - // settlement — never a `store.hostFailure`. + // Defer host abort listeners until after the guest built-in returns. + // Cancellation is already resolved, so abort-induced settlements + // hit the discard guards. Promise reactions also work in bare shells + // without queueMicrotask. deferCancel imports never reach this arm. Promise.resolve().then(() => controller.abort()); } }; @@ -2930,7 +1890,7 @@ export function createLoweredImport(input: { onResolve(toResults(raw)); } - // definitions.py line 2284: a sync-*typed* callee must have resolved. + // `canon_lower`: a sync-typed callee must have resolved. assert_( ft.async || subtask.resolved(), `${name}: a non-async-typed import must resolve before returning`, @@ -2950,7 +1910,7 @@ export function createLoweredImport(input: { return flatResults; } - // --- async lower (definitions.py lines 2289-2309) ---------------------- + // Async lower: eager resolution needs no handle or event. if (subtask.resolved()) { // Eager-resolve fast path: no handle, no event, no waitable — the guest // learns the call is done from the return value alone. @@ -2968,15 +1928,8 @@ export function createLoweredImport(input: { } /** - * The callback-ABI dispatch loop of `canon_lift` (definitions.py lines - * 2183-2214), factored out so both entry points share one implementation: - * - * * a host-boundary lift (`liftBody` above), and - * * a FACT cross-component call, where the host invokes an async-lifted - * callee on the caller's behalf (`intrinsics/fact_calls.ts`). - * - * `packed` is the code the *initial* activation returned; the loop runs until - * it sees EXIT, invoking the callback export with each delivered event. + * `canon_lift` callback loop shared by host lifts and FACT calls. + * Start with the initial activation's packed code, dispatching events until EXIT. */ export function* runCallbackLoop(input: { name: string; @@ -2991,27 +1944,15 @@ export function* runCallbackLoop(input: { let [code, si] = unpackCallbackResult(input.packed); while (code !== CallbackCode.EXIT) { - // definitions.py line 2187, verbatim shape: the implicit thread of a - // needs-exclusive callback task holds the slot on every loop iteration. - // (The former per-iteration `holding` check tolerated a resolved task - // that had released the slot at a mid-frame block — the release-at-BLOCK - // divergence removed by issue #43. Under the hold rule, which is both the - // reference's and wasmtime's — `do_not_enter` is set for each callback - // invocation, concurrent.rs :942/:960 — the invariant is unconditional.) + // Each invocation holds exclusivity through mid-frame suspension, even + // after task.return. Only the between-invocation wait releases it. assert_( task.needsExclusive() && inst.exclusiveThread === task.implicitThread, "callback loop without holding the exclusive thread", ); - // Releasing the exclusive thread across the wait is what lets *another* - // task of the same instance enter and run while this one waits — the - // whole point of the callback ABI (definitions.py line 2188). Equally, - // RETAKING it below is what defers event delivery to a parked-between- - // invocations task while any invocation of this instance is mid-frame: - // the `() => inst.exclusiveThread === null` guard on the wait is the - // reference's `wait_for_event_and(lambda: not inst.exclusive_thread)` - // (line 2199) and wasmtime's `GuestCall::is_ready` DeliverEvent arm, - // which requires `!do_not_enter` (concurrent.rs :765). + // Admit other needs-exclusive tasks between invocations. Event delivery + // and cancellation wait for the slot to be free before reclaiming it. inst.exclusiveThread = null; let event: EventTuple; switch (code) { diff --git a/runtime/src/exec/executor.ts b/runtime/src/exec/executor.ts index d44535c..f11b364 100644 --- a/runtime/src/exec/executor.ts +++ b/runtime/src/exec/executor.ts @@ -6,7 +6,7 @@ // - formatVersion validation (via plan loader), fail fast // - strict initializer order; semantics per wasmtime GlobalInitializer // - instantiate-time (not call-time) failure for unsupported trampolines / -// ops (milestone-aware, contracts/intrinsics.md) +// ops (contracts/intrinsics.md) // - component hash verification against plan.component import type { ComponentValue, FuncType, ValType } from "../cabi/types.ts"; @@ -81,8 +81,8 @@ const SUSPENDABLE_TRACE = (() => { * `{ "ns:pkg/iface": { f: (…) => … } }`. * * Leaf values by import kind: - * - `func` — a JS function; arguments/results are host-shaped - * component values (contracts/descriptor-ir.md). + * - `func` — a JS function using raw component values, not facade + * conventions (contracts/descriptor-ir.md). * - `resource` — a `HostResourceType` (see `hostResourceType`). * - `instance` — a plain object; only its leaves are ever read. * - `module` — not supported (see `InstantiateModule::Import` below). @@ -127,30 +127,20 @@ export interface InstantiateInput { /** Verify plan.component.sha256 against componentBytes (default true). */ verifyHash?: boolean; /** - * Opt in to JSPI-backed suspension (docs/architecture.md §6 role 1-3). - * - * Off by default, and deliberately so: in this mode every lifted export - * returns a Promise (empirical fact (e) — `WebAssembly.promising` always - * does), which is an API-shape change. Ignored on an engine without JSPI, - * where every blocking site keeps raising the precise `NeedsJspi` it raises - * today (the browser-matrix degradation path; see `just browsers`). + * Override automatic JSPI selection. By default, the plan's blocking sites + * or a suspending-marked host import enable JSPI when the engine supports + * it. `false` forces plain mode; `true` requests JSPI even without that + * evidence. Unsupported engines stay plain and blocking sites raise + * NeedsJspi. Promising-wrapped raw exports return Promises. */ jspi?: boolean; /** * A plan already converted by `loadPlan`, used instead of re-loading. * - * Why this exists: the conventions layer (`src/embedder/`) must have the - * per-instantiation `ResourceTypeInfo` identity tokens and the converted - * types table *before* instantiation begins, because host imports genuinely - * fire DURING it — a core module's `start` function runs inside - * `runInitializers`, and real guests do call imports from it (Go's runtime - * calls `monotonic-clock.now()` from `schedinit`). Reading them off the - * returned `ComponentHandle.loadedPlan` is therefore too late. Handing the - * same `LoadedPlan` in keeps the tokens identical on both sides. - * - * Contract: one `LoadedPlan` per instantiation (tokens must be fresh per - * component instance), and `loadedPlan.wire` must be `plan`. Both are - * checked. + * The facade needs resource tokens before core start functions can invoke + * its imports. Sharing this object keeps both sides' type identities equal. + * Use a fresh LoadedPlan per instantiation; only `loadedPlan.wire === plan` + * is checked here, not whether the LoadedPlan was previously used. */ loadedPlan?: LoadedPlan; /** @@ -159,7 +149,7 @@ export interface InstantiateInput { * An async-typed export whose task parks on something only a *later* call * can ready — a `future.read` on an intra-component future — is not a * deadlock: definitions.py `canon_lift` runs its trapping driving loop only - * `if not ft.async_` (line 2189), leaving the driving to the embedder. So + * `if not ft.async_`, leaving the driving to the embedder. So * by default such an export's Promise simply stays pending. Setting this * restores the trap, which is what a *blocking* call wants — wasmtime's * `run_concurrent_trap_on_idle` behind `[Typed]Func::call_async`. @@ -192,18 +182,10 @@ export interface ComponentHandle { */ omittedExports: Map; /** - * The plan as loaded for THIS instantiation. - * - * Exposed for the conventions layer (`src/embedder/`), which needs the two - * things only the executor's own `loadPlan` call can supply: the per-instance - * `ResourceTypeInfo` identity tokens (`resourceTokens`) — the same objects - * the `own`/`borrow` types in every signature point at, and the only route to - * a resource's destructor for a *host-initiated* drop of a guest handle - * (definitions.py `canon_resource_drop` runs `rt.dtor(rep)`; the host holds - * reps, never table indices, so there is no handle to drop through) — and the - * converted `types` table it reads function signatures from. - * - * Introspection only: mutating it is undefined behaviour. + * This instantiation's converted types and resource identity tokens, + * including destructor wiring for host-held reps. May be supplied by the + * facade through InstantiateInput.loadedPlan. Introspection only: do not + * mutate it after instantiation. */ loadedPlan: LoadedPlan; } @@ -226,9 +208,8 @@ const moduleCache = new WeakMap(); export async function instantiateComponent( input: InstantiateInput, ): Promise { - // Re-load per instantiation: resource identity tokens must be fresh per - // component instance (descriptor-ir.md open item on ResourceTypeInfo). - // Re-load unless the caller already did (see `InstantiateInput.loadedPlan`). + // Fresh resource identities per instantiation, unless the caller supplied + // its own fresh conversion (InstantiateInput.loadedPlan). const loaded = input.loadedPlan ?? loadPlan(input.plan); if (loaded.wire !== input.plan) { throw new PlanError( @@ -300,16 +281,8 @@ class Executor { /** The single in-flight FACT `prepare-call` state (intrinsics/fact_calls.ts). */ readonly preparedCall: { current: PreparedCall | null } = { current: null }; /** - * One `LiveMemory` per `RuntimeMemoryIndex`, memoized. - * - * definitions.py's `LiftOptions.equal` (line 643) compares memories by - * *identity* (`lhs.memory is rhs.memory`), and `canon_task_return` requires - * the options at the `task.return` site to equal the lifted export's. A - * fresh wrapper per `resolveOptions` call would make that comparison fail - * for every component that actually uses a memory — it only ever passed - * before because the async fixtures in play had `memory: null` on both - * sides. Memoizing restores wasmtime's semantics, where the comparison is - * on `RuntimeMemoryIndex`. + * One LiveMemory per RuntimeMemoryIndex. LiftOptions.equal compares memory + * identity, so task.return and its lifted export must share this wrapper. */ readonly liveMemories = new Map(); /** Set by the entry/import wrapping sites; checked in `finish`. */ @@ -347,30 +320,13 @@ class Executor { readonly factStartScopes: FactStartScope[] = []; /** - * Core functions exported by a core instance that imports at least one - * `Suspending`-wrapped trampoline (`trampolineCanBlock`, per DECLARATION — - * the async form of a copy built-in never blocks, is not wrapped and does - * not mark) or a function from an already-marked instance. FACT consults - * this to decide whether a callee needs its own `promising` entry. - * - * WRAPPED IMPLIES MARKED, and must: jspi pin (c) traps a Suspending import - * called from a non-promising activation unconditionally, plain-value path - * included (see `importValue`). - * - * Instance granularity is still an over-approximation — a module exporting - * both a blocking and a non-blocking function marks both — and so is the - * marking of `async-start-call`/`subtask-cancel` importers, whose wrap - * exists for a park that often does not happen. Neither produces wrong - * answers on the official corpus, because a needlessly-wrapped callee no - * longer changes observable state: `async-start-call` parks the caller - * until the callee is determinate (fact_calls.ts), reconstructing the - * reference's synchronous run-to-first-block across the engine's microtask - * hops (jspi pin (j)) — so an eagerly-completing callee still reports - * RETURNED rather than STARTED. - * - * Per-FUNCTION reachability (a call-graph pass in the translator, where - * wasmparser already is) would still shrink the set — as a wrapping-cost - * optimization now, not a correctness need. + * Core exports needing a promising entry when called through FACT. + * Classification is transitive and module-wide, not per-function + * reachability. Blocking imports mark guest and adapter exports; wrap-only + * imports mark guest exports but only propagate wrapFuncs through adapters. + * A Suspending import needs a promising activation even on a plain-value + * return. FACT's determinacy park preserves run-to-first-block results + * across that entry's mandatory microtask hop. */ readonly suspendableFuncs = new WeakSet(); @@ -410,32 +366,9 @@ class Executor { this.hostImports = input.imports ?? {}; this.verifyHash = input.verifyHash ?? true; this.trapOnIdle = input.trapOnIdle ?? false; - // AUTO-DETECTION IS ON by default. `chooseMode` picks jspi when the - // embedder opts in OR when the plan needs suspension: a stackful async - // lift, or a genuinely blocking built-in — classified per DECLARATION - // (`trampolineNeedsSuspension`; the async form of a copy/cancel built-in - // never blocks and is not evidence). An explicit `jspi: false` still - // forces plain, and a sync-only component never detects as needing - // suspension, so the synchronous API is untouched (pinned by - // bridge_test "plain mode: lifted exports still return values" and the - // planNeedsSuspension(hello) === false pin beside it). - // - // The detection-on failure inventory that kept this off is CLOSED — all - // suspension sites lit, zero failures over the full corpus. What - // protects each closed class: - // * STARTED-vs-RETURNED / eager-callee wrapping (big-interleaving's - // expect-codes, cross-abi's six): per-declaration classification + - // `async-start-call`'s determinacy park — pinned by - // tests/jspi/cross_abi_differential_test.ts (KNOWN_DIVERGENT is - // EMPTY and asserted empty) and fastpath_hop_test.ts (pin (j): the - // Suspending fast path still defers the continuation); - // * park/resume of a sync-lowered caller: handshake_test.ts pins; - // * stall-vs-trap verdicts (incl. the YIELD-spin starvation and the - // stale-race guard in exec/boundary.ts): deadlock_test.ts pins; - // * trap poisoning through rejections (`Thread.resumeWith` bracket) - // and start-function suspension mapping: the conformance suite's - // builtin-trap-poisons-instance / dont-block-start files, green - // under detection. + // Explicit false wins over plan/import evidence; unsupported engines + // remain plain (chooseMode). Declaration-level classification distinguishes + // a blocking operation from its nonblocking async form. this.suspensionMode = chooseMode( input.jspi, // Auto-detection evidence, two independent sources: the PLAN (a @@ -472,19 +405,9 @@ class Executor { } async compileModules(): Promise { - // Reuse compiled modules across instantiations of the same plan. - // Fresh-instance-per-case suite runs re-instantiate one component - // thousands of times, and recompiling costs real time per instantiation - // even when V8's byte-keyed module dedup hits (it still re-hashes every - // byte — ~7 ms for a 14 MB component). `WebAssembly.Module` is immutable - // and freely instantiable many times, so reuse cannot change semantics - // PROVIDED the compile inputs are the same objects: a hit requires the - // plan (WeakMap key), the component bytes, and every adapter buffer to - // be identical by reference. In-place *content* mutation of a reused - // buffer is caught before this runs by `verifyComponent`'s sha256 check - // whenever `verifyHash` is on (the default); a caller who disables that - // and mutates reused buffers gets stale modules — the same caller error - // as mutating them mid-compile today. + // Reuse immutable modules when plan, component bytes and adapter buffers + // match by identity. Callers must not mutate reused inputs. verifyHash + // checks component content, not adapter content or plan mutations. const cached = moduleCache.get(this.wire); if ( cached !== undefined && @@ -499,14 +422,8 @@ class Executor { } const compiled = Promise.all(this.wire.modules.map((m, i) => { if (m.kind === "embedded") { - // Defense-in-depth (polyengine#187): the loader now refuses - // negative/non-integer `offset`/`len` at load time (loader.ts - // `validateModule`), but a negative offset silently slices the - // *wrong* bytes from the tail of `componentBytes` - // (`Uint8Array.slice` treats negative indices as relative to the - // end) rather than tripping the old upper-bound-only check — - // belt-and-braces here in case a `LoadedPlan` ever reaches this - // path without going through `loadPlan`. + // Recheck supplied LoadedPlans before slicing: negative indices would + // select bytes from the tail instead of failing bounds validation. if ( !Number.isInteger(m.offset) || m.offset < 0 || !Number.isInteger(m.len) || m.len < 0 @@ -606,18 +523,8 @@ class Executor { ); } const importObject: WebAssembly.Imports = {}; - // Per-CORE-INSTANCE suspendability. `planNeedsSuspension` answers - // the question for a whole component; FACT needs it for the specific - // callee it is about to invoke, because that is what decides whether - // the callee must be `promising`-wrapped (see `mkCalleeTask`). - // - // The trampoline declarations cannot answer it: `sync-start-call` - // and `async-start-call` carry no `instance` field (verified against - // real plans). What CAN answer it is right here -- the import list - // of the module being instantiated. A core instance whose imports - // include a blocking trampoline is one whose code can reach a - // suspension point; every function it exports is therefore - // potentially-blocking, and everything else is not. + // Collect suspendability from this core module's imports. FACT + // needs callee-specific evidence, not the whole plan's mode choice. this.sawBlockingImport = false; this.sawWrapImport = false; // Which component instance this core module belongs to — the plan @@ -627,24 +534,13 @@ class Executor { this.#declaringInstance = init.instance === null ? null : this.componentInstance(init.instance); - // ISSUE #88: core wasm permits two imports with the same - // (module, field) pair (trusted wasmtime-environ 47.0.3 info.rs - // :438-445 gives one flat positional CoreDef per import slot, but - // WebAssembly.Module.imports(module) and the JS import object are - // both keyed by (module, field) name, not by slot). If two slots - // share a name and resolve to different values, the second object - // write silently wins and BOTH slots receive the last value — the - // JS API cannot express per-slot values for duplicate names. Detect - // this here and fail loudly rather than wire the wrong function in - // silently; identical values are safe (the API cannot distinguish - // the slots in that case, so nothing is actually lost). + // Wasm imports are positional; the JS import object is name-keyed. + // Duplicate names can only be represented when their resolved values + // are identical. Reject conflicts instead of overwriting a slot. const seenAt = new Map(); declared.forEach((imp, i) => { const before = this.sawBlockingImport; const value = this.importValue(init.args[i]); - // Standing probe (CE_COPY_TRACE): which import made this core - // instance suspendable — the first question to ask whenever a - // FACT callee is promising-wrapped that should not be. if (!before && this.sawBlockingImport && SUSPENDABLE_TRACE) { console.error( `[suspendable] module ${init.module}: import ` + @@ -701,16 +597,9 @@ class Executor { throw e; } if (this.sawBlockingImport || this.sawWrapImport) { - // Wrap-only evidence gives a GUEST module its own `promising` - // entry (jspi pin (c)) but leaves a FACT adapter - // (`instance: null`) out of `suspendableFuncs`: the adapter's - // pass-through export is what `*-start-call` receives as a LIFT - // CALLEE, and promising-wrapping an eagerly-completing callee is - // the STARTED-vs-RETURNED divergence `trampolineCanBlock` - // (jspi/bridge.ts) warns about — measured as - // test/async/drop-subtask.wast:140 under POLYENGINE_SCHED_SEED=1. - // Either way the evidence PROPAGATES, in its own tier, so the - // guest importing that adapter export still gets marked. + // Wrap-only evidence requires promising guest entries, but must + // not promote adapter pass-through lift callees. Propagate the + // tier so guests importing those adapters still get marked. const promising = this.sawBlockingImport || init.instance !== null; for (const exported of Object.values(instance.exports)) { if (typeof exported !== "function") continue; @@ -773,8 +662,7 @@ class Executor { } case "resource": { // Wire the dtor + implementing instance into every concrete - // resource-table token for this defined resource - // (tolerate-if-unreferenced; plan-format.md open item). + // resource-table alias for this defined resource. const dtor = init.dtor === null ? null : this.resolveFunction(init.dtor, `resource ${init.index} dtor`); @@ -789,32 +677,11 @@ class Executor { const token = this.loaded.resourceTokens[tableIndex]; token.impl = inst; token.dtor = dtor; - // #85/#160: the host-initiated-drop entry. A host-initiated - // drop is a full canonical LIFT of the dtor (definitions.py - // `canon_resource_drop`, line 2319), so it is built here with - // the same harness every lifted export uses — that is what - // gives the dtor's activation a real Task/Thread, and what - // releases the impl instance's entry bracket at the first park - // instead of holding it across the whole activation (#160). - // - // The `promising` entry wrapping (docs §7: in jspi mode a dtor - // may legally reach a `Suspending` import) is applied INSIDE - // `createLiftedFunction` per `suspensionMode`, and only when - // the dtor is suspension-capable (`suspendableFuncs`: its core - // instance imports a blocking trampoline). A non-suspendable - // dtor cannot legally suspend, so the plain entry is exact for - // it and avoids `promising`'s unconditional microtask hop - // (jspi pin (j)). The hop no longer risks a drop-then-call - // trap either way — the bracket is released before the drive, - // and the hop-quiescence entry gate covers the sequence — but - // the plain path stays the cheaper and more deterministic one. - // - // `WebAssembly.promising` rejects non-wasm callables (a dtor - // CoreDef can resolve to a JS trampoline) with a TypeError; - // fall back to the plain entry, where `awaitCore` still parks - // on a returned Promise. Deliberately does NOT set - // `wrappedEntries`: `finish()`'s invariant inventories the two - // primary wrapping sites; this is an auxiliary entry. + // Canonical lifted destructor: createDtorEntry supplies its + // Task/Thread so the store can service suspended completion. + // Use JSPI only for suspendable dtors, avoiding an unnecessary hop. + // A JS-trampoline dtor cannot be promising-wrapped; retry plain. + // This auxiliary entry is outside wrappedEntries' inventory. const suspendable = dtor !== null && this.suspensionMode === "jspi" && this.suspendableFuncs.has(dtor as unknown as object); @@ -884,13 +751,8 @@ class Executor { /** * Materialize one plan export. * - * The result is an explicit discriminated union rather than - * `unknown | undefined`: an earlier `if (built !== undefined)` filter meant - * *any* path that happened to yield `undefined` removed the export from the - * component's surface with no diagnostic anywhere. Only `type` exports are - * legitimately absent from the runtime surface, and they say so with a - * reason that is recorded on the handle (`omittedExports`); everything else - * either produces a value or throws. + * Only type exports may be omitted, with a recorded reason. An undefined + * value is not an omission; every other export must materialize or throw. */ buildExport( exp: WireExport, @@ -914,20 +776,10 @@ class Executor { // Async-typed exports only; see `InstantiateInput.trapOnIdle`. trapOnIdle: this.trapOnIdle, }); - // Every SYNC-TYPED export additionally carries a plain-entered - // variant (see SYNC_ENTRY, contracts/embedder-api.md §"Functions and async"): - // in jspi mode the promising-wrapped entry above necessarily returns - // a Promise, which some host contexts cannot use however promptly it - // resolves — a JS class constructor cannot await it at all, and the - // embedder's `sync()` adapter exists to ask for the synchronous form - // of any sync-typed export. Async-typed exports have no synchronous - // form by definition and get none. - // - // Deliberately NOT noteEntry()-recorded — this is the documented - // exception to the bridge invariant (entries wrapped iff imports - // wrapped), safe because a synchronously-completing activation never - // reaches the Suspending seam. sync() extends the exception from - // constructors to all sync entries. + // SYNC_ENTRY bypasses promising for sync() and resource constructors. + // It must not reach a Suspending import, even one returning a plain + // value. Async-typed exports have no sync form. This auxiliary entry + // is outside noteEntry's primary wrapping inventory. if (this.suspensionMode === "jspi" && !ft.async) { (value as unknown as Record)[ SYNC_ENTRY @@ -941,11 +793,8 @@ class Executor { trapState: this.trapState, syncCallStack: this.syncCallStack, allInstances: () => this.componentInstances.values(), - // sync() arm 2: a synchronous caller cannot be deferred by the - // hop-quiescence gate, so it refuses (SyncEntryBusy) instead. - // This deliberately changes constructor behaviour: the - // constructor sync entry previously bypassed the gate - // entirely, a latent lift-corruption window. + // A synchronous caller cannot await entry-hop quiescence; + // refuse with SyncEntryBusy instead of bypassing the gate. refuseOnEntryHops: true, }); } @@ -1037,13 +886,8 @@ class Executor { */ importValue(def: WireCoreDef): Importable { const value = this.resolveCoreDef(def); - // Suspendability is TRANSITIVE. FACT does not put blocking trampolines in - // the guest's own module: it generates an adapter module that imports - // them, and the guest imports the adapter's exported function. So a core - // instance is suspendable if it imports a blocking trampoline OR imports a - // function from an already-suspendable instance. Missing this closure is - // what made `async-calls-sync`'s sync-lifted middle look non-blocking and - // broke the handshake pins. + // Propagate both evidence tiers through imported exports, including FACT + // adapters between a guest and the actual blocking trampoline. if ( typeof value === "function" && this.wrapFuncs.has(value as unknown as object) @@ -1063,13 +907,8 @@ class Executor { return value; } const decl = this.wire.trampolines[def.index]; - // Per-DECLARATION blocking classification (jspi/bridge.ts): the async - // form of a copy/cancel built-in never blocks, so importing one neither - // needs a `Suspending` wrap nor marks the importer suspendable. The - // kind-only version of this test pulled every async-form consumer into - // `suspendableFuncs`, promising-wrapping FACT callees that complete - // eagerly — the STARTED-vs-RETURNED and missed-synchronous-cancellation - // divergences big-interleaving-test.wast asserts against. + // Classify the declaration, not just its kind. Blocking and async forms + // differ; trampolineCanBlock also includes JSPI determinacy parks. if (decl === undefined) return value; const optionsAsync = (i: number) => this.wire.canonicalOptions[i]?.async === true; @@ -1094,25 +933,9 @@ class Executor { ) as unknown as Importable; } if (!trampolineCanBlock(d, optionsAsync)) return value; - // WRAPPED IMPLIES MARKED. Anything handed to wasm as a - // `WebAssembly.Suspending` makes its importer's frames suspendable, so - // that importer's entries must be `promising`-wrapped — jspi pin (c) is - // unconditional: a Suspending import called from a non-promising - // activation traps EVEN WHEN it produces its value synchronously - // (measured: "trying to suspend without WebAssembly.promising"; with an - // outer promising entry and JS frames in between, "trying to suspend JS - // frames"). `async-start-call` and `subtask-cancel` used to be wrapped - // WITHOUT marking, on the grounds that a needlessly-promising callee - // reported STARTED where the reference reports RETURNED. That reason has - // expired: `async-start-call`'s determinacy park (intrinsics/fact_calls.ts) - // reconstructs the reference's run-to-first-block across the hop, which is - // what the note above `suspendableFuncs` already records as the mitigation. - // What the omission cost was a hard trap in the one shape where the - // importer had no other blocking import: test/async/reentrance.wast:429, - // whose `$MC` imports only `b`'s async-start-call, `waitable-set.new` and - // `waitable.join` — it trapped with SuspendError instead of reaching the - // deadlock verdict. The two tiers (`blockingFuncs`/`wrapFuncs`) keep the - // marking from over-reaching — see the post-instantiate marking. + // Every Suspending wrapper contributes evidence, even if it returns a + // plain value. Preserve the blocking/wrap-only distinction when marking + // the importer (see suspendableFuncs and post-instantiate propagation). if (trampolineNeedsSuspension(d, optionsAsync)) { this.sawBlockingImport = true; } else { @@ -1138,7 +961,7 @@ class Executor { case "trampoline": return this.trampoline(def.index); case "unsafe-intrinsic": - // plan v1: wasmtime compile-time builtins imported directly by a core + // Wasmtime compile-time builtins imported directly by a core // module. `context.{get,set}` become host functions over the *current // thread's* context slots (definitions.py `Thread.storage`); every // other symbol fails here, at instantiate time. @@ -1249,10 +1072,7 @@ class Executor { return this.componentInstance(instance); }, errorContextTableInstance: (i) => { - // plan v3: the error-context tables' own index space - // (`TypeComponentLocalErrorContextTableIndex`). Loud on absence — the - // predecessor of this accessor borrowed the *resource*-table mapping - // and could answer with a different instance's table (polyengine#89). + // TypeComponentLocalErrorContextTableIndex, not a resource-table index. const instance = this.loaded.errorContextTableInstances[i]; if (instance === undefined) { throw new PlanError( @@ -1315,26 +1135,12 @@ class Executor { const ft = this.funcType(decl.type, `import '${label}'`); const opts = this.resolveOptions(decl.options); const suspendable = isSuspending(value); - // cancellation discard (contracts/embedder-api.md §"Functions and async"): does this import - // opt out of cancel-discard? Unlike `suspendable` above, this needs no - // executor-state detour — the brand is consumed by `createLoweredImport` - // itself (it only decides which `onCancel` the lowered import installs, not - // whether the CoreFn gets wrapped), so nothing downstream has to read a - // brand off a replaced function identity. + // Cancellation marks are consumed inside createLoweredImport, before + // trampoline wrappers replace the function identity. const deferCancel = isDeferCancel(value); - // abortable() (same section): does this import want a per-call `AbortSignal`? - // Read exactly like `deferCancel` above and for the same reason — the - // brand is consumed inside `createLoweredImport`, which mints the - // controller and appends the signal itself, so no function identity is - // replaced downstream of the read. const abortable_ = isAbortable(value); - // The Suspending-wrap decision is taken in `importValue`, which sees the - // trampoline only AFTER `createTrampoline`'s trap-recording wrapper has - // replaced this function's identity — a brand on the CoreFn would die - // there (measured: the returned Promise coerced to 0 through the - // unwrapped import). Record the decision as executor state instead, - // keyed by LoweredIndex; `importValue` runs later on the same call - // stack, so the set is populated by construction when it reads. + // importValue reads this after createTrampoline wraps the function. + // Preserve suspension evidence by LoweredIndex, not function identity. if (suspendable) this.suspendableLowerings.add(decl.lowered); return createLoweredImport({ name: label, @@ -1378,7 +1184,7 @@ class Executor { } /** - * Element types of an interned *results tuple* — the `results` field of a + * Element types of an interned results tuple — the `resultType` field of a * `task-return` trampoline (the shim interns a lifted function's result * list as a single tuple type, `intern_results_tuple`). */ diff --git a/runtime/src/exec/host_streams.ts b/runtime/src/exec/host_streams.ts index 95d11bc..d517b4e 100644 --- a/runtime/src/exec/host_streams.ts +++ b/runtime/src/exec/host_streams.ts @@ -1,72 +1,17 @@ -// Host-side stream and future ends: the minimal embedder surface for the -// async value types. +// Host-side stream/future ends over task/streams.ts's shared rendezvous. +// HostBuffer supplies JS chunks; direct sessions instead use a scoped view +// of the peer's bytes. The shared object retains identity across lift/lower. // -// =========================================================================== -// WHY THIS IS SMALL -// =========================================================================== +// Host operations return Promises and pump the store when needed. HostActivity +// registers a wakeup promise while the host retains an end, so waiting for +// the embedder is not mistaken for component deadlock. Created wrappers keep +// their writable end across lowers; lifted wrappers give up their readable +// end on lower and regain it on re-lift (bindOnLower). // -// The rendezvous in `task/streams.ts` never touches linear memory. It only -// ever calls four methods on whatever buffer it is handed — `read`, `write`, -// `remain`, `isZeroLength` — and it passes the *shared* stream object around -// by identity. So a host end needs exactly two new things: -// -// * `HostBuffer`, a sibling of `GuestBuffer` implementing that same -// four-method surface over a plain JS array instead of guest memory; and -// * a way to park a host read/write until the guest shows up. -// -// Everything else is existing machinery. In particular the *value* that -// crosses the component boundary is the `SharedStreamImpl` itself, so passing -// a host stream to a guest goes through the ordinary `lowerStream` path -// (definitions.py `lower_stream`, line 1828 — wrap the shared object in a -// fresh `ReadableStreamEnd` in the callee's table) and a guest-returned stream -// arrives as the same kind of object from `liftStream`. No lift/lower code was -// added for this file. -// -// =========================================================================== -// SCHEDULING -// =========================================================================== -// -// A host read/write that cannot rendezvous immediately parks, exactly as a -// guest one does, and hands back a Promise. Two cases: -// -// * The guest is still running (it is what will complete the rendezvous). -// The host's `onCopyDone` fires synchronously inside the guest's -// `stream.read`/`stream.write` trampoline and the Promise resolves. -// * The *guest* is the parked side and only the embedder can make progress. -// Then `drive()` would otherwise see no ready thread and no outstanding -// host call and declare deadlock — correctly, for a component that really -// is stuck, but wrongly here. `HostActivity` below registers a -// re-arming promise in `store.pendingHostCalls` for as long as the host -// RETAINS a way to act, which is precisely the signal `driveAsync` -// already understands: "progress is possible, but only after a turn of -// the event loop". -// -// Retention, stated as the rule the arm implements (#162, embedder-api -// §"Streams and futures"): the arm is live iff the host holds a retained end, a parked -// host operation, or an unfinished producer pump. Which ends the host holds -// follows from where the wrapper came from — a host-CREATED stream keeps its -// writable end across every lower (only readable ends transfer, -// definitions.py `lower_stream` line 1828), while a LIFTED one holds just the -// readable end the guest passed out, so lowering that same object back into a -// guest (the `identity: async func(s: stream) -> stream` round trip) -// hands the host's last end away and the arm disarms. A later re-lift -// re-arms. See `bindOnLower` and `HostActivity` for the mechanism. -// -// The consequence, stated plainly: an embedder that lowers a host stream into -// a guest and then never writes to it or drops it will *hang* rather than -// trap. That is the honest outcome — the component is not deadlocked, the -// embedder simply has not done its half — and it matches how any other -// unresolved Promise behaves in JS. That policy is unchanged by deadlock-verdict suppression; what -// changed is that the claim now EXPIRES with retention, so a store that once -// round-tripped a stream through the host no longer misreports every later -// genuine deadlock as this hang. -// -// The inverse case is NOT a hang (#66, contracts/embedder-api.md §"Streams and futures"): when the -// GUEST side dies — a trap poisons the instance holding the peer end — the -// poisoned table's ends are retired (task/streams.ts -// `retireInstanceAsyncEnds`), so a parked host operation settles DROPPED- -// shaped here and the conventions layer rejects it with `PeerTrappedError`. -// Only embedder negligence hangs; a component fault is always loud. +// A retained producer that never acts can leave the guest waiting indefinitely. +// Poisoning instead retires ends still in the failed instance's handle table; +// the conventions layer distinguishes that retirement from clean stream end. +// See contracts/embedder-api.md, "Streams and futures". import { assert_ } from "../cabi/trap.ts"; import { despecialize } from "../cabi/types.ts"; @@ -94,17 +39,9 @@ import { } from "../task/mod.ts"; /** - * The `inst` a host end presents to the rendezvous. definitions.py compares it - * against `pending_inst` for the "same instance" restriction — a guard against - * interleaving two *lifts in one component instance's linear memory*, which is - * why it exempts number types (definitions.py `none_or_number_type`). A host - * end has no linear memory, so that restriction can never apply to it: each - * end gets its OWN sentinel (never equal to a real `ComponentInstanceState`, - * and never equal to the peer end's), so a host writer and a host reader may - * rendezvous directly for every element type. One shared sentinel used to - * stand for "the host" here, which made a post-pass-through host↔host copy of - * a non-number element type trap as "intra-component" (found by the #54 - * pass-through investigation). + * Distinct identity per host end. The reference's same-instance restriction + * concerns copies within one guest's memory, not host-to-host copies of + * nonnumeric values. Neither sentinel equals a guest or the opposite end. */ function hostEndInstance(role: "read" | "write"): unknown { return Object.freeze({ hostEnd: role }); @@ -133,12 +70,8 @@ export class HostBuffer { private readonly values: PayloadChunk | null, readonly length: number, ) { - // definitions.py `Buffer.MAX_LENGTH` (:919) is asserted on every buffer - // the spec builds (`BufferGuestImpl.__init__`, :938); `GuestBuffer` traps - // on it. A host buffer is not guest-visible, so a violation is embedder - // misuse rather than a component fault — hence a loud typed JS error and - // not a `Trap`. Caught at construction: an over-long host offer would - // otherwise silently exceed the spec bound (#97). + // Apply Buffer.MAX_LENGTH at the host boundary too. Invalid host capacity + // is embedder misuse (RangeError), not a guest Trap. if (!Number.isInteger(length) || length < 0) { throw new RangeError( `host buffer length must be a non-negative integer, got ${length}`, @@ -192,9 +125,7 @@ export class HostBuffer { if (this.#chunks.length === 1 && this.#chunks[0] instanceof Uint8Array) { return this.#chunks[0]; } - // Multiple chunks, or a raw-layer plain-array writer: pack. Element - // coercion matches Uint8Array.from, which is what the conventions - // layer applied to these values before chunks stayed whole. + // Pack multiple chunks or a raw-layer plain-array offer into bytes. const out = new Uint8Array(this.progress); let o = 0; for (const c of this.#chunks) { @@ -214,19 +145,9 @@ export class HostBuffer { return out; } - // --- `ByteWindow` (embedder-api.md §"Streams and futures" ("Direct-access byte edges"), polyengine#128) --- - // - // A host buffer can be the PEER of a direct session on the other end of a - // host↔host rendezvous. Which of the two shapes it takes follows from the - // direction it was built for, exactly as `read`/`write` above do: - // - // * SOURCE (`values !== null`, a parked `write`): the window is a view of - // the offered chunk itself — the stream/future round-trip borrow, scoped to the callback. No - // extra copy at all. - // * DESTINATION (`values === null`, a parked/arriving `read(max)`): there - // is no landing zone to view, so the window is a fresh scratch; the - // marked prefix becomes the delivered chunk (ownership passes with it, - // and `taken()` hands a sole chunk through unsliced). + // ByteWindow for a host peer: sources expose the borrowed offered chunk; + // destinations allocate scratch whose marked prefix becomes the owned + // result. Keep that scratch stable until this callback invocation ends. /** The synthesized destination window, live for one direct invocation. */ #scratch: Uint8Array | null = null; @@ -257,12 +178,8 @@ export class HostBuffer { "host direct advance beyond remaining", ); if (this.values === null) { - // A callback may mark bytes it never actually looked at the window to - // write (nonsense, but the runtime must stay total rather than trip an - // internal assertion). The acknowledged prefix is then whatever the - // synthesized landing zone held — zeroes — which is the faithful - // analogue of the guest-peer case, where it would be whatever the - // reader's memory already contained. + // Marks without a preceding byteView acknowledge zero-filled scratch, + // just as a guest destination acknowledges its existing memory contents. const scratch = this.#scratch ?? new Uint8Array(k); // Delivered as an owned chunk; `write` is the same call the reference // copy would have made, so `remain()`/`taken()` stay consistent. @@ -278,49 +195,18 @@ export class HostBuffer { } /** - * Every live `HostActivity` arm, by identity. These are the promises this - * module parks in `store.pendingHostCalls` purely to say "the embedder may - * still act"; they are NOT outstanding work, so the host pump must not treat - * their presence as a reason to keep looping (that is the "activity keeps - * pendingHostCalls non-empty forever" hazard: a pump whose exit condition is - * `pendingHostCalls.size === 0` would never exit). - * - * The registry and the two predicates over it (`hasRealHostCall`, - * `storeQuiescent`, imported above as `quiescent`) moved to - * task/scheduler.ts so that boundary.ts's settlement pump — the OTHER - * between-calls driver — shares the same classification without an import - * cycle. Arms are minted here and marked via `markHostActivityArm`. - */ - -/** - * Keeps `store.pendingHostCalls` non-empty while a host end is live, so the - * driving loop treats "waiting for the embedder" as progress-is-possible - * rather than deadlock. Re-arms after every notification. - * - * RETENTION IS THE LIVENESS RULE (#162, contracts/embedder-api.md §"Streams and futures"). The arm - * is live iff the host retains a way to act on this shared object: a retained - * end, a parked host operation, or an unfinished producer pump. The claim it - * makes to the deadlock verdicts — "the embedder may still act" — therefore - * *expires*. Three state transitions implement it: - * - * * `close()` — terminal: DROPPED, an explicit drop, or the shared object's - * drop observers (either end, the loud component fault teardown walk). Nothing can revive - * the wrapper. - * * `disarm()` — NON-terminal: the host handed its last end back to a guest - * (a lifted stream/future lowered back in — the identity round trip). The - * object is still alive; the host merely holds nothing. - * * `rearm()` — the inverse: a re-lift handed the readable end back. - * - * The embedder-negligence policy of the module header is unchanged — an - * embedder that lowers a host-CREATED stream and never writes still hangs - * rather than traps, because it genuinely retains the writable end. + * A rearming wakeup in pendingHostCalls while a host end is retained. + * Arms signal possible external progress, not outstanding work; scheduler + * quiescence excludes them so a pump can stop without declaring deadlock. + * `disarm` relinquishes a lifted end, `rearm` restores it, and `close` is + * terminal on drop. Removing an arm also resolves it to wake existing races. */ class HostActivity { #store: Store | null = null; #promise: Promise | null = null; #resolve: (() => void) | null = null; #closed = false; - /** Retention is momentarily zero; revivable via `rearm()` (#162). */ + /** No retained end; revivable via `rearm()`. */ #disarmed = false; #pumping = false; @@ -351,30 +237,14 @@ class HostActivity { } /** - * Drive the guest until it can make no more progress. - * - * A host operation that lands *between* export calls has no driving loop - * running — `drive()` returned when the last export call resolved. So after - * initiating a host read/write (or a drop) we pump the store ourselves. - * Synchronously first (the common case: the guest is merely waiting on a - * scheduler condition our rendezvous just satisfied, and the host op's - * promise resolves before we return), then — if anything is still - * outstanding — by handing the store to the *same* loop an export call - * would have used, `driveStoreAsync`. Without the asynchronous half a guest - * parked in a background forwarding task would never be resumed to consume - * what we just offered, and the host read would await forever (host-pump - * starvation: the previous local drain only serviced `store.awaiting` and - * never awaited `store.pendingHostCalls`, so a writer parked on a - * Promise-returning host import stalled the reader). + * Drain settled activations and ready threads synchronously, then drive + * asynchronous work if the store is not quiescent. This gives host + * operations progress between export calls, including guest dependencies + * on Promise-returning host imports. * - * Traps from the synchronous half propagate to the caller of the host - * operation AND are recorded on `store.hostFailure`, the same channel - * `#pumpAsync` uses: propagation alone is not enough, because the caller is - * a host op's promise executor whose promise may already have been settled - * by the poisoning retirement walk (the trapping instance held an end of - * this very stream), in which case the throw is discarded and the fault - * would be mute. A component fault is always loud - * (contracts/embedder-api.md §"Streams and futures"). + * Record synchronous failures as well as throwing them: retirement may + * already have settled the host operation's Promise, whose executor would + * then discard the throw. The next driver can still report hostFailure. */ pump(): void { const store = this.#store; @@ -393,12 +263,7 @@ class HostActivity { throw e; } if (this.#pumping) return; - // Nothing is outstanding that only an event-loop turn could advance ⇒ no - // asynchronous pump needed. In particular an embedder that lowered a host - // end into a guest and then never did its half lands here: we return, no - // spin and no deadlock trap, and the operation's promise simply stays - // pending — the documented "hangs rather than traps" behaviour (see the - // module header). + // Retention alone is not work to drive; leave the host Promise pending. if (quiescent(store)) return; this.#pumping = true; void this.#pumpAsync(store); @@ -406,24 +271,10 @@ class HostActivity { async #pumpAsync(store: Store): Promise { try { - // This pump is the FALLBACK driver — the one for host operations that - // land BETWEEN export calls — so it stands down whenever an export - // call's loop is live: that loop already races `pendingHostCalls` and - // `store.awaiting` and so pumps host activity on our behalf. When it - // exits, we take over. `whenStoreDriverIdle` is edge-triggered, not - // polled, so waiting costs no turns. - // - // The stand-down is COOPERATIVE, not exclusion: an export call can - // start while we are parked mid-`await`, and we only notice at the next - // `done()` evaluation, so a bounded overlap window remains by - // construction (concurrent export calls have always overlapped too). - // That is safe for the resume-once invariant — `resumeWith` deletes - // from `store.awaiting` synchronously and every resumption site - // re-checks membership *and* promise identity first; see the invariant - // write-up on `storeDriverDepth` in boundary.ts. Standing down is about - // not interleaving two loops' `serviceSettled`/`tick` phases, which is - // what tripped `Trap: table entry empty` out of `runCallbackLoop` when - // this pump first drove unconditionally alongside an export call. + // Yield to an existing driver. This is cooperative: another may enter + // while we await, so done() checks depth again. Resume sites recheck + // awaiting membership and Promise identity before consuming a result + // (boundary.ts storeDriverDepth), preventing double resumption. while (!quiescent(store)) { if (storeDriverDepth(store) > 0) { await whenStoreDriverIdle(store); @@ -431,23 +282,9 @@ class HostActivity { } await driveStoreAsync( store, - // Quiescence, not completion: this pump exists to keep the guest - // moving; the host operation's own promise is what the caller - // awaits. Three exit clauses: - // - // * nothing left that a turn of the event loop could advance - // (`quiescent`); - // * `pendingHostCalls` empty, which is the precondition of BOTH - // of `driveAsync`'s deadlock traps. Returning true there keeps - // this between-calls pump from converting the documented - // embedder-never-acts hang (module header) into a trap that - // would surface, misattributed, on some later export call. - // Deadlock detection for genuine component deadlock stays where - // it belongs: in the driving loop of the export call the guest - // is blocked in; - // * another driver appeared (an export call started while we were - // parked) — hand the store back to it, per the single-driver - // rule. Our depth is 1 while we are inside, hence `> 1`. + // Stop on quiescence, before an idle deadlock verdict, or when + // another driver enters. Our own depth is 1 inside this loop. + // The caller awaits its operation, not this fallback pump. () => store.pendingHostCalls.size === 0 || quiescent(store) || @@ -463,16 +300,8 @@ class HostActivity { } finally { this.#pumping = false; } - // The pump advanced the guest OUTSIDE any export call's driving loop. A - // `driveAsync` parked on `Promise.race([...pendingHostCalls])` re-evaluates - // its `done` predicate only when something it raced settles — and - // everything the pump just did (resume the callback task, deliver the - // event, watch the guest `task.return`) may have settled nothing that race - // can see. Re-arm through `notify()` so a parked driver wakes and - // re-checks; without this the lifted call's Promise never resolves even - // though the task resolved (observed: future-user's `double-future` under - // jspi auto-detection — the guest finished, the embedder's await hung - // forever). + // Guest progress may have settled a task without settling anything in + // another driver's pendingHostCalls race. Wake it to recheck done(). this.notify(); } @@ -489,15 +318,8 @@ class HostActivity { } /** - * The host retains no way to act: its lifted end was lowered back into a - * guest, which now owns it (#162, §"Streams and futures"). NON-terminal — a re-lift - * of the same shared object restores retention via `rearm()`. - * - * Resolving the stale arm is required, not tidiness: a `driveAsync` parked - * on `Promise.race([...pendingHostCalls])` re-evaluates its `done` predicate - * and its deadlock preconditions only when something it raced settles. An - * arm merely deleted from the set would leave that driver asleep on a - * promise nobody will ever settle. + * Relinquish the host's lifted end and wake drivers racing its old arm. + * Nonterminal: a re-lift of the same shared object restores retention. */ disarm(): void { const p = this.#promise, r = this.#resolve; @@ -511,10 +333,7 @@ class HostActivity { } /** - * A lift handed the host the readable end again — the stream/future round-trip cache-hit wrapper - * for a shared object that round-tripped back out of the guest (#162). - * A no-op for a closed activity (the object is gone for good) and for one - * that was never disarmed. + * Restore retention on re-lift; a closed activity cannot be revived. */ rearm(): void { if (this.#closed) return; @@ -527,8 +346,7 @@ class HostActivity { // Direct-access byte edges (embedder-api.md §"Streams and futures" ("Direct-access byte edges") (polyengine#128)) // --------------------------------------------------------------------------- // -// wasmtime `DirectSource`/`DirectDestination`-shaped (`component::concurrent`, -// 47.0.3). For `stream` only, a host end may park a *direct session* +// For `stream` only, a host end may park a direct session // instead of a chunk: at every rendezvous with a peer operation of nonzero // capacity the session's callback runs exactly once, synchronously, inside the // rendezvous, against a scoped view of the peer's bytes — so an external @@ -574,10 +392,9 @@ export type DirectVerdict = "more" | "done"; * because the callback itself returned `"done"`, rather than because the peer * dropped / the operation was cancelled / the peer's instance trapped. * - * The conventions layer needs the distinction for loud component fault precision — a session - * the producer already completed keeps its resolution even if the peer then - * trapped — and `Promise` is the contract's return shape, so it rides - * here rather than in the resolved value. + * The conventions layer preserves a callback-completed result across a later + * peer trap. Otherwise recorded peer poisoning rejects with the progress + * count; ordinary drop/cancel resolve the count. */ export interface DirectSessionInfo { endedByVerdict: boolean; @@ -669,7 +486,7 @@ class DirectSession implements DirectBuffer { total = 0; /** The callback said `"done"`, or the session failed / was settled. */ ended = false; - /** `ended` because the callback said so (loud component fault precision; see `DirectSessionInfo`). */ + /** `ended` because the callback said so; see `DirectSessionInfo`. */ endedByVerdict = false; /** Installed in the shared object's pending slot right now. */ pending = false; @@ -728,8 +545,8 @@ class DirectSession implements DirectBuffer { try { verdict = this.invoke(scope); } catch (e) { - // "A callback that throws rejects the session with that error, and the - // invocation's marks are discarded" — so nothing touches `peer`. + // Discard marks on throw. Progress is unchanged; writes the callback + // already made through the byte view are not rolled back. scope.die(); this.#fail(e); return "failed"; @@ -749,8 +566,7 @@ class DirectSession implements DirectBuffer { const k = scope.marked; if (k === 0) { if (verdict === "done") { - // Retraction: the speculative-park correction. The session ends with - // its running total and the peer's operation stays parked. + // Retract without completing the peer's parked operation. this.ended = true; this.endedByVerdict = true; return "retracted"; @@ -858,8 +674,8 @@ export interface HostWritableEnd { * of the offer unsent. When the host arrives *first* it stays parked and is * drained across several guest reads. `writeAll` papers over the difference. * - * Resolves with the total accepted, which is less than `values.length` only - * if the reader dropped. + * Resolves with the total accepted; cancellation or reader drop can leave + * a short count. The writable end stays reserved between offers. */ writeAll(values: T[], info?: { progress: number }): Promise; /** @@ -884,11 +700,9 @@ export interface HostWritableEnd { info?: DirectSessionInfo, ): Promise; /** - * Cancel an in-flight `write`/`writeAll` (definitions.py - * `SharedStreamImpl.cancel` -> `CopyResult.CANCELLED`). No-op when nothing - * of ours is parked. Surfaced per the R-fix review's stream advisory 1: the - * cancel channel existed on the shared object but had no embedder-facing - * spelling, so a host writer could only be abandoned, never retracted. + * Cancel a write or direct session, resolving with progress so far. + * Cancels the whole `writeAll` helper, including gaps between offers. + * Does not cancel a peer's operation or drop the stream. */ cancelWrite(): void; /** definitions.py `SharedStreamImpl.drop`: notifies a parked reader. */ @@ -943,24 +757,10 @@ export interface HostStream { /** * Attach host-activity bookkeeping to a shared object at the CABI seam. * - * `kind` is the retention model (#162, §"Streams and futures") — WHICH ends the host - * holds, which is decided entirely by where the wrapper came from: - * - * * `"created"` — `hostStream()`/`hostFuture()`. Only READABLE ends - * transfer across the boundary (definitions.py `lower_stream`, line 1828, - * wraps the shared object in a fresh `ReadableStreamEnd` in the callee's - * table), so lowering hands the guest the readable end and the host keeps - * the WRITABLE one. Retention survives every lower; the arm ends only at - * drop/end-of-pump. - * * `"lifted"` — `hostStreamFor()`/`hostFutureFor()`. The host holds exactly - * the readable end the guest passed out (`lift_async_value`, line 1530). - * Lowering that same object back into a guest transfers it away, so - * retention hits zero and the activity disarms; a later re-lift restores - * it through the `onLifted` hook. - * - * The hooks live here rather than in the conventions layer's `takeValue` so - * that BOTH the conventions layer and the raw boundary are covered, with no - * window between "the embedder said transfer" and "the transfer happened". + * Created wrappers retain their writable end when the readable end lowers. + * Lifted wrappers hold only the readable end, so lower disarms and re-lift + * rearms them. Hooks at the CABI seam cover raw and conventions callers at + * the actual transfer, not merely when a facade value is prepared. */ function bindOnLower( shared: SharedStreamImpl | SharedFutureImpl, @@ -972,14 +772,8 @@ function bindOnLower( onLowered?: ((i: ComponentInstanceState) => void) | null; onLifted?: ((i: ComponentInstanceState) => void) | null; }; - // INTERNAL INVARIANT (not the embedder-facing policy): two live wrappers - // on one shared object would mean two HostActivities pumping it, and the - // second `onLowered` hook would silently orphan the first wrapper's - // activity binding for future lowers (review advisory, host-streams - // round). The public entry points cannot get here with a wrapped object — - // `hostStreamFor`/`hostFutureFor` return the cached wrapper instead - // (§"Streams and futures") — so a trip here is a bug in this module. The class field - // initializes to null; == null covers both sentinels. + // One low-level wrapper per shared object: replacing hooks would orphan + // the original activity. hostStreamFor/hostFutureFor enforce this by cache. assert_( holder.onLowered == null, "internal: a second host wrapper was built for an already-wrapped " + @@ -991,7 +785,7 @@ function bindOnLower( "already-wrapped stream/future (the wrapper cache should have " + "returned the first)", ); - // `lowerStream`/`lowerFuture` (cabi/async_values.ts :177/:204) fire this on + // `lowerStream`/`lowerFuture` fire this on // EVERY lower, not just the first — the hook persists, and the asserts // above only forbid installing a SECOND one. holder.onLowered = (inst) => { @@ -1005,23 +799,13 @@ function bindOnLower( activity.bind(inst.store); } }; - // Fired by `liftAsyncValue` (cabi/async_values.ts :126) whenever this + // Fired by `liftAsyncValue` whenever this // object is lifted out of a guest table. For a "created"-kind wrapper // `rearm()` is a harmless no-op (it is never disarmed), so the hook is // installed uniformly. holder.onLifted = () => activity.rearm(); - // Release the arm when the shared object dies, whatever kills it. This is - // the single point that covers three otherwise-separate leaks of one class: - // the `dropForTeardown` asymmetry (embedder/streams.ts — a teardown with - // nothing parked never reached `close()`), a guest dropping its end with no - // host operation parked (the `settle(DROPPED)` -> `close()` path only runs - // for a parked op), and `HostFuture.readResult`'s already-dropped fast path - // (which answers synchronously without touching the activity). - // - // Note on the guest-to-guest composed hop: a value lifted from the caller - // and immediately lowered into the callee, both synchronously inside one - // call's lower phase, fires rearm-then-disarm on any host wrapper that - // happens to exist for it. The pair nets out to the correct final state. + // Release even with no operation parked, including poisoned-end teardown. + // A composed guest-to-guest hop rearms then disarms synchronously. shared.whenDropped(() => activity.close()); // A stream that came *out* of a guest was lifted, never lowered, so the // `onLowered` hook above will not fire first; `boundStore` was recorded at @@ -1043,25 +827,15 @@ function mkStreamEnds( const parked = { read: false, write: false }; let writeAll: "active" | "cancelled" | null = null; /** - * Settle bookkeeping for a completed copy. `DROPPED` means the peer end is - * gone: no further host activity on this end is possible, so the activity - * arm is *closed* rather than re-armed (R-fix review advisory 2 — a live arm - * after end-of-stream keeps `pendingHostCalls` non-empty forever and masks - * a genuine deadlock as "the embedder might still act"). + * Drop ends retention; other outcomes wake the driver and rearm activity. */ const settle = (result: CopyResult): void => { if (result === CopyResult.DROPPED) activity.close(); else activity.notify(); }; /** - * Withdraw an operation that never got to finish: a trap out of - * `activity.pump()`'s synchronous half unwinds through the op's promise - * executor with our bookkeeping half-done — the `parked` flag set and our - * buffer still in the shared object's pending slot, which wedges the end - * ("a write is already in flight") for good. Same withdrawal - * `cancelWrite`/`cancelRead` perform; `shared.cancel()` only while the - * pending side is still literally ours, since `SharedBase.cancel` asserts - * that something is pending and the poisoning walk may have retired it. + * Withdraw after a pump failure. Cancel only if the pending buffer is still + * ours: poisoning may already have retired it or notified a peer. */ const withdraw = (side: "read" | "write", buf: unknown): void => { if (!parked[side]) return; @@ -1075,21 +849,10 @@ function mkStreamEnds( write: null, }; /** - * Drive one direct session from park to end. - * - * Two shapes reach us, and the difference is *which side arrived second*: - * - * * the session is the PENDING side — every rendezvous fires `onCopy`, and - * the `"more"` verdict simply declines to `reclaim()`, so the session - * stays in the pending slot for the next peer operation. This is - * `write()`'s "stay parked until the offer is exhausted" mechanism, with - * the callback's verdict in place of `buf.remain() > 0`. - * * the session ARRIVED second — the rendezvous completes it with - * `onCopyDone(COMPLETED)`, so a `"more"` verdict has to re-issue. The - * re-issue rides the loop below (one `await` apart), which is exactly - * `writeAll`'s re-offer shape and therefore inherits its ordering: the - * peer's pending event is delivered and its buffer reclaimed before we - * can rendezvous against it a second time. + * A pending session stays parked on "more" by declining to reclaim. + * An arriving session completes that issuance and must reissue after an + * await. The pump services the peer's event/reclamation before the next + * issuance. Reserve the end for the entire session, including these gaps. */ const runDirectSession = async ( side: "read" | "write", @@ -1160,11 +923,8 @@ function mkStreamEnds( }; /** Shared tail of `cancelWrite`/`cancelRead` for a parked direct session. */ const cancelDirect = (session: DirectSession): void => { - // direct-access byte edge: cancelling RETRACTS the session — it resolves with its running - // total (future abandonment's indistinguishability caveats unchanged). `shared.cancel()` - // only when the session actually holds the pending slot: a session caught - // between two issuances holds nothing, and `SharedBase.cancel` asserts - // that something is pending. + // Retraction resolves with the running total. Between issuances there is + // no pending buffer to cancel; retractDirect handles both states. retractDirect(session); activity.notify(); activity.pump(); @@ -1208,14 +968,8 @@ function mkStreamEnds( return { writable: { write(values: T[], info?: { progress: number }): Promise { - // One in-flight operation per end — the host-side spelling of the - // `CopyEnd` busy trap guests get from the table. Without it a second - // write would find the FIRST write's buffer in the shared object's - // pending slot and "rendezvous" write-against-write, silently - // copying into the parked buffer's accumulation (observed as a - // write resolving `1` against a peer that no longer exists — the - // #66 repro). Reading while a write is parked stays legal: that is - // the pass-through data plane (two different ends). + // One operation per direction prevents write-against-write rendezvous. + // A simultaneous read is legal and serves host-to-host round trips. if (parked.write || writeAll !== null) { throw new TypeError( "a write is already in flight on this stream's writable end; " + @@ -1240,10 +994,8 @@ function mkStreamEnds( while ( sent < values.length && !shared.dropped && writeAll === "active" ) { - // Re-offers keep `write`'s borrow semantics: the first round is the - // chunk itself and later rounds a `subarray` VIEW for typed chunks - // (review F1: a `slice` here cost a second full copy on the very - // path the one-copy contract names), a `slice` for plain arrays. + // Re-offer typed chunks by view, preserving the original borrow + // until the whole helper settles without an extra byte copy. const rest = sent === 0 ? values : values instanceof Uint8Array @@ -1357,16 +1109,8 @@ function mkStreamEnds( }); }, cancelRead() { - // #97, DELIBERATE AND PINNED: cancelling resolves the in-flight - // `read` promise with whatever the buffer took so far — for a read - // that had not yet rendezvoused, the empty chunk. An empty chunk is - // also this layer's end-of-stream signal (see `HostReadableEnd.read` - // and embedder/streams.ts `Stream.read`), so **a host-cancelled read - // is indistinguishable from EOS at the conventions layer**. That is - // accepted rather than papered over: the code that calls - // `cancelRead()` is the same code that observes the result, so it - // already knows which of the two happened. Nothing else can reach - // this state — a guest cannot cancel the host's read. + // Resolve with progress so far. An empty cancelled chunk is + // indistinguishable from EOS; the caller knows it requested cancel. if (!parked.read) return; const session = direct.read; if (session !== null) return cancelDirect(session); @@ -1385,12 +1129,9 @@ function mkStreamEnds( } /** - * One host wrapper per shared object, by identity (contracts/embedder-api.md - * §"Streams and futures"). A stream/future value that round-trips host → guest → host lifts back - * as the SAME wrapper the host already holds, so wrapping is idempotent — - * there is never a second `HostActivity` competing to pump one shared object - * (the hazard the old double-wrap assert guarded against), and the readable - * end stays transferable across as many boundary hops as the spec allows. + * One low-level wrapper/activity per shared object across round trips. + * Facade Stream/Future handles may be fresh; shared rendezvous identity and + * these per-direction operation guards remain the same. */ const streamWrappers = new WeakMap>(); const futureWrappers = new WeakMap>(); @@ -1437,21 +1178,19 @@ export interface HostFuture { * value, so `read`'s `undefined` is ambiguous between "the value was * `undefined`" (a `future`) and "the write end dropped without ever * writing" — the case the conventions layer must turn into a - * `DroppedError` (R-fix review advisory 4). `result` disambiguates: + * `DroppedError`. `result` disambiguates: * `COMPLETED` iff `value` is real. */ readResult(): Promise<{ value: T | undefined; result: CopyResult }>; /** Cancel an in-flight `read`/`write`; see `HostWritableEnd.cancelWrite`. */ cancel(): void; /** - * Release this future. Total and idempotent (#90): it never throws, and a - * second call is a no-op. - * - * Three cases, per the #90 ruling: + * Release this future. Shared-state drop is idempotent; pumping the store + * can still surface a guest failure. * * * the value was already delivered (the normal write-then-drop path) — * plain state cleanup, the spec's `WritableFutureEnd.drop` precondition - * (definitions.py:1183-1184) is satisfied; + * is satisfied; * * never written, and the future was **lowered** into a guest (the guest * holds the readable end, so this wrapper plays the spec's writable * role) — *abandon*: the reader can never be satisfied, so it is armed @@ -1515,7 +1254,7 @@ function mkFuture( const writeInst = hostEndInstance("write"); const readInst = hostEndInstance("read"); const parked = { read: false, write: false }; - /** Set once the future's one value has actually crossed (#90). */ + /** Set once the future's one value has actually crossed. */ let delivered = false; const settle = (side: "read" | "write", result: CopyResult): void => { parked[side] = false; @@ -1523,7 +1262,7 @@ function mkFuture( if (result === CopyResult.DROPPED) activity.close(); else activity.notify(); }; - /** See `mkStreamEnds`' `withdraw`: the pump-trap unwind path (F1). */ + /** See `mkStreamEnds`' `withdraw`: the pump-trap unwind path. */ const withdraw = (side: "read" | "write", buf: unknown): void => { if (!parked[side]) return; parked[side] = false; @@ -1532,7 +1271,8 @@ function mkFuture( }; const self: HostFuture = { write(v: T): Promise { - // Opposite ends may rendezvous after a guest round trip. + // One operation per direction; opposite ends may rendezvous after a + // guest round trip. This is a busy guard, not a delivered-value guard. if (parked.write) { throw new TypeError( "an operation is already in flight on this future; " + @@ -1603,9 +1343,8 @@ function mkFuture( activity.pump(); }, drop() { - // #90. Never throws, idempotent: `SharedFutureImpl.drop` and - // `abandonSharedFuture` both no-op on an already-dropped future, and - // neither can raise. See the `HostFuture.drop` doc for the three cases. + // A lowered future still owing a value is abandoned; other drops are + // plain cleanup. Pump failures propagate even if shared state is gone. if (!delivered && lowering.lowered && !shared.dropped) { abandonSharedFuture( shared, diff --git a/runtime/src/intrinsics/async_builtins.ts b/runtime/src/intrinsics/async_builtins.ts index bd77597..d2d280d 100644 --- a/runtime/src/intrinsics/async_builtins.ts +++ b/runtime/src/intrinsics/async_builtins.ts @@ -1,46 +1,20 @@ // The 0.3 async canonical built-ins, as host trampolines // (contracts/intrinsics.md §B): task.{return,cancel}, -// backpressure.{set,inc,dec}, waitable-set.{new,wait,poll,drop}, +// backpressure.{inc,dec}, waitable-set.{new,wait,poll,drop}, // waitable.join, subtask.{drop,cancel} and thread.yield. // -// Every one is a direct port of the correspondingly named `canon_*` function -// in definitions.py (cited per function), with one systematic substitution. +// Semantics follow the corresponding `canon_*` functions in definitions.py, +// with JSPI timing differences documented at the waits below. // -// ## `current_instance()` vs the trampoline's declared instance +// Instance-scoped built-ins use the trampoline's declared instance. It is +// available during core start functions before a task exists, and identifies +// the right handle table even when a FACT adapter runs under another +// instance's task. Operations needing a task/thread still read the ambient. // -// definitions.py reads `current_instance()` (line 312), defined as -// `current_task().inst` — it can, because in the reference a canonical -// built-in is only ever reached from inside a task. That is not true of a -// real component: wasmtime lets a core module's **start function** call -// instance-scoped built-ins (`waitable-set.new`, `backpressure.inc`, ...) -// during instantiation, before any task exists. The official suite exercises -// exactly this (e.g. `test/async/dont-block-start.wast`). -// -// wasmtime resolves it by naming the owning component instance *statically* -// in every trampoline declaration (`Trampoline::WaitableSetNew { instance }` -// and friends — the `instance` field the plan carries). So the built-ins -// below take their instance from the declaration, which is well-defined at -// instantiation time and identical to `current_instance()` whenever a task is -// running. Built-ins that genuinely need the *task* or *thread* -// (`task.return`, `task.cancel`, `subtask.cancel`, `thread.yield`) still read -// the current-thread stack: they are meaningless outside a task, and the -// reference's `trap_if`s are what report that. -// -// ## Blocking built-ins in a stackless world -// -// `waitable-set.wait`, `thread.yield` and the synchronous `subtask.cancel` -// all *block the calling wasm frame* in the reference. A callback-ABI guest -// is stackless: there is no suspendable wasm stack to park, so blocking here -// genuinely requires JSPI (docs/architecture.md §6, JSPI role 2) and these built-ins say -// so at the precise point, loudly, instead of faking a wait. -// -// They are not, however, unconditionally unavailable. Where the reference can -// complete *without* suspending — `waitable-set.wait` on a set that already -// has a pending event, `waitable-set.poll` always, `subtask.cancel` on a -// subtask that resolved eagerly — this module returns the answer directly. -// That is not a shortcut: definitions.py's `Thread.wait_until` (line 396) may -// legitimately return without blocking when `ready_func()` already holds, so -// taking that branch is a conforming schedule. +// Blocking a wasm frame requires JSPI, including from callback-ABI code. +// Returning WAIT/YIELD callback codes is the stackless alternative. A built-in +// may complete immediately when its condition already holds, as permitted by +// `Thread.wait_until`; otherwise plain mode reports `NeedsJspi`. import { blockCurrentActivation } from "../jspi/mod.ts"; import type { SuspensionMode } from "../jspi/mod.ts"; @@ -85,7 +59,7 @@ export interface AsyncTrampolineContext { } /** - * `BLOCKED` sentinel of `canon_subtask_cancel` (definitions.py line 2467). + * `BLOCKED` sentinel of definitions.py `canon_subtask_cancel`. */ export const BLOCKED = 0xffff_ffff; @@ -93,17 +67,16 @@ export const BLOCKED = 0xffff_ffff; // task.return / task.cancel // --------------------------------------------------------------------------- -/** definitions.py `canon_task_return` (line 2384). */ +/** definitions.py `canon_task_return`. */ export function createTaskReturn( decl: { results: number; resultType: number | null; options: number }, ctx: AsyncTrampolineContext, ): CoreFn { const opts = ctx.options(decl.options); - // plan v3: `resultType` is the interned `plan.types` entry; `results` is the + // `resultType` is the interned `plan.types` entry; `results` is the // raw wasmtime `TypeTupleIndex` (the FACT `task_return_type` key, consumed // by the loader's dictionary). `null` is wire-legal for a task with no - // declared result type; today's producer always emits the empty tuple - // instead, so this degenerates to `[]` either way. + // declared result type and means an empty results list here. const resultTypes = decl.resultType === null ? [] : ctx.resultTypes(decl.resultType); @@ -114,54 +87,24 @@ export function createTaskReturn( "task.return: cannot leave component instance (may_leave violation)", ); trapIf(!task.opts.async_, "task.return from a non-async task"); - // `trap_if(result_type != task.ft.result)` (definitions.py:2388): the + // `canon_task_return`'s `trap_if(result_type != task.ft.result)`: the // trampoline's interned result tuple must be the lifted function's result // type. Compared structurally — the plan's type table interns by // structure, so identity comparison would reject valid components. // - // Plan v3 enables this for FACT cross-component tasks too: the callee - // task's declared result type is now resolvable from the raw - // `TypeTupleIndex` `prepare-call` carried (the task-return trampoline's - // raw `results` key + interned `resultType`, contracts/plan-format.md - // schema; wired in fact_calls.ts). It remains skipped for the one - // case v3 does not answer — a callee the plan maps no `task.return` - // tuple for, where `ft.results` is a placeholder rather than a - // declaration (`factResultTypesKnown === false`); comparing against a - // placeholder would be a false rejection, not a check. + // FACT tasks with no mapping for their raw TypeTupleIndex carry placeholder + // results, not a declaration; only those skip this comparison. trapIf( (!task.factPassthrough || task.factResultTypesKnown) && !valTypesEqual(resultTypes, task.ft.results), "task.return with a result type that is not the task's result type", ); - // `trap_if(not LiftOptions.equal(opts, task.opts))` (definitions.py:2389). - // The MEMORY half stays skipped for FACT tasks, and plan v3 does NOT - // change that: the relaxation was never about the type mapping. The - // task's memory is reconstructed from `prepare-call`'s `memory` field, - // which is the *adapter's* view of the lift options - // (`adapter.lift.options...memory`) and is `None` for callees whose own - // `task.return` options do name a memory — the 17-param async-lifted - // callees of `test/async/cross-abi-calls.wast` are exactly that shape. - // The information simply is not in the plan, at v3 as at v2; restoring - // the check needs `prepare-call`'s indices related to the callee's - // canonical options, which remains open contract friction. - // - // definitions.py `LiftOptions.equal` (line 643) compares string encoding - // *and* memory identity. Both halves are checked for a host-boundary task. - // - // For a FACT task the memory half is skipped, and the reason is specific - // rather than "we can't be bothered": the task's memory is reconstructed - // from `prepare-call`'s `memory` field, which carries the *adapter's* view - // of the lift options (`adapter.lift.options...memory`) and is `None` - // for callees whose own `task.return` options do name a memory — - // `test/async/cross-abi-calls.wast`'s 17-param async-lifted callees are - // exactly that shape. wasmtime tolerates the mismatch because its check is - // *one-sided*: `concurrent.rs:3344-3358` treats "the `task.return` site - // specifies no memory" as valid and only compares when it does, against a - // lift memory it holds first-hand. We hold ours second-hand, so applying - // either form of the memory comparison produces a false rejection. - // - // The string-encoding half IS checked on both paths: `prepare-call` passes - // the encoding directly, so that reconstruction is exact. + // `LiftOptions.equal` compares string encoding and memory identity. + // Both are checked for host-boundary tasks. FACT tasks skip memory + // identity: prepare-call carries the adapter's memory, which can be null + // even when the callee's task.return names one (cross-abi-calls.wast's + // 17-param async lifts). Restoring this check needs a reliable mapping + // to the callee's lift memory. Encoding is passed directly and checked. trapIf( !liftOptionsEqual( { stringEncoding: opts.stringEncoding, memory: opts.memory }, @@ -171,10 +114,7 @@ export function createTaskReturn( ), "task.return with canonical options differing from the task's", ); - // Type-aware per-lane normalization: `normalizeFlat`'s blanket `>>> 0` - // silently truncated float lanes (a `task.return` of f64 -1.1 arrived at - // `[async-return]` as 4294967295). `normalizeCoreValues` consults the - // declared lane types, so only i32 lanes are coerced. + // Normalize by declared lane type; integer coercion must not touch floats. const flat = normalizeCoreValues( flatArgs, opts.coreType.params, @@ -194,7 +134,7 @@ export function createTaskReturn( }; } -/** definitions.py `canon_task_cancel` (line 2397). */ +/** definitions.py `canon_task_cancel`. */ export function createTaskCancel(): CoreFn { return () => { const task = currentTask() as Task; @@ -211,14 +151,7 @@ export function createTaskCancel(): CoreFn { // backpressure // --------------------------------------------------------------------------- -// `canon_backpressure_set` is not ported: wasmtime 47 emits no -// `BackpressureSet` trampoline (`component/info.rs` has only -// `BackpressureInc`/`BackpressureDec`), and the reference's own copy was -// unreachable dead code until upstream removed it (CM PR #690; see -// upstream-component-model-repo-findings.md CM-2, RESOLVED). The counter -// below is the live interface. - -/** definitions.py `canon_backpressure_inc` (line 2368). */ +/** definitions.py `canon_backpressure_inc`. */ export function createBackpressureInc(inst: ComponentInstanceState): CoreFn { return () => { assert_( @@ -230,7 +163,7 @@ export function createBackpressureInc(inst: ComponentInstanceState): CoreFn { }; } -/** definitions.py `canon_backpressure_dec` (line 2375). */ +/** definitions.py `canon_backpressure_dec`. */ export function createBackpressureDec(inst: ComponentInstanceState): CoreFn { return () => { assert_( @@ -246,7 +179,7 @@ export function createBackpressureDec(inst: ComponentInstanceState): CoreFn { // waitable sets // --------------------------------------------------------------------------- -/** definitions.py `canon_waitable_set_new` (line 2406). */ +/** definitions.py `canon_waitable_set_new`. */ export function createWaitableSetNew(inst: ComponentInstanceState): CoreFn { return () => { trapIf( @@ -258,16 +191,9 @@ export function createWaitableSetNew(inst: ComponentInstanceState): CoreFn { } /** - * definitions.py `canon_waitable_set_wait` (line 2414). - * - * The reference blocks the calling thread until the set has an event. From a - * stackless (callback-ABI) guest there is no wasm stack to suspend, so this - * only succeeds when an event is *already* pending — which is the reference's - * own non-blocking branch of `Thread.wait_until`. Otherwise: `needsJspi`. - * - * A guest using the callback ABI is expected to return the `WAIT` callback - * code rather than call this built-in; hitting the JSPI path here means the - * component uses the stackful async ABI. + * definitions.py `canon_waitable_set_wait`. Returns a pending event directly, + * or suspends the calling wasm frame using JSPI until an event or cancellable + * task cancellation arrives. Plain mode cannot perform the blocking case. */ export function createWaitableSetWait( decl: { options: number }, @@ -276,16 +202,11 @@ export function createWaitableSetWait( mode: SuspensionMode = "plain", ): CoreFn { const opts = ctx.options(decl.options); - // `cancellable` is a *canonical option*, not a trampoline field: wasmtime's - // `Trampoline::WaitableSetWait` carries only `{instance, options}` - // (wasmtime-environ 47.0.3 `component/info.rs:815`), while - // `CanonicalOptions.cancellable` (info.rs:540) is what the guest declared. - // It reaches definitions.py as `canon_waitable_set_wait`'s first parameter - // (line 2414). + // `cancellable` is a canonical option, not a trampoline field. const cancellable = opts.cancellable; return (si?: number, ptr?: number) => { // Guest-supplied index/pointer are u32; core wasm delivers i32 args - // signed (F3, R2). Normalize at the entry boundary. + // signed. Normalize at the entry boundary. si = (si ?? 0) >>> 0; ptr = (ptr ?? 0) >>> 0; trapIf( @@ -298,36 +219,16 @@ export function createWaitableSetWait( if (task.deliverPendingCancel(cancellable)) { event = [EventCode.TASK_CANCELLED, 0, 0]; } else if (wset.hasPendingEvent()) { - // Non-blocking branch: definitions.py `Thread.wait_until` may return - // immediately when the condition already holds. - // - // This deliberately skips `wait_for_event_and`, and with it the - // `num_waiting += 1 / -= 1` bracket around the block - // (`WaitableSet.wait_for_event_and`, line 829). That is unobservable: - // `num_waiting` is read only by `WaitableSet.drop` - // (`trap_if(self.num_waiting > 0)`, line 852), and since we never yield - // between the increment and the decrement here, no other code could run - // to observe a non-zero value. Incrementing and immediately decrementing - // would be pure ceremony. + // No waiter count is needed for immediate delivery: no other thread can + // observe the reference's increment/decrement bracket without a yield. traceCopy(`waitable-set.wait si=${si} FAST (pending event)`); event = wset.getPendingEvent(); } else if (mode === "jspi") { traceCopy(`waitable-set.wait si=${si} BLOCKS`); - // SITE 2 (lit). definitions.py `WaitableSet.wait_for_event_and` - // (line 829): block until the set has an event, then take it. - // - // The `num_waiting` bracket is real now. Skipping it was justified only - // while this path could not actually yield; a genuine block CAN be - // observed, because `WaitableSet.drop` traps on `num_waiting > 0` - // (line 852). Incremented before blocking and decremented in - // `onSettled`, which runs exactly once on EVERY terminal transition — - // normal resume, cancelled resume, produce-throw, and `abandon` - // (#106: decrementing in `produce` missed the abandon leg, leaving - // `numWaiting` elevated forever and a later `waitable-set.drop` - // trapping spuriously). The decrement is not idempotent, so it lives - // ONLY here, not in `produce` as well; nothing can observe the still- - // elevated count between `produce` and the hook — both run - // synchronously inside the settle, before any other code. + // `WaitableSet.drop` must see this blocked waiter. Decrement exactly + // once in onSettled, including abandonment and produce-time traps. + // produce and the hook run synchronously, so no thread observes a + // completed wait with its count still elevated. wset.numWaiting += 1; return blockCurrentActivation({ store: inst.store, @@ -355,7 +256,7 @@ export function createWaitableSetWait( }; } -/** definitions.py `canon_waitable_set_poll` (line 2431). */ +/** definitions.py `canon_waitable_set_poll`. */ export function createWaitableSetPoll( decl: { options: number }, ctx: AsyncTrampolineContext, @@ -377,10 +278,10 @@ export function createWaitableSetPoll( }; } -/** definitions.py `canon_waitable_set_drop` (line 2441). */ +/** definitions.py `canon_waitable_set_drop`. */ export function createWaitableSetDrop(inst: ComponentInstanceState): CoreFn { return (i?: number) => { - // Guest-supplied index is u32; core wasm delivers i32 args signed (F3, R2). + // Guest-supplied index is u32; core wasm delivers i32 args signed. i = (i ?? 0) >>> 0; trapIf( !inst.mayLeave, @@ -396,7 +297,7 @@ export function createWaitableSetDrop(inst: ComponentInstanceState): CoreFn { }; } -/** definitions.py `canon_waitable_join` (line 2451). */ +/** definitions.py `canon_waitable_join`. */ export function createWaitableJoin(inst: ComponentInstanceState): CoreFn { return (wi?: number, si?: number) => { wi = (wi ?? 0) >>> 0; @@ -406,10 +307,7 @@ export function createWaitableJoin(inst: ComponentInstanceState): CoreFn { trapIf(!(w instanceof Waitable), "waitable.join: handle is not a waitable"); trapIf( (w as Waitable).hasSyncWaiter, - // Wording per the suite's assertions - // (test/async/trap-if-sync-and-waitable-set.wast:301-307, - // test/async/reentrance.wast:837): a waitable claimed synchronously and - // a waitable in a set are the two halves of one rule, spelled the same. + // A synchronous claim and waitable-set membership are mutually exclusive. "waitable cannot be used synchronously while added to a waitable set " + "(waitable.join)", ); @@ -426,7 +324,7 @@ export function createWaitableJoin(inst: ComponentInstanceState): CoreFn { // subtasks // --------------------------------------------------------------------------- -/** definitions.py `canon_subtask_drop` (line 2494). */ +/** definitions.py `canon_subtask_drop`. */ export function createSubtaskDrop(inst: ComponentInstanceState): CoreFn { return (i?: number) => { i = (i ?? 0) >>> 0; @@ -438,14 +336,6 @@ export function createSubtaskDrop(inst: ComponentInstanceState): CoreFn { }; } -/** - * definitions.py `canon_subtask_cancel` (line 2469). - * - * The synchronous form blocks (`thread.wait_until(subtask.resolved)`) when the - * callee does not resolve promptly; from a stackless guest that is JSPI - * territory. The async form returns `BLOCKED` instead of blocking, and is - * fully supported. - */ /** * The tail shared by `subtask.cancel`'s blocking and non-blocking exits: * take the delivered SUBTASK event, check it is the one we expect, and report @@ -470,6 +360,11 @@ function finishSubtaskCancel( }; } +/** + * definitions.py `canon_subtask_cancel`: sync waits for resolution; async + * reports BLOCKED if unresolved. JSPI also waits for callee determinacy, + * making the async form non-atomic (docs/architecture.md §6, issue #92). + */ export function createSubtaskCancel( decl: { async?: boolean }, inst: ComponentInstanceState, @@ -478,19 +373,8 @@ export function createSubtaskCancel( const async_ = decl.async === true; return (i?: number) => { i = (i ?? 0) >>> 0; - // The handle table is the **declared** instance's, not - // `current_thread().task.inst`. definitions.py `canon_subtask_cancel` - // (line 2469) uses the latter because the reference has no fused - // adapters, so the running task and the subtask's owner always coincide. - // With FACT they do not: `async-start-call` adds the subtask to - // `prepare-call`'s `caller_instance`, which for a nested component is a - // *different* instance from the one whose task is running — observed as - // caller=2 vs task.inst=3 in `big-interleaving-test.wast:1584`, where the - // lookup then failed with "table index out of range". wasmtime names the - // owner on the trampoline for exactly this reason - // (`Trampoline::SubtaskCancel { instance, .. }`), which is the same - // correction already applied to every other instance-scoped built-in — - // see this module's header. + // FACT adds the handle to prepare-call's caller instance, which need not + // be the ambient task's instance. Use the declared table owner. trapIf(!inst.mayLeave, "subtask.cancel: cannot leave component instance"); const subtask = inst.handles.get(i); trapIf( @@ -510,10 +394,7 @@ export function createSubtaskCancel( // definitions.py `canon_subtask_cancel`: `trap_if(subtask.in_waitable_set())` // is unconditional — BOTH forms trap, because either form may claim the // subtask synchronously (`has_sync_waiter`, below) and a subtask in a set - // is not the claimer's to take. Corroborated by - // test/async/trap-if-sync-and-waitable-set.wast:325-327, which asserts the - // trap for `subtask-cancel-sync-when-in-set` AND - // `subtask-cancel-async-when-in-set`, with the wording used here. + // is not the claimer's to take. trapIf( st.inWaitableSet(), "waitable cannot be used synchronously while added to a waitable set " + @@ -536,47 +417,22 @@ export function createSubtaskCancel( // `on_cancel()` can run the cancelled callee synchronously, and that // callee may reenter this instance: the reentrant frame must see the // subtask as claimed and trap in `canon_waitable_join` - // (`trap_if(w.has_sync_waiter)`) — test/async/reentrance.wast:837. - // `parked` hands the clear over to the park's `produce`/`onSettled` - // (the async form clears it at park entry instead — #295, below) - // (#106: `abandon` never runs `produce`, so the flag needs the - // `onSettled` backstop or it stays set forever and a later - // `waitable.join` traps spuriously). + // (`trap_if(w.has_sync_waiter)`). A sync park keeps the claim until + // produce/onSettled; an async determinacy park ends it at park entry. st.hasSyncWaiter = true; let parked = false; try { st.onCancel!(inst); - // Is the callee's state safe to READ yet? Under jspi it may not be. - // `request_cancellation` delivers TASK_CANCELLED by settling the - // callee's suspension, and the engine runs the resumed activation on - // a LATER microtask (pin (j)); the reference has no such hop — - // `Task.request_cancellation` -> `Thread.resume` runs the resumed - // thread to its next block point or exit synchronously, so when - // `on_cancel()` returns the callee is never mid-hop. - // - // DETERMINATE = every callee thread finished, or the callee is parked - // on a scheduler condition — exactly `async-start-call`'s rule - // (fact_calls.ts). Deliberately NOT "or `st.resolved()`": - // resolved-but-mid-hop is precisely the stale state this park exists - // to avoid. A callee whose callback already ran `task.cancel` and - // returned EXIT is resolved while its `Thread` is still parked on - // `awaitValue` holding `inst.exclusiveThread`; answering from here - // then makes the NEXT `subtask.cancel` see the instance excluded and - // report BLOCKED, and the guest's `subtask.drop` traps "not yet - // resolved" (test/async/reentrance.wast:517). + // Cancellation may resume the callee on a later JSPI microtask. + // Wait until all its threads finish or it genuinely parks in the + // scheduler. Unlike async-start-call, resolution alone is NOT enough: + // a resolved-but-mid-hop callee may still hold exclusiveThread and + // affect the next cancellation's answer. Host-import subtasks have + // no callee task and are immediately determinate, even if unresolved. // - // A callee with a pending (undeliverable) cancel sits parked - // non-cancellably, which is determinate, so the genuine BLOCKED - // answer is still immediate. Host-import subtasks carry no callee - // task and cannot be mid-hop: the default onCancel resolves them - // before this point, and a `deferCancel` import's no-op onCancel - // leaves them simply unresolved — either way the answer is immediate. - // - // NAMED DIVERGENCE (docs/architecture.md §6, #92): this park makes - // the async built-in non-atomic — other ready threads may run while - // it waits, a reordering within the reference's Store.tick freedom - // taken one built-in early. + // The async determinacy park is the named non-atomicity divergence + // (#92): other ready threads may run before this built-in returns. const callee = st.calleeTask as | { threads: { done(): boolean }[] } | null; @@ -590,10 +446,8 @@ export function createSubtaskCancel( // The SYNC form additionally blocks until the callee actually // resolves (definitions.py `canon_subtask_cancel`: // `thread.wait_until(subtask.resolved)`), then reports the resolved - // state through the same tail as the non-blocking path — SITE 5 - // (lit), mirroring SITE 4 (stream_builtins.ts) and - // `Waitable.waitForPendingEvent`. The ASYNC form answers BLOCKED as - // soon as the callee is determinate and still unresolved. + // state through the same tail as the non-blocking path. The ASYNC + // form answers BLOCKED once determinate and still unresolved. const ready = (): boolean => determinate() && (async_ || st.resolved()); if (mode !== "jspi") { @@ -608,16 +462,10 @@ export function createSubtaskCancel( } if (!ready()) { parked = true; - // #295: the ASYNC form's claim ends where the reference's does — - // `canon_subtask_cancel` clears `has_sync_waiter` before returning - // BLOCKED (definitions.py:2459-2461), so the flag is held only - // across the synchronous claim window around `on_cancel()`. The - // #92 determinacy park below is not part of that window; holding - // the flag across it would trap a sibling thread's - // `waitable.join` on this subtask where the reference succeeds - // (#92 licenses a reordering, not a new trap condition). The SYNC - // form keeps it set across its wait, as the reference does - // (`thread.wait_until(subtask.resolved)` precedes the clear). + // Async cancellation releases its synchronous claim before the + // determinacy park, allowing a sibling's waitable.join. Issue #92 + // permits reordering, not a new trap condition. Sync cancellation + // retains the claim through its resolution wait, as in the reference. if (async_) st.hasSyncWaiter = false; return blockCurrentActivation({ store: inst.store, @@ -647,7 +495,7 @@ export function createSubtaskCancel( // --------------------------------------------------------------------------- /** - * definitions.py `canon_thread_yield` (line 2728). + * definitions.py `canon_thread_yield`. * * Yielding blocks the calling wasm frame until the scheduler comes back to * it. A callback-ABI guest expresses the same intent by returning the `YIELD` @@ -669,11 +517,11 @@ export function createThreadYield( // (definitions.py `Thread.yield_` -> `wait_until` -> `deliver_pending_cancel`). if (thread.task.deliverPendingCancel(cancellable)) return 1; if (mode === "jspi") { - // SITE 3 (lit). definitions.py `Thread.yield_` is + // definitions.py `Thread.yield_` is // `wait_until(lambda: True, cancellable)`: immediately ready, but it // goes through the scheduler, so other threads get a turn first. A // suspension point with an always-true `readyFunc` is exactly that -- - // `Store.tick` will resume it, after whatever else is already ready. + // `Store.tick` selects it under the configured scheduling policy. return blockCurrentActivation({ store: thread.task.inst.store, task: thread.task, @@ -707,15 +555,12 @@ function requireWaitableSet( } /** - * The two event payload words. Hoisted out of `unpackEvent` because - * cabi/layout.ts and cabi/types.ts memoize on `ValType` identity (issue - * #261): a fresh literal per call is a guaranteed cache miss plus a - * `WeakMap.set` on immediate garbage, twice per event delivered. + * Shared type for both payload words so event delivery reuses cached layouts. */ const EVENT_PAYLOAD_TYPE: ValType = Object.freeze({ kind: "u32" }); /** - * definitions.py `unpack_event` (line 2422): store the two payload words at + * definitions.py `unpack_event`: store the two payload words at * `ptr` and return the event code. */ function unpackEvent( @@ -731,9 +576,5 @@ function unpackEvent( return event; } -// Structural `ValType` equality (the circular-structure bugfix) moved to cabi/types.ts -// (`valTypesEqual`) when the #18 tls smoke found its stream-element sibling; -// the contract note lives there now. - -/** Unused-import guard: `trap` is re-exported for symmetry with cabi. */ +/** Unused-import guard. */ void trap; diff --git a/runtime/src/intrinsics/context.ts b/runtime/src/intrinsics/context.ts index 3e93603..bffaead 100644 --- a/runtime/src/intrinsics/context.ts +++ b/runtime/src/intrinsics/context.ts @@ -1,14 +1,11 @@ -// `CoreDef::UnsafeIntrinsic` (plan v1 / contracts/plan-format.md v0.3): +// `CoreDef::UnsafeIntrinsic` (contracts/plan-format.md): // wasmtime compile-time builtins that a component's core modules import // directly, bypassing the trampoline table. // -// Of wasmtime-environ 47.0.3's 21 unsafe intrinsics -// (`component/intrinsic.rs`, `for_each_unsafe_intrinsic!`) only four have -// Component Model meaning: `context-{get,set}-i32-{0,1}`, the canonical -// `context.get` / `context.set` built-ins. The other seventeen -// (`*-native-load` / `*-native-store` / `store-data-address`) are raw host -// memory access for wasmtime's own internals; they have no portable meaning -// in a JS host and are refused at instantiate time. +// Supported symbols are `context-{get,set}-i32-{0,1}`, the canonical +// `context.get` / `context.set` built-ins. Native-memory symbols such as +// `*-native-load`, `*-native-store` and `store-data-address` have no portable +// meaning in a JS host and are refused at instantiate time. import { assert_, trapIf } from "../cabi/trap.ts"; import { currentThreadForInstance } from "../task/mod.ts"; @@ -19,19 +16,17 @@ import { UnsupportedFeatureError } from "./errors.ts"; /** * Number of `i32` context slots per thread. definitions.py `Thread.storage` - * is initialised `[0,0]` (line 347) and `canon_context_{get,set}` assert + * is initialised `[0,0]` and `canon_context_{get,set}` assert * `i < len(thread.storage)` — so exactly two, matching the intrinsic names * `context-*-i32-0` and `context-*-i32-1`. */ export const NUM_CONTEXT_SLOTS = 2; /** - * definitions.py `canon_context_get` (line 2348). + * definitions.py `canon_context_get`. * * The storage is **per thread**, not per task: two threads of one task have - * independent context. wit-bindgen 0.60 keeps its async-executor task pointer - * in slot 0, which is why this intrinsic is the entry blocker for async - * guests. + * independent context. */ export function canonContextGet(i: number, inst?: unknown): number { const thread = currentThreadForInstance(inst); @@ -42,9 +37,7 @@ export function canonContextGet(i: number, inst?: unknown): number { return result >>> 0; } -// Standing probe (CE_CTX_TRACE=1): per-call context-slot traffic with the -// full ambient state — the instrument that isolated issue #24. Cheap and -// env-gated; keep. +// CE_CTX_TRACE=1 logs context-slot traffic and ambient thread attribution. const CTX_TRACE = (() => { try { return Deno.env.get("CE_CTX_TRACE") === "1"; @@ -66,7 +59,7 @@ function trace(msg: string, thread: unknown): void { ); } -/** definitions.py `canon_context_set` (line 2358). */ +/** definitions.py `canon_context_set`. */ export function canonContextSet(i: number, v: number, inst?: unknown): void { const thread = currentThreadForInstance(inst); assert_(i < NUM_CONTEXT_SLOTS, `context.set slot ${i} out of range`); diff --git a/runtime/src/intrinsics/errors.ts b/runtime/src/intrinsics/errors.ts index 436e198..1c396c4 100644 --- a/runtime/src/intrinsics/errors.ts +++ b/runtime/src/intrinsics/errors.ts @@ -1,6 +1,5 @@ -// Shared failure type for trampolines/intrinsics gated on a not-yet-built -// runtime capability. Split out of ./mod.ts so sibling intrinsic modules can -// raise it without importing the (much larger) trampoline dispatcher. +// Unsupported trampoline/intrinsic diagnostic, shared without importing +// the trampoline dispatcher. /** Instantiate-time failure for functionality gated on a missing capability. */ export class UnsupportedFeatureError extends Error { diff --git a/runtime/src/intrinsics/fact_calls.ts b/runtime/src/intrinsics/fact_calls.ts index 5f1fd18..2623fb2 100644 --- a/runtime/src/intrinsics/fact_calls.ts +++ b/runtime/src/intrinsics/fact_calls.ts @@ -6,19 +6,16 @@ // definitions.py, because the reference has no fused adapters: there, // `canon_lower` calls the callee's `FuncInst` directly and the host performs // every copy. wasmtime instead compiles a FACT adapter that hoists the copying -// into wasm and asks the host to do only the task bookkeeping. The *semantics* -// are the reference's; only the division of labour differs. +// into wasm and asks the host to do the task bookkeeping. JSPI timing and +// instance-poisoning differences are documented at the relevant sites below. // // =========================================================================== -// THE PROTOCOL (wasmtime-environ 47.0.3) +// THE PROTOCOL (wasmtime-environ FACT) // =========================================================================== // -// Emission sites: `fact/trampoline.rs` — `call_prepare` (line 513), -// `compile_async_to_async_adapter` (474), `compile_sync_to_async_adapter` -// (607), `compile_async_to_sync_adapter` (643). Signatures: `fact.rs` -// `import_prepare_call` (584), `import_sync_start_call` (620), -// `import_async_start_call` (643), with `PREPARE_CALL_FIXED_PARAMS` at -// `fact.rs:47`. +// Emission sites: `fact/trampoline.rs` `call_prepare` and the +// `compile_*_to_*_adapter` functions. Signatures: `fact.rs` +// `import_prepare_call`, `import_sync_start_call`, `import_async_start_call`. // // prepare-call(start: funcref, return: funcref, // caller_instance: i32, callee_instance: i32, @@ -31,16 +28,15 @@ // async-start-call(callee: funcref, param_count: i32, // result_count: i32, flags: i32) -> i32 (packed subtask) // -// **The `start` / `return` funcrefs are the reference's `on_start` / -// `on_resolve`.** That is the load-bearing finding, and their signatures -// (`fact/signature.rs`) say so exactly: +// The `start` / `return` funcrefs implement the reference's `on_start` / +// `on_resolve` copying (`fact/signature.rs`): // -// `[async-start]` (async_start_signature, line 61) +// `[async-start]` (async_start_signature) // params = the *caller's* flattened params (what prepare-call stashed) // results = the *callee's* flattened params (hand straight to the callee) // i.e. "given the caller's arguments, produce the callee's" — `on_start`. // -// `[async-return]` (async_return_signature, line 145) +// `[async-return]` (async_return_signature) // params = the *callee's* flattened results (+ a retptr when the caller // is async-with-results, or when the caller's results spill) // results = the *caller's* flattened results (empty if async/spilled) @@ -50,22 +46,9 @@ // arguments, and calls `return` with whatever the callee produced. This is why // a FACT task's payload is flat core values (`Task.factPassthrough`). // -// Two details that fall out of the emission sites: -// -// * `prepare-call` must NOT run the callee. The callee may be exerting -// backpressure, and the whole point of splitting prepare from start is to -// let the host stash the parameters until it clears (fact.rs:580-583). -// The stashed state feeds `Task.enterImplicitThread`, which is exactly -// where the reference's backpressure gate lives. -// * Reentrance between *related* instances is resolved statically: -// `trampoline.rs:116-127` emits an unconditional -// `trap(Trap::CannotEnterComponent)` when the lower and lift instances are -// the same or are ancestors of one another. The runtime-side counterpart -// — "is the callee instance currently executing" — does not exist at the -// pinned reference (definitions.py @ 2f13265 has no `may_enter`, CM#705), -// so the only refusal at these sites is polyengine's poisoned-corpse check -// (`entryRefusal`). The flat-instance-tree gap recorded in task/mod.ts is -// doubly not load-bearing here. +// `prepare-call` must not run the callee: parameters remain stashed until +// `Task.enterImplicitThread` admits it past backpressure and exclusivity. +// Reentrance is valid; `entryRefusal` checks only runtime instance poisoning. import { assert_, trap } from "../cabi/trap.ts"; import { MAX_FLAT_RESULTS } from "../cabi/mod.ts"; @@ -99,14 +82,14 @@ import { } from "../exec/boundary.ts"; import { traceCopy } from "./stream_builtins.ts"; -/** `PREPARE_ASYNC_NO_RESULT` (wasmtime-environ `component.rs:39`). */ +/** `PREPARE_ASYNC_NO_RESULT` (wasmtime-environ `component.rs`). */ const PREPARE_ASYNC_NO_RESULT = 0xffff_ffff; -/** `PREPARE_ASYNC_WITH_RESULT` (`component.rs:45`). */ +/** `PREPARE_ASYNC_WITH_RESULT` (`component.rs`). */ const PREPARE_ASYNC_WITH_RESULT = 0xffff_fffe; -/** `START_FLAG_ASYNC_CALLEE` (`component.rs:52`). */ +/** `START_FLAG_ASYNC_CALLEE` (`component.rs`). */ export const START_FLAG_ASYNC_CALLEE = 1; -/** Number of fixed leading parameters of `prepare-call` (`fact.rs:47`). */ +/** Number of fixed leading parameters of `prepare-call` (`fact.rs`). */ const PREPARE_FIXED = 8; /** The state `prepare-call` stashes for the following `*-start-call`. */ @@ -127,8 +110,7 @@ export interface PreparedCall { params: CoreValue[]; /** * Where the *caller's* results go, decoded from - * `result_count_or_max_if_async` exactly as wasmtime's `ResultInfo` - * (`concurrent.rs:2815-2836`): + * `result_count_or_max_if_async` as wasmtime's `ResultInfo`: * * * async caller **with** a result -> `Heap`, retptr = last param * * async caller without a result -> `Stack` @@ -137,25 +119,17 @@ export interface PreparedCall { * * sync caller otherwise -> `Stack` * * In the `Heap` case the retptr must be **appended** to the - * `[async-return]` arguments (`concurrent.rs:2916-2919`) — it is the last - * parameter of `async_return_signature` (fact/signature.rs:166,178), not + * `[async-return]` arguments: it is the last + * parameter of `async_return_signature` (`fact/signature.rs`), not * something the callee produced. */ resultInfo: { kind: "heap"; retptr: CoreValue } | { kind: "stack" }; /** True when the caller used the async ABI *and* has a result. */ asyncCallerWithResult: boolean; /** - * The memory `prepare-call` names (`component/info.rs:1059`: "the memory - * used to verify that the memory specified for the `task.return` that is - * called at runtime matches the one specified in the lifted export"). - * - * Decoded faithfully, but NOT usable for that verification: it is the - * *adapter's* view (`adapter.lift.options...memory`) and is `None` for - * callees whose own `task.return` options do name a memory. wasmtime gets - * away with the check because it holds the lift memory first-hand and its - * comparison is one-sided (concurrent.rs:3344-3358). Kept because it is the - * wire field and the task's options are structurally built from it; see the - * comment on the memory half of the check in async_builtins.ts. + * The adapter's lift memory, as named by `prepare-call`. It does not + * reliably identify the callee's `task.return` memory; see the skipped + * FACT memory check in `async_builtins.ts`. */ memory: unknown | null; } @@ -167,7 +141,7 @@ export interface FactCallContext { resultTypes(index: number): ValType[]; /** * Element types for a *raw* wasmtime `TypeTupleIndex` (what `prepare-call` - * passes as `task_return_type`), or `null` if the plan maps none — plan v3, + * passes as `task_return_type`), or `null` if the plan maps none; * see `LoadedPlan.resultTupleTypes`. */ resultTypesForTuple(tupleIndex: number): ValType[] | null; @@ -188,7 +162,7 @@ export interface FactCallContext { * The single in-flight prepared call. wasmtime keeps this per *task*; a * single slot is equivalent here because `prepare-call` and its * `*-start-call` are emitted back-to-back in one adapter body - * (`trampoline.rs:486-508`) with no suspension point between them, so two + * with no suspension point between them, so two * preparations can never be outstanding at once. Asserted, not assumed. */ prepared: { current: PreparedCall | null }; @@ -204,15 +178,9 @@ function taskOptionsFor( calleeUsesAsyncAbi: boolean, ): TaskOptions { return { - // NOTE the distinction definitions.py draws and this code initially got - // wrong: `Task.ft.async` is the *function type*'s asyncness (what - // `prepare-call` passes as `callee_async`), while `Task.opts.async_` is - // the *canonical options*' asyncness — and `canon_lift` branches on the - // latter (`if not opts.async_:` at line 2168). A function can be - // async-*typed* yet lifted with sync options, in which case the reference - // takes its plain synchronous path. Branching on the type instead sent - // those callees down the stackful path and reported a bogus JSPI - // requirement (`test/async/cross-abi-calls.wast`'s `async-calls-sync-*`). + // Canonical ABI asyncness is distinct from `prepared.calleeAsync` (the + // function type). An async-typed function can use the sync lift ABI; + // `canon_lift` selects result handling from its canonical options. async_: calleeUsesAsyncAbi, callback: callback !== null, stringEncoding: stringEncodingName(prepared.stringEncoding), @@ -268,22 +236,9 @@ export function createPrepareCall( ctx.prepared.current === null, "prepare-call with a preparation already outstanding", ); - // GAP (tracked): wasmtime performs a `check_blocking` here — - // if let (CallerInfo::Sync { .. }, true) = (&caller_info, callee_async) { - // store.0.check_blocking()?; // concurrent.rs:2802-2807 - // } - // i.e. a *sync-lowered* caller reaching an *async-typed* callee must - // itself have been created by an async export, else it traps: only a task - // that is allowed to block may make a blocking call. We cannot evaluate it - // yet — it needs a "may this task block" bit on `Task`, which the reference - // models through its thread/task structure rather than a flag. Its absence - // means we accept some components wasmtime rejects; it never causes a - // wrong answer for an accepted one. `test/async/trap-if-block-and-sync.wast` - // is the file that exercises it, and that file is independently blocked on - // the wasmparser pin drift, so nothing observable depends on it today. const params = args.slice(PREPARE_FIXED) as CoreValue[]; const rc = Number(rc_) >>> 0; - // wasmtime `ResultInfo` (concurrent.rs:2815-2836). + // Decode the caller's result location before the copy adapter runs. const lastParam = (): CoreValue => { assert_(params.length > 0, "prepare-call: retptr missing"); return params[params.length - 1]; @@ -348,9 +303,8 @@ function mkCalleeTask(input: { canBlock?: boolean; /** * Called when `[async-start]` has actually run, i.e. when the callee really - * started. wasmtime sets its `Status::Started` event at exactly this point, - * inside the `lower_params` closure (concurrent.rs:2903-2908) — *not* when - * the call was prepared. Under backpressure `enter_implicit_thread` blocks + * started, not when the call was prepared. Under backpressure + * `enter_implicit_thread` blocks * first, so a subtask observed before this fires must still report STARTING. */ onStarted?: () => void; @@ -362,7 +316,7 @@ function mkCalleeTask(input: { onCallerResults: (r: CoreValue[] | null) => void; /** * Caller-side lender registrar for borrows transferred during - * `[async-start]` (definitions.py `lift_borrow` line 1517 adds lenders to + * `[async-start]` (definitions.py `lift_borrow` adds lenders to * the caller's Subtask). async-start-call passes its `Subtask` (whose * `deliverResolve` releases them); sync-start-call passes a scope it * releases when the blocked caller frame gets its results. @@ -376,36 +330,16 @@ function mkCalleeTask(input: { } { const { prepared, callee, callback, postReturn, ctx, calleeUsesAsyncAbi } = input; - // CONTRACT: default to `plain` when the context predates this field. Only - // `jspi` may wrap, and wrapping a non-wasm callee throws outright, so the - // conservative reading of an absent mode is "no suspension discipline". + // Absent suspension information must not force a promising entry. const mode = input.mode ?? "plain"; - // CONTRACT: default false -- a context that cannot answer the question gets - // the non-wrapping (plain-shaped) behaviour, which is the conservative one: - // it never forces asynchrony that the ABI forbids. const canBlock = input.canBlock ?? false; const memory = prepared.memory; const inst = prepared.calleeInst; - // `ft` for the task: only `async` and `results` are consulted — - // `Task.needsExclusive` reads the former, `canon_task_return`'s result-type - // check reads the latter (which is why `prepare-call` carries - // `task_return_type` at all: fact.rs's comment on `PrepareCall.memory` says - // the same for the memory check). - // `task_return_type` arrives as wasmtime's *own* `TypeTupleIndex` — a - // runtime argument, not a plan field. The task-return trampoline's raw - // `results` key + interned `resultType` (contracts/plan-format.md schema) - // supplies the dictionary for it: every `task-return` - // trampoline decl carries that raw index alongside its interned - // `plan.types` entry, so the callee task CAN now carry its declared result - // types and `canon_task_return`'s `trap_if(result_type != task.ft.result)` - // applies to FACT tasks too (async_builtins.ts). - // - // `null` = the plan has no `task.return` trampoline for this tuple, i.e. - // the callee cannot call `task.return` at all (a sync-lifted callee reached - // through an async-to-sync adapter). Then the check has nothing to compare - // against and stays skipped — flagged by `factResultTypesKnown` rather than - // by an empty-results coincidence. + // `task_return_type` is a raw wasmtime `TypeTupleIndex`, not a `plan.types` + // index. The loader maps it through task-return declarations. A missing + // mapping leaves placeholder results and disables the result-type check; + // an empty tuple with a mapping is a known, checkable type. const declaredResults = ctx.resultTypesForTuple(prepared.taskReturnType); const ft: FuncType = { params: [], @@ -418,17 +352,15 @@ function mkCalleeTask(input: { taskOptionsFor(prepared, callback, memory, calleeUsesAsyncAbi), inst, // on_start: the adapter's `[async-start]` turns the caller's flat params - // into the callee's flat params (fact/signature.rs:61). + // into the callee's flat params (`async_start_signature`). // // An async caller that has a result passes its retptr as the *last* flat // parameter; `[async-start]` does not declare it, so it is chopped off - // here exactly as wasmtime does (concurrent.rs:2869-2876, "Async callers, - // if they have a result, use the last parameter as a return pointer so - // chop that off"). Sync callers forward everything directly. + // here. Sync callers forward everything directly. () => { // Open the FACT borrow window for the duration of the copy adapter: // `[async-start]` is where argument resource transfers run, and it - // cannot block (see the WASM-ENTRY note below), so push/pop brackets a + // cannot block, so push/pop brackets a // strictly synchronous window. Borrow bookkeeping lands on this // (callee) task's `numBorrows` and the caller's lender scope — see // intrinsics/mod.ts `FactStartScope`. @@ -448,7 +380,7 @@ function mkCalleeTask(input: { return calleeArgs; }, // on_resolve: the adapter's `[async-return]` turns the callee's flat - // results into the caller's (fact/signature.rs:145). + // results into the caller's (`async_return_signature`). (result) => { if (result === null) { // Cancelled before returning: there is nothing for `[async-return]` @@ -459,10 +391,7 @@ function mkCalleeTask(input: { } // `[async-return]` takes the callee's flat results and, when the // caller's results live in linear memory, the caller-supplied return - // pointer as a trailing argument (fact/signature.rs:166,178; appended by - // wasmtime at concurrent.rs:2916-2919). Omitting it made the adapter - // read `undefined` for that parameter, which coerces to 0 — every - // spilled result was written to linear-memory address 0. + // pointer as a trailing argument, not a callee-produced result. const args = result as CoreValue[]; const withRetptr = prepared.resultInfo.kind === "heap" ? [...args, prepared.resultInfo.retptr] @@ -481,45 +410,12 @@ function mkCalleeTask(input: { if (!(yield* task.enterImplicitThread(thread))) return; const calleeArgs = task.start(); traceCopy(`mkCalleeTask callee canBlock=${canBlock} mode=${mode}`); - // WASM ENTRY (3 of 3 that can reach a blocking built-in). - // - // The other two — a lifted export's core function and a callback export — - // are entered through `awaitCore`, which establishes the - // activation-attached ambient. This one was not, and it is precisely the - // entry that owns a FACT sync-call bracket: `enter-sync-call` runs here - // under this task, and if the callee suspends, the engine resumes it later - // with no driver. Without the ambient travelling with the activation the - // matching `exit-sync-call` had no task in scope at all (traced as - // `ENTER-SYNC owner=K26` / `EXIT-SYNC owner=EXECUTOR`). - // - // Entries deliberately NOT wrapped: `realloc`, `post-return`, resource - // destructors and the `[async-start]`/`[async-return]` copy adapters. - // None of them may block — they cannot reach a canonical built-in that - // suspends — so the engine can never resume them, and wrapping would only - // cost an ALS frame on the hot copy path. - // The callee is its own activation and must get its own `promising` - // entry, not merely an ambient scope: otherwise it runs *inside* whatever - // `Suspending` trampoline invoked us, putting our JS frame between the - // caller's promising entry and any suspension the callee reaches -- - // `SuspendError: trying to suspend JS frames` (jspi pin (b), mechanics.ts - // line 12). This is only coherent together with site 1 below blocking - // rather than raising `NeedsJspi`, since a promising callee resolves on a - // later turn by construction. - // NOTE: this wrap is RIGHT for a callee that blocks and - // Wrap ONLY a callee that can actually reach a suspension point. - // - // The wrap is required when the callee blocks: without its own `promising` - // entry it would suspend inside whatever `Suspending` trampoline invoked - // us, with our JS frame in between (`SuspendError: trying to suspend JS - // frames`, jspi pin (b)). But `enterWasm` returns a Promise - // unconditionally, so wrapping a callee that CANNOT block forces - // asynchrony the ABI forbids: an eagerly-completing callee must report its - // subtask RETURNED, and a wrapped one reports STARTED. That broke all six - // `async-calls-sync-*` cases of cross-abi-calls.wast. - // - // There is no per-CALL discriminator -- the same call site serves both -- - // so the answer is per-callee, derived from whether the callee's core - // instance imports any blocking trampoline (`Executor.suspendableFuncs`). + // `awaitCore` carries the thread ambient across resumption, including + // its FACT borrow brackets. A suspendable callee also needs its own + // promising entry so this JS frame is not on the suspended wasm stack. + // Do not wrap non-blocking callees: promising forces a microtask hop and + // would turn eager RETURNED status into STARTED. Classification comes + // from `Executor.suspendableFuncs`, not from function-type asyncness. const raw = yield* awaitCore( canBlock ? enterWasm(callee, mode) : callee, calleeArgs as CoreValue[], @@ -527,26 +423,17 @@ function mkCalleeTask(input: { ); if (!calleeUsesAsyncAbi) { - // Sync canonical options (definitions.py `canon_lift` line 2168, `if not - // opts.async_`): the callee returns its results directly and resolves + // Sync canonical options (definitions.py `canon_lift`): the callee + // returns its results directly and resolves // before returning. Reached via `compile_async_to_sync_adapter`, which // passes flags without `START_FLAG_ASYNC_CALLEE`. task.return_(raw as CoreValue[]); if (postReturn !== null) { assert_(inst.mayLeave, "post-return with may_leave already false"); inst.mayLeave = false; - // NO local try/finally here, deliberately (#91, verified rather than - // assumed). definitions.py `canon_lift` (lines 2170-2174) has the - // same bare bracket: a trapping post-return skips `may_leave = True` - // and, since `Store.lift`'s `leave_to` is also skipped, leaves the - // instance poisoned — restoring `may_leave` locally would contradict - // both. What this runtime additionally needs, because it supports - // post-trap re-entry, is that no *live* instance is stranded with - // `may_leave === false`; exec/boundary.ts `unwind` covers exactly - // that: at the host boundary no lift or lower is in flight, so it - // asserts that resting state for every instance outside the poisoned - // entered set. This instance is either in that set (poisoned, left - // as the trap left it) or restored there. + // No local finally: a trap skips the reference's restore. Host-boundary + // unwind may restore sibling flags; see createLiftedFunction's + // entry-identity rule in exec/boundary.ts. callCore(postReturn, raw as CoreValue[]); inst.mayLeave = true; ctx.stats.postReturnsRun++; @@ -556,20 +443,9 @@ function mkCalleeTask(input: { } if (callback === null) { - // Stackful async lift -- definitions.py `canon_lift` line 2178: - // - // if not opts.callback: - // [] = call_and_trap_on_throw(callee, flat_args) - // task.exit_implicit_thread() - // return - // - // That is the whole path. The callee runs to completion on its own - // stack, returning NO results and calling `task.return` itself; any - // blocking happened *inside* it, through the canonical built-ins. Which - // is exactly what the callee's own `promising` entry provides when it - // can block -- the `awaitCore` above parks the CALLEE's thread, not the - // caller's, so nothing here parks an async-lowered caller (the mistake - // the cross-abi differential caught). + // Stackful async lift: results arrive through `task.return`, not the + // core return. `awaitCore` parks the callee's thread, never the + // async-lowered caller's frame waiting for completion. normalizeCoreValues(raw, [], "stackful callee result"); task.exitImplicitThread(thread); return; @@ -582,16 +458,8 @@ function mkCalleeTask(input: { task, thread, inst, - // The callback re-entry is the second of the three entries that can - // reach a blocking built-in (jspi/bridge.ts's invariant) and gets the - // same per-callee treatment as the initial entry above: a callee that - // parks (WAIT) and then, on a later callback activation, reaches a - // *synchronous* blocking built-in (wit-bindgen's `block_on` shape — - // e.g. a composed iroh endpoint signing a CertificateVerify via - // `waitable-set.wait` mid-handshake) suspends inside the plain - // callback frame otherwise: `SuspendError` (jspi pin (c)). Caught by - // the first composed consumer workload (wosh client), not by - // cross-abi-calls.wast, whose callees only ever block via WAIT codes. + // Callback-ABI code can also call synchronous blocking built-ins; + // apply the same suspension classification on callback re-entry. callback: canBlock ? enterWasm(callback!, mode) : callback!, packed, stats: ctx.stats, @@ -616,8 +484,8 @@ function takePrepared(ctx: FactCallContext, what: string): PreparedCall { /** * A sync-lowered import calling an async-lifted export - * (`compile_sync_to_async_adapter`, trampoline.rs:607). The caller's wasm frame - * is blocked for the duration, so this must produce the results *now*. + * (`compile_sync_to_async_adapter`). The caller's wasm frame cannot continue + * until this intrinsic delivers its results. */ export function createSyncStartCall( decl: { callback: number | null }, @@ -633,7 +501,7 @@ export function createSyncStartCall( let callerResults: CoreValue[] | null = null; // The caller's frame is blocked for the whole call, so resolution // delivery = this intrinsic returning results: release lenders then - // (the sync analogue of `Subtask.deliver_resolve`, definitions.py 904). + // (the sync analogue of definitions.py `Subtask.deliver_resolve`). // Inlined rather than reusing `SyncCallScope` to keep this module free // of a value-level import cycle with intrinsics/mod.ts. const lentHandles: { numLends: number }[] = []; @@ -654,7 +522,7 @@ export function createSyncStartCall( postReturn: null, ctx, // `sync-start-call` exists only for "sync-lowered import to async-lifted - // export" (fact.rs:608), so the callee always uses the async ABI. + // export", so the callee always uses the async ABI. calleeUsesAsyncAbi: true, mode: ctx.suspensionMode, canBlock: ctx.calleeCanBlock?.(callee) ?? false, @@ -670,9 +538,7 @@ export function createSyncStartCall( lenderScope, }); - // Reference `Store.lift` (@ 2f13265) has no reentrance gate; the only - // refusal left is polyengine's poisoned-corpse check, whose message names - // the original trap (polyengine#145). + // Refuse a poisoned callee, not reentrance into a live one. { const refusal = entryRefusal( prepared.calleeInst, @@ -688,90 +554,25 @@ export function createSyncStartCall( // A trap poisons the callee instance. A *capability signal* does not — // see the `isCapabilitySignal` note in exec/boundary.ts. if (!(e instanceof NeedsJspi) && !(e instanceof PendingCapability)) { - // Retire the poisoned CALLEE's stream/future ends (#66): the trap + // Retire the poisoned callee's stream/future ends: the trap // unwinds to a hooked site that walks only the CALLER's chain — a // composed component's callee would otherwise strand its host peers. notifyInstancePoisoned(prepared.calleeInst, e); } - // The lent handles are the CALLER's, and the caller is not poisoned by - // either exit (contracts/intrinsics.md §A's trap-unwind/lender-release - // obligation: this runtime - // deliberately supports post-trap re-entry on the caller side, where - // the reference kills the whole store, so the sync-call scopes it - // skipped have to be unwound explicitly). Leaving `numLends` elevated - // would make every later `lift_own`/`resource.drop` of those handles - // trap "handle still lent out" (#91). Release is idempotent, and the - // success path below is unchanged. + // Release the caller's lenders on failure too: surviving callers must + // not retain a borrow obligation from a call that can no longer deliver. lenderScope.releaseLenders(); throw e; } if (callerResults === null) { - // The callee did not resolve within its first activation. definitions.py - // `canon_lower`'s sync path blocks here — `thread.wait_until( - // subtask.resolved)` (line 2286) — suspending the *caller's* wasm frame - // while the scheduler runs other threads. That is JSPI role 2 (docs/architecture.md - // §6), and it is the first place a purely stackless runtime genuinely - // cannot proceed. - // - // Note this is NOT the sync driving loop of `canon_lift`: that loop - // drives the callee instance's own threads and is only correct when the - // callee can finish without anything from the caller. Here the caller is - // mid-frame and may be exactly what the callee is waiting for, so - // pumping the callee alone would spin rather than make progress. - // Note: the callee's thread stays parked in `store.waiting` when we bail - // here. That is deliberate — unwinding it would run callee cleanup the - // guest never asked for — but it does mean the store keeps a thread that - // will never be resumed. Harmless today (the enclosing host call is - // failing anyway, and the instance is not poisoned because no trap - // escaped a task), and it disappears once JSPI lets this path actually - // block instead of bailing. + // The sync `canon_lower` wait suspends the caller while the scheduler + // runs other threads. Do not substitute a callee-only driving loop: + // progress may depend on work in the caller's instance. if (ctx.suspensionMode === "jspi") { - // JSPI role 2 (docs/architecture.md §6): park the *caller's* wasm activation until - // the callee resolves, exactly as definitions.py `canon_lower`'s sync - // path does with `thread.wait_until(subtask.resolved)` (line 2286). - // The scheduler keeps ticking the callee meanwhile; when it produces - // results our `readyFunc` goes true and the engine resumes the caller. - // - // Not cancellable: a sync-lowered caller has no way to observe or - // request cancellation mid-call -- the reference's wait here carries - // no cancellation branch. - // LENDER RELEASE ON EVERY SETTLE PATH (#102). - // - // Enumeration of how this `SuspensionPoint` can reach a terminal - // state (jspi/bridge.ts `SuspensionPoint`), and whether `produce` - // runs on each: - // - // 1. `resume(false)` -> `produce` returns the packed result. - // RUNS. This is the success path; release stays INSIDE `produce`, - // before the results are shaped, so its ordering relative to the - // produced value is unchanged by this fix. - // 2. `resume(false)` -> `produce` throws (a trap computed at resume - // time). PARTIALLY RUNS. Release is `produce`'s first statement - // so it is already discharged here, but the `onSettled` backstop - // makes that independent of statement order. - // 3. `resume(true)` — a CANCELLED resume. Unreachable by - // construction: this park is `cancellable: false` and - // `SuspensionPoint.resume` asserts `cancellable || !cancelled` - // (#93). Note the assert fires BEFORE `#done` is set, so such a - // call leaves the point still parked and never settles it — a - // scheduler bug, not a guest-reachable exit; there is no - // non-poisoning continuation to release into. - // 4. `abandon(reason)` — store teardown / abandonment: fails the - // import's Promise WITHOUT calling `produce`. DOES NOT RUN. This - // is the #102 hole; `onSettled` covers it. - // 5. Never settled at all (the store is dropped while this point - // sits in `store.waiting`, e.g. the caller's whole host call was - // abandoned). No JS runs, so nothing can release; the lent - // handles die with the store, which is the reference's own - // outcome. Out of scope for the trap-unwind/lender-release obligation (no non-poisoning exit). - // 6. Trap-poisoning of the parked instance: does not settle this - // point by itself — it reaches the guest either as (2) (a - // produce-time trap) or as (4) (teardown abandons the park), so - // it is covered by those two rows, not a third mechanism. - // - // `releaseLenders` is idempotent (#91), so the backstop is a no-op - // whenever `produce` already ran. + // The reference's sync wait is non-cancellable. Release lenders + // before returning results; the idempotent onSettled backstop also + // covers abandonment, which never invokes produce. return blockCurrentActivation({ store: prepared.callerInst.store, task: currentTask(), @@ -784,9 +585,7 @@ export function createSyncStartCall( onSettled: () => lenderScope.releaseLenders(), }); } - // A capability signal is expressly NON-poisoning (see above), so - // stranding the caller's lenders here is strictly worse than on the - // trap path: the caller is guaranteed to keep running (#91). + // Capability failure is non-poisoning, but still ends this lender scope. lenderScope.releaseLenders(); needsJspi( "sync-start-call whose async-lifted callee did not resolve in its " + @@ -800,11 +599,7 @@ export function createSyncStartCall( /** * Release a never-delivered subtask's lenders after a trap or capability bail - * broke the `[async-start-call]` bracket (#91). - * - * Now a thin alias of `Subtask.unwindLenders` — the same unwind serves the - * host-import parks (exec/boundary.ts, #106) — kept for the local name the - * `[async-start-call]` comments reference. + * broke the `[async-start-call]` bracket. */ function unwindSubtaskLenders(subtask: Subtask): void { subtask.unwindLenders(); @@ -826,7 +621,7 @@ function shapeResults(out: CoreValue[] | null): CoreValue | undefined { * / `compile_async_to_sync_adapter`). Returns the packed subtask status the * guest already knows how to interpret — the same * `state | (subtaski << 4)` encoding `canon_lower` produces - * (definitions.py line 2308), so the caller's callback loop and waitable sets + * (definitions.py `canon_lower`), so the caller's callback loop and waitable sets * work unchanged. */ export function createAsyncStartCall( @@ -865,7 +660,7 @@ export function createAsyncStartCall( : ctx.callback(decl.postReturn), ctx, // `compile_async_to_async_adapter` sets START_FLAG_ASYNC_CALLEE; - // `compile_async_to_sync_adapter` passes 0 (trampoline.rs:508 and :764). + // `compile_async_to_sync_adapter` passes 0. calleeUsesAsyncAbi: ((flags ?? 0) & START_FLAG_ASYNC_CALLEE) !== 0, mode: ctx.suspensionMode, canBlock: ctx.calleeCanBlock?.(callee) ?? false, @@ -874,7 +669,7 @@ export function createAsyncStartCall( subtask.state = SubtaskState.STARTED; // `onProgress` is a no-op until the guest has a handle for this // subtask, mirroring `canon_lower`'s `maybe_on_progress` - // (definitions.py line 2296): a call that starts before + // in definitions.py: a call that starts before // `async-start-call` returns reports STARTED in its packed result // instead, with no event. onProgress(); @@ -887,7 +682,7 @@ export function createAsyncStartCall( if (!subtask.resolved()) { subtask.resolve( r === null - // definitions.py `canon_lower`'s `on_resolve` (line 2267): a + // definitions.py `canon_lower`'s `on_resolve`: a // cancelled callee resolves CANCELLED_BEFORE_{STARTED,RETURNED} // depending on how far it got. ? (subtask.state === SubtaskState.STARTING @@ -900,18 +695,18 @@ export function createAsyncStartCall( onProgress(); }, // Borrow lenders attach to the caller-side subtask, released by its - // `deliverResolve` (definitions.py `Subtask.deliver_resolve`, line 904). + // `deliverResolve` (definitions.py `Subtask.deliver_resolve`). lenderScope: subtask, }); // Cross-component cancellation: `subtask.cancel` forwards to the callee - // task's `request_cancellation` (definitions.py line 519), which delivers + // task's `request_cancellation`, which delivers // TASK_CANCELLED to a cancellable block point — for a callback-ABI callee // that is its WAIT/YIELD, so the guest observes the cancellation and calls // `task.cancel`, resolving this subtask CANCELLED_BEFORE_RETURNED. subtask.onCancel = (callerInst) => task.requestCancellation(callerInst); subtask.calleeTask = task; - // A poisoned callee's refusal names the original trap (polyengine#145). + // A poisoned callee's refusal names the original trap. { const refusal = entryRefusal( prepared.calleeInst, @@ -927,23 +722,19 @@ export function createAsyncStartCall( } catch (e) { // See the sync form above and `isCapabilitySignal` in exec/boundary.ts. if (!(e instanceof NeedsJspi) && !(e instanceof PendingCapability)) { - // Retire the poisoned callee's ends (#66), as in the sync form above. + // Retire the poisoned callee's ends, as in the sync form above. notifyInstancePoisoned(prepared.calleeInst, e); } // The subtask never reached `report()`, so it has no handle in the - // caller's table and nothing will ever deliver its resolution — but it - // holds `num_lends` on the caller's handles. Resolve it as cancelled - // (the state the reference's `on_resolve(None)` would give a call that - // never started/returned) and deliver, which is what releases the - // lenders (definitions.py `Subtask.deliver_resolve`, line 902). See the - // sync form above for why the caller must not be left holding them. + // caller's table and nothing will deliver its resolution. Unwind its + // lenders explicitly so surviving callers can use their handles again. unwindSubtaskLenders(subtask); throw e; } const report = (): CoreValue => { if (subtask.resolved()) { - // Eager completion: no handle, no event (definitions.py line 2293). + // Eager completion: no handle, no event (`canon_lower`). subtask.deliverResolve(); traceCopy(`async-start-call -> RETURNED (eager)`); return SubtaskState.RETURNED; @@ -958,80 +749,20 @@ export function createAsyncStartCall( return packed; }; - // NO WAIT FOR RESOLUTION HERE, deliberately. An async-lowered caller must - // not block on its callee's *completion* -- that is the entire point of - // async lowering: it takes a subtask handle and learns of completion - // through events. An earlier attempt parked the caller here - // until the callee resolved. It made cross-abi-calls agree in both modes, - // and it broke the thing it had no business touching: the caller's - // activation was now suspended, so the sync-lowered parked caller of - // `handshake_test.ts` was never resumed and the run hung. Correct-looking, - // semantically wrong. - // - // What jspi mode DOES need is a wait for **determinacy** (jspi pin (j), - // `fastpath_hop_test.ts`): the engine defers a promising callee's - // continuation to a microtask at EVERY Suspending call -- even one whose - // value was available synchronously -- so a callee the reference would - // run to completion inside this call (`canon_lift` drives the thread to - // its first real block point before `canon_lower` returns) is still - // mid-hop when `report()` runs. Reporting then is reporting a state the - // reference can never observe: STARTED for a call that eagerly RETURNED - // (big-interleaving's `call-import` scripts), or a missed synchronous - // cancellation (its `subtask-cancel` scripts). - // - // "Determinate" is exactly one of: - // * the subtask resolved (task.return ran mid-activation), or - // * the callee's thread finished (results flow through the body), or - // * the callee genuinely parked on a scheduler condition -- its - // SuspensionPoint (or its body's own wait) sits in `store.waiting`. - // A genuinely-blocking callee reaches its first real block point without - // anything from the caller, so unlike Fix 1 this wait cannot deadlock: - // it is the reference's atomic run-to-first-block, reconstructed across - // the engine's microtask hops. - // - // THE DEFERRED ENTRY DECISION (issue #43; wasmtime's model — source - // walkthrough distilled on the issue, exam kit archived at - // 4f3351f:exams/wasmtime-exclusivity/). The determinacy wait above is - // also where the initial *status* is decided, so it is where the - // deferral lives. - // - // In wasmtime a guest->guest call queues the callee's `StartImplicit` - // and the caller suspends until the first subtask status event - // (concurrent.rs :3040-3160); the executor first drains the work queued - // ahead of it, so a ready gate holder runs to invocation exit and - // releases `do_not_enter` BEFORE the new call's readiness is evaluated - // (:1497-1522). polyengine's callee thread is likewise already spawned and - // parked at `enter_implicit_thread`'s gate wait at this point; what - // changes here is only WHEN the caller reads `subtask.state`. - // - // Order-robust formulation (issue #43; a non-normative scheduler - // policy — entry-status timing is not normative — chosen over wasmtime's - // FIFO-dependent one so the seeded-shuffle reruns stay green): while the - // callee is still parked at the entry gate, the caller waits until the - // callee instance's runnable work is exhausted - // (`Store.hasRunnableWork`). Then: - // * the holder was ready -> it ran to `exit_implicit_thread`, released - // the gate, the callee entered: `subtask.state` is STARTED (or the - // callee already RETURNED) -- test/async/sync-streams.wast:145; - // * the holder was NOT ready (parked mid-frame on an un-rendezvous'd - // operation), or the holder IS this caller (a nested lower, excluded - // from the scan): quiescence is immediate and the caller reports - // STARTING -- hold semantics, observably. + // Wait for a reportable state, not for completion: an async-lowered caller + // must receive a handle when its callee genuinely blocks. JSPI defers even + // an immediately satisfied Suspending call to a microtask, so reporting + // mid-hop could mislabel eager completion as STARTED (jspi pin (j)). // - // Backpressure-queue admission is untouched: the callee registered in - // `num_waiting_to_enter` synchronously at `thread.resume()` above, - // before any draining, so the deterministic-profile ordering pins - // (async-calls-sync.wast) see the same admission order as before. + // Outside the entry gate, resolution, thread completion, or a genuine + // scheduler park makes the state determinate. At the entry gate we first + // drain the callee instance's runnable work, excluding the caller task: + // a runnable holder may release the gate; a blocked holder or this caller + // cannot. This entry-status timing is scheduler policy, not a spec rule. + // Queue admission already happened synchronously in thread.resume(). // - // PLAIN MODE IS DELIBERATELY UNTOUCHED, and provably needs no drain: a - // needs-exclusive task holds `exclusiveThread` only across a core - // invocation (the callback loop releases it across every wait), and - // without JSPI a wasm frame cannot park mid-invocation at all. So in - // plain mode the gate, when held, is held by the *currently running* - // activation -- the one obstacle a drain can never remove. Zero cost for - // sync-only components, and no suspendability reclassification: - // `async-start-call` was already `Suspending`-wrapped for the - // determinacy park (exec/executor.ts, "async-start-call is wrapped"). + // Plain mode needs no drain: a wasm frame cannot park mid-invocation, + // so any held gate belongs to the currently executing activation. if (ctx.suspensionMode === "jspi") { const store = prepared.callerInst.store; const calleeInst = prepared.calleeInst; @@ -1054,20 +785,9 @@ export function createAsyncStartCall( thread.done() || store.waiting.some((w) => w.task === task); if (!determinate()) { - // Same settle-path enumeration as the sync form above (#102). Here - // the lender scope is the `Subtask` itself, discharged by - // `deliverResolve`, and `report()` is what eventually delivers it — - // either eagerly (the resolved branch) or, for a live subtask, via - // the handle it hands the guest. So the backstop must fire ONLY when - // `report()` did not complete: on the success path the subtask is - // typically still live and in the caller's table, and unwinding it - // there would cancel a perfectly good call. - // - // `report()` not completing means the guest never received the - // subtask index (it either threw before `handles.add`, or after it - // with the index lost), so nothing will ever deliver this subtask's - // resolution — exactly the state `unwindSubtaskLenders` exists for - // (contracts/intrinsics.md §A's trap-unwind/lender-release obligation). + // Unwind only if report() fails or the park is abandoned. A successful + // report may hand the guest a live subtask; its lenders must remain + // registered until the guest observes resolution. let produced = false; return blockCurrentActivation({ store: prepared.callerInst.store, diff --git a/runtime/src/intrinsics/mod.ts b/runtime/src/intrinsics/mod.ts index 0cb78f0..a52c5f7 100644 --- a/runtime/src/intrinsics/mod.ts +++ b/runtime/src/intrinsics/mod.ts @@ -1,18 +1,6 @@ -// Host trampolines (contracts/intrinsics.md §B) and FACT-adapter intrinsic -// obligations (§A) — the core subset, with instantiate-time (never -// call-time) capability-gated failures for everything else. -// -// Implemented from the core subset: -// lower-import host function call through descriptor-IR lift/lower -// trap FACT `Trap` import -> ComponentTrap -// enter/exit-sync-call degenerate sync-call bookkeeping (assert-and-count) -// resource-new/rep/drop sync resource paths over cabi handle tables -// (resources capability, implemented early: the -// resources fixture references them at instantiation) -// -// Everything else fails at instantiate time naming the capability that -// intrinsics.md §B schedules it under — "this component needs the task core" -// is a feature, not a crash. +// Host trampolines and FACT-adapter intrinsics (contracts/intrinsics.md). +// Referenced unsupported kinds fail during instantiation; implemented blocking +// forms use the configured suspension discipline. import { canonResourceDrop, @@ -97,43 +85,27 @@ export * from "./stream_builtins.ts"; /** * Where a host trap thrown *inside* a FACT adapter is remembered. * - * FACT wraps every adapter body in a `try_table … catch_all` exception - * barrier (wasmtime-environ 47.0.3 `fact/trampoline.rs:3939` - * `enter_exception_barrier`) so that a guest exception escaping a component - * becomes a trap rather than unwinding into the caller. In wasmtime a host - * trap unwinds out of band and is unaffected; in a JS host our traps *are* - * JS exceptions, so the barrier swallows them and re-raises the generic - * `UncaughtException` FACT trap — which would violate contracts/intrinsics.md - * §"Universal semantics" 2 ("traps ... must not be catchable by guest code"). + * FACT's `enter_exception_barrier` (`fact/trampoline.rs`) converts escaping + * exceptions to `UncaughtException`. Host traps are JS exceptions too, so we + * remember and restore them to preserve their cause across nested barriers. + * A guest exception with no pending host trap keeps the generic message. * - * The fix is to remember the trap on the way out and restore it when the - * barrier reports `UncaughtException`. A genuine guest exception leaves - * `pending` untouched and keeps the generic trap. - * - * **Residual limitation (inherent to a JS host).** wasmtime's traps are - * unforgeable and uncatchable: they unwind out of band, and no guest - * construct can observe or swallow one. Ours are ordinary JS exceptions, so - * a guest that wraps a call in its own `try_table (catch_all …)` *can* catch - * a host trap mid-flight and continue — the Component Model says that must be - * impossible. Recovering full unforgeability needs an out-of-band channel - * (e.g. a poison flag consulted at every host boundary crossing) and is not - * attempted here; the barrier case above is the one that occurs in practice, - * because FACT emits it on every adapter. Recorded as a known gap. + * Limitation: this preserves diagnostics, not uncatchability. A guest's own + * `try_table catch_all` can catch a host trap and continue, contrary to the + * Component Model's trap semantics. No out-of-band mechanism prevents that. */ export interface HostTrapState { pending: unknown; } /** - * Trap-code → message, from wasmtime-environ 47.0.3 `trap_encoding.rs` + * Trap-code → message, from wasmtime-environ `trap_encoding.rs` * (`generate_trap_type!`), whose ordinals are what FACT passes to the - * `runtime.trap` import. Only the codes a sync FACT adapter can raise are - * listed; anything else falls back to the numeric code. + * `runtime.trap` import. Unlisted codes fall back to the numeric code. * * Rendered with wasmtime's `"wasm trap: "` prefix (its `impl Display for * Trap`), because that is the text the official suite's `assert_trap` - * commands expect for adapter-raised traps - * (e.g. `values/realloc.wast:67,94`). + * commands expect for adapter-raised traps (e.g. `values/realloc.wast`). */ const FACT_TRAP_MESSAGES: Record = { 9: "wasm `unreachable` instruction executed", @@ -154,7 +126,7 @@ const TRAP_UNCAUGHT_EXCEPTION = 49; export { UnsupportedFeatureError } from "./errors.ts"; -/** Capability at which each trampoline kind stops instantiate-failing. */ +/** Diagnostic capability category for unsupported trampoline kinds. */ const TRAMPOLINE_CAPABILITY: Record< string, "core" | "resources" | "task-core" @@ -198,21 +170,16 @@ export class SyncCallScope { readonly lenders: ResourceHandle[] = []; /** - * definitions.py `Subtask.add_lender` (line 890) — note there is **no** - * `own` check, and `lift_borrow` (line 1516) calls it unconditionally: a - * component that received a borrow may lend it onward, and the borrow - * handle's own `num_lends` is what blocks `resource.drop` on it until the - * onward call returns (`canon_resource_drop`, line 2325, traps on - * `num_lends != 0` for owning *and* borrowed handles alike). - * wasmtime 47.0.3 `vm/component/resources.rs:285` (`resource_lift_borrow`) - * agrees. + * definitions.py `Subtask.add_lender`: borrowed handles can be lent onward + * too. `canon_resource_drop` checks `num_lends` for both own and borrow + * handles, so the source remains undroppable until delivery releases it. */ addLender(h: ResourceHandle): void { h.numLends += 1; this.lenders.push(h); } - /** definitions.py `Subtask.deliver_resolve` (lines 902-906): releases lenders at delivery time. */ + /** definitions.py `Subtask.deliver_resolve`: release lenders at delivery. */ releaseLenders(): void { for (const h of this.lenders) h.numLends -= 1; this.lenders.length = 0; @@ -226,15 +193,11 @@ export class SyncCallScope { * `on_start` (the copy adapters cannot block, so the window never suspends). * * Reference mapping (definitions.py): `taskScope` is the callee `Task` — - * `lower_borrow` (line 1821) counts `num_borrows` there, and + * `lower_borrow` counts `num_borrows` there, and * `Task.return_`/`cancel` trap while it is non-zero; `lenders` is the * caller-side `Subtask` (async-start-call) or a plain scope released when * the caller's blocked frame gets its results (sync-start-call) — - * `lift_borrow` (line 1517) adds lenders there, released at - * `deliver_resolve` (line 904). Found by the #18 polymorph-tls smoke: the - * suite is the first corpus with borrow-carrying composed calls through - * prepare/start adapters, which previously asserted "transfer-borrow - * outside an enter-sync-call/exit-sync-call bracket". + * `lift_borrow` adds lenders there, released at `deliver_resolve`. */ export interface FactStartScope { /** The callee task (satisfies cabi's `TaskBorrowScope`). */ @@ -269,8 +232,7 @@ export interface TrampolineContext { * last; see `FactStartScope`). Separate from `syncCallStack` because the * prepare/start protocol has no enter/exit-sync-call bracket — the borrow * bookkeeping attaches to the callee `Task` and the caller-side subtask - * instead (definitions.py `lower_borrow` line 1821 / `lift_borrow` line - * 1517). + * instead (definitions.py `lower_borrow` / `lift_borrow`). */ factStartScopes: FactStartScope[]; /** See `FactCallContext.calleeCanBlock` (intrinsics/fact_calls.ts). */ @@ -292,15 +254,15 @@ export interface TrampolineContext { prepared: { current: PreparedCall | null }; /** Suspension discipline (jspi/bridge.ts). */ suspensionMode: import("../jspi/mod.ts").SuspensionMode; - /** Element types of the plan v2 stream/future tables. */ + /** Element types of the plan's stream/future tables. */ streamElem(index: number): import("../cabi/types.ts").ValType | null; futureElem(index: number): import("../cabi/types.ts").ValType | null; streamTableInstance(index: number): ComponentInstanceState; futureTableInstance(index: number): ComponentInstanceState; /** * The component instance owning error-context table `index` - * (`TypeComponentLocalErrorContextTableIndex`, plan v3 `errorContextTables`). - * Its own index space — NOT the resource-table one it used to borrow. + * (`TypeComponentLocalErrorContextTableIndex`, `errorContextTables`). + * This index space is distinct from resource-table indices. */ errorContextTableInstance(index: number): ComponentInstanceState; /** @@ -331,15 +293,14 @@ interface ResourceTrampolineDecl { * initializer/arg/export resolution — i.e. at instantiate time — so an * unsupported kind fails instantiation, not the first call * (plan-format.md "Executor obligations"). Unreferenced trampolines are - * never created and therefore never fail (intrinsics.md §B tolerates e.g. - * an unreferenced task-return until the task core exists). + * never created and therefore never fail. */ export function createTrampoline( decl: WireTrampoline, ctx: TrampolineContext, ): CoreFn { const fn = createTrampolineBody(decl, ctx); - // Remember host traps so the FACT exception barrier cannot swallow them + // Preserve host-trap diagnostics across the FACT exception barrier // (see `HostTrapState`). This wraps the `trap` trampoline too, which is // what keeps a specific trap specific across *nested* adapters: the inner // barrier's `trap` trampoline restores and rethrows the real trap, this @@ -355,18 +316,12 @@ export function createTrampoline( }; } -/** - * The component instance a trampoline is declared in (wasmtime names it in - * every instance-scoped `Trampoline` variant). This is the static answer to - * definitions.py's `current_instance()`, and unlike it, it is defined during - * instantiation — when a core module's start function may already be calling - * these built-ins. See the header of ./async_builtins.ts. - */ /** Narrow the trampoline context to what the stream built-ins need. */ function sctx(ctx: TrampolineContext): StreamTrampolineContext { return ctx as unknown as StreamTrampolineContext; } +/** Static instance identity, available even to instantiation-time start functions. */ function declaredInstance( decl: WireTrampoline, ctx: TrampolineContext, @@ -379,12 +334,6 @@ function declaredInstance( return ctx.componentInstance(instance); } -/** - * The FACT sync-call bracket stack in force right now: the running task's - * (activations interleave since background activations exist, so a single - * executor-wide stack is not a stack), or the executor's when no task is - * running — instantiation-time start functions. - */ // deno-lint-ignore no-explicit-any const SCOPE_TRACE = (() => { try { @@ -394,6 +343,10 @@ const SCOPE_TRACE = (() => { } })(); +/** + * Brackets belong to the running thread because activations can interleave. + * Instantiation-time start functions have no thread and use the executor stack. + */ function syncScopes(ctx: TrampolineContext, site = "?"): any[] { const thread = maybeCurrentThread() as | { syncCallStack: any[] } @@ -446,12 +399,7 @@ function createTrampolineBody( }; } - // Sync-call task bookkeeping (intrinsics.md §A) — assert-and-count. - // wasmtime 47 signatures: - // enter-sync-call carries the caller/callee instance pair, which is what - // the reentrance gate below needs; balance of the bracket is asserted at - // component teardown by tests. - // Signatures (wasmtime-environ 47.0.3 `fact.rs:743,754`): + // FACT sync-call borrow brackets (wasmtime-environ `fact.rs`): // async.enter-sync-call(caller_instance: i32, async: i32, // callee_instance: i32) -> () // async.exit-sync-call() -> () @@ -461,24 +409,8 @@ function createTrampolineBody( async_?: number, calleeInstance?: number, ) => { - // ENTRY REFUSAL at the fused sync-call boundary. - // - // The reference has no reentrance gate (CM#705; definitions.py @ - // 2f13265 has no `may_enter`/`entering_set`/`enter_from`): a - // guest->guest call through `Store.lift` runs `canon_lift` - // unconditionally, and host-mediated reentrance — host -> A.f -> C.g - // -> host import -> host invokes C.g — is simply valid. wasmtime's - // fused adapters agreed all along: `enter_guest_sync_call` - // (47.0.3 `runtime/component/concurrent.rs:1723`) performs no - // reentrance check, and `fact/trampoline.rs:120-127` decides the - // caller==callee / ancestor pairs statically at compile time (what - // `test/async/trap-on-reenter.wast` cases 2 and 3 pin — a translation - // -time trap, not this site). - // - // What this site does check is polyengine's per-instance poisoning: a - // callee that trapped is a corpse and may never be entered again, - // and the refusal names the original trap (polyengine#145). That is - // the whole content of this check. + // Reentrance is valid. `entryRefusal` enforces per-instance poisoning, + // a runtime divergence, and reports the original trap. if ( typeof callerInstance === "number" && typeof calleeInstance === "number" @@ -492,27 +424,12 @@ function createTrampolineBody( ); if (refusal !== null) trap(refusal); } - // `async_` records whether the callee is *async-lifted*. wasmtime - // stores it on the guest task it creates here - // (`concurrent.rs:1723` `enter_guest_sync_call`, whose `callee_async` - // parameter flows into `GuestTask::new`) and never traps on it. - // - // Before the task core existed this trampoline refused `async_ == 1` - // rather than silently treating an async callee as sync. That guard is - // now stale and actively wrong: a sync-lowered caller reaching an - // async-lifted export is the `sync-start-call` path - // (intrinsics/fact_calls.ts), and the task it needs is created by - // `prepare-call`, not here. What remains of this bracket for us is the - // borrow bookkeeping (`SyncCallScope`), which applies either way. + // This bracket manages borrows regardless of the callee's asyncness; + // the prepare/start protocol creates any separate callee task. void async_; ctx.stats.enterSyncCalls++; - // Per task where there is one; the executor-wide stack is the - // fallback for a start function running at instantiation time, which - // has no task (see `maybeCurrentTask`). - // Invariant, per ACTIVATION: every `enter` is matched by exactly one - // `exit` on the same stack. Recorded here so the `exit` side can - // assert it structurally rather than only by depth (CE_SCOPE_TRACE - // proved this is where the interesting failures live). + // Normal return must match enter/exit on the same activation's stack; + // trap unwind releases any scopes whose exit was skipped. const scopes = syncScopes(ctx, "enter"); scopes.push(new SyncCallScope()); }; @@ -527,14 +444,11 @@ function createTrampolineBody( assert_( scope !== undefined, // If this fires, an `exit` reached an activation that never ran the - // matching `enter` -- the bracket is attached to the wrong unit - // again. See `Thread.syncCallStack`. + // matching `enter`. See `Thread.syncCallStack`. "exit-sync-call with an empty sync-call stack", ); // definitions.py `Task.return_`: the callee may not return while it - // still holds borrow handles. Wording parity with wasmtime's - // exit-time check (drop-cross-task-borrow.wast:309 pins the async - // path; the sync bracket is the same check). + // still holds borrow handles. trapIf( scope!.numBorrows > 0, "borrow handles still remain at the end of the call", @@ -542,8 +456,7 @@ function createTrampolineBody( scope!.releaseLenders(); }; - // Guest-side resource built-ins (sync paths of docs/architecture.md §7 over the cabi - // handle tables). rep is always i32 in current wasmtime. + // Guest-side resource built-ins; reps and handle indices are i32. case "resource-new": { const d = decl as unknown as ResourceTrampolineDecl; const inst = ctx.componentInstance(d.instance); @@ -565,12 +478,6 @@ function createTrampolineBody( }; } - // FACT resource transfer (contracts/intrinsics.md §A, wasmtime-environ - // 47.0.3 `fact.rs:721` — signature `(i32 src_handle, i32 src_table, - // i32 dst_table) -> i32 dst_handle`). These are the fused-adapter form of - // `lift_own`/`lower_own` and `lift_borrow`/`lower_borrow` - // (definitions.py) with the src/dst tables named by index rather than - // implied by the running instance. // FACT string transcoders (contracts/intrinsics.md §B). The plan // carries the op name plus the source/destination `RuntimeMemoryIndex`es; // `./transcode.ts` holds the twelve operations. @@ -604,8 +511,7 @@ function createTrampolineBody( } // --- 0.3 async built-ins (contracts/intrinsics.md §B) ------------- - // All ported in ./async_builtins.ts; the ones that would have to block a - // wasm frame fail there, at the call site, with a JSPI-shaped message. + // Blocking forms require JSPI when they cannot complete immediately. case "task-return": return createTaskReturn( decl as unknown as { @@ -617,13 +523,6 @@ function createTrampolineBody( ); case "task-cancel": return createTaskCancel(); - // No `backpressure-set` case on purpose: wasmtime-environ 47.0.3 has only - // `Trampoline::BackpressureInc` / `BackpressureDec` - // (`component/info.rs:775,781`) — there is no `BackpressureSet` variant to - // dispatch, so a case for it would be unreachable code implying a wire - // shape that cannot occur. definitions.py's own dead - // `canon_backpressure_set` was removed upstream (CM PR #690); see - // upstream-component-model-repo-findings.md CM-2, RESOLVED. case "backpressure-inc": return createBackpressureInc(declaredInstance(decl, ctx)); case "backpressure-dec": @@ -787,13 +686,8 @@ function createTrampolineBody( case "error-context-transfer": return createErrorContextTransfer( ctx as unknown as AsyncTransferContext, - // plan v3: the transfer's table arguments are - // `TypeComponentLocalErrorContextTableIndex`es (fact/trampoline.rs: - // 3526-3539), resolved through the plan's own `errorContextTables` - // section. Before v3 this went through `resourceTableInstance` — a - // different index space, which mis-routed silently whenever a - // concrete resource table happened to exist at the colliding slot - // (polyengine#89). + // Table arguments are `TypeComponentLocalErrorContextTableIndex`es, + // not resource-table indices. (t) => ctx.errorContextTableInstance(t), ); @@ -852,25 +746,11 @@ function transferOwn( /** * `lift_borrow` from the source table followed by `lower_borrow` into the - * destination table. The source handle stays in place; the destination gets a - * non-owning handle. - * - * Two deviations from the plain lift/lower pair, both taken from - * definitions.py: - * - * - `lower_borrow` returns the *rep* directly when the destination instance - * is the one that implements the resource ("own the resource" fast path), - * since a component always has direct access to its own reps. - * - lender / `num_borrows` bookkeeping is attached to the enclosing - * `SyncCallScope` (the `enter-sync-call` / `exit-sync-call` bracket), - * which is this path's stand-in for the callee `Subtask`/`Task` of - * definitions.py. + * destination table. The source handle stays in place. The implementing + * instance receives the rep directly; other destinations get a non-owning + * handle. Lenders and borrow counts attach to the FACT start window or sync + * bracket that represents the reference's Subtask/Task for this call. */ -// CONTRACT: contracts/intrinsics.md §A describes ResourceTransfer* only as -// "handle-table moves between component instances" — the borrow-scope -// interaction beyond the lender registration is taken from definitions.py -// (`lift_borrow`/`lower_borrow` + `Subtask.lenders`/`Task.num_borrows`) and -// is what makes `test/resources/borrows.wast:162` (`lend-trap`) trap. function transferBorrow( ctx: TrampolineContext, handle: number, diff --git a/runtime/src/intrinsics/stream_builtins.ts b/runtime/src/intrinsics/stream_builtins.ts index a62960c..f2d593a 100644 --- a/runtime/src/intrinsics/stream_builtins.ts +++ b/runtime/src/intrinsics/stream_builtins.ts @@ -1,6 +1,5 @@ // The stream / future / error-context canonical built-ins -// (definitions.py `canon_stream_new` line 2504 through `canon_error_context_drop` -// line 2803). +// (definitions.py `canon_stream_new` through `canon_error_context_drop`). // // The copy built-ins all share one shape, which is worth stating once: // @@ -14,10 +13,8 @@ // *blocked* — `BLOCKED` for the async form, and for the sync form a // genuine wasm-frame block, i.e. JSPI. // -// Step 4 is where the reference's `e.wait_for_pending_event()` sits. A -// stackless runtime cannot do that, so the sync form returns its answer when -// the rendezvous already happened and reports `NeedsJspi` otherwise — the same -// rule already applied to `waitable-set.wait` and `sync-start-call`. +// Step 4 implements `e.wait_for_pending_event()`: the sync form requires JSPI +// only when no event is ready. Plain mode then reports `NeedsJspi`. import { blockCurrentActivation } from "../jspi/mod.ts"; import type { SuspensionMode } from "../jspi/mod.ts"; @@ -58,10 +55,7 @@ import { BLOCKED } from "./async_builtins.ts"; import { removeHandleWithUnwind } from "../task/scheduler.ts"; /** - * Standing probe (CE_COPY_TRACE=1): per-call return codes of the copy / - * cancel built-ins, in both modes. This is the plain-vs-jspi differential's - * data source — a divergence list of packed codes beats stack traces when the - * guest asserts exact `expect-code` values (big-interleaving-test.wast). + * CE_COPY_TRACE=1 logs copy/cancel return codes for plain-vs-JSPI diagnostics. */ const COPY_TRACE = (() => { try { @@ -79,11 +73,11 @@ export function traceCopy(msg: string): void { export interface StreamTrampolineContext { componentInstance(index: number): ComponentInstanceState; options(index: number): ResolvedOptions; - /** Element type of a `TypeStreamTableIndex` (plan v2 `streamTables`). */ + /** Element type of a `TypeStreamTableIndex` (`streamTables`). */ streamElem(index: number): ValType | null; - /** Element type of a `TypeFutureTableIndex` (plan v2 `futureTables`). */ + /** Element type of a `TypeFutureTableIndex` (`futureTables`). */ futureElem(index: number): ValType | null; - /** Suspension discipline; decides whether site 4 blocks or signals. */ + /** Suspension discipline for sync copy/cancel waits. */ suspensionMode?: SuspensionMode; } @@ -92,7 +86,7 @@ export interface StreamTrampolineContext { // --------------------------------------------------------------------------- /** - * definitions.py `canon_stream_new` (line 2504) / `canon_future_new` (2512). + * definitions.py `canon_stream_new` / `canon_future_new`. * Returns both handles packed into an i64: `ri | (wi << 32)`. */ export function createStreamNew( @@ -134,7 +128,7 @@ function packEnds(ri: number, wi: number): bigint { type EndCtor = new (shared: never) => CopyEnd; -/** definitions.py `stream_copy` (line 2530). */ +/** definitions.py `stream_copy`. */ function streamCopy(input: { EndT: EndCtor; reading: boolean; @@ -176,10 +170,7 @@ function streamCopy(input: { const cx = new LiftLowerContext(cabiOptions(opts), inst, null); const buffer = new GuestBuffer(elem, cx, ptr, n); - // definitions.py `stream_copy`: `assert(not isinstance(stream_t, CharType))` - // — plan validation is expected to reject a char-typed stream before this - // point (streams of `char` are not a representable component type), so this - // documents that invariant rather than defending against a reachable case. + // This implementation's copy path does not accept a char element. assert_(elem === null || elem.kind !== "char", "stream copy: char element"); // definitions.py `stream_event`: the payload is computed at *delivery* time, @@ -196,8 +187,7 @@ function streamCopy(input: { buffer.progress <= BUFFER_MAX_LENGTH, "stream progress out of packing range", ); - // definitions.py `stream_copy`/`stream_event`: `assert(0 <= result < 2**4)`. - // `CopyResult` is a fixed 0..2 enum so this can't fire; kept for parity. + // Low four bits hold the result; the remaining bits count elements. assert_( result >= 0 && result < 2 ** 4, "stream event: packed result out of 4-bit range", @@ -223,7 +213,7 @@ function streamCopy(input: { // future.{read,write} // --------------------------------------------------------------------------- -/** definitions.py `future_copy` (line 2584). */ +/** definitions.py `future_copy`. */ function futureCopy(input: { EndT: EndCtor; reading: boolean; @@ -242,11 +232,7 @@ function futureCopy(input: { trapIf(!(e instanceof EndT), "future copy: wrong end type for this handle"); const end = e as ReadableFutureEnd | WritableFutureEnd; trapIf(!sameElem(end.shared.t, elem), "future copy: element type mismatch"); - // The writable side reaches DONE by two different routes — its own write - // completed, or it was notified the readable end dropped — and wasmtime's - // message names both (`futures_and_streams.rs:3522`). The shorter text the - // other two assertions in `trap-if-done.wast` use (:446, :448) is a prefix - // of this one, so the single full string satisfies all three. + // Writable DONE covers either a completed write or a dropped readable end. trapIf( end.state === CopyState.DONE, reading @@ -280,20 +266,11 @@ function futureCopy(input: { end.state = CopyState.COPYING; const onCopyDone = (result: CopyResult) => { - // #84/#90: an unwritten future whose writable side was torn down (a - // trap-poisoned instance's table, or the host's `drop()` door) can never - // satisfy this reader. definitions.py keeps that state unreachable - // (:1183-1184 traps the early writable drop, so :2607 may assert a - // readable end never sees DROPPED); where we bypass the trap we owe the - // reader a *trap at its rendezvous point* instead of a DROPPED answer. - // - // The pending event stays a thunk, so the trap is raised exactly where - // the reader observes it: `waitable-set.wait`'s delivery (both the - // fast-path and the JSPI `produce`, intrinsics/async_builtins.ts:290/311), - // the callback loop's `waitForEventAnd` (exec/boundary.ts:1766), and - // `finishCopy`'s `take()` below — every one of which is inside the - // reader's guest activation, so the throw propagates as that task's trap - // and poisons *its* instance, and nothing else. + // Host abandonment or instance poisoning can tear down an unwritten + // future, unlike the reference's ordinary writable-drop path, which + // traps. Report a trap when the reader observes its event, never a + // readable DROPPED result. The thunk keeps that trap in the reader's + // activation whether delivery is through a waitable set or finishCopy. const abandoned = reading && result === CopyResult.DROPPED ? abandonReasonOf(end.shared) : null; @@ -344,7 +321,7 @@ function finishCopy( // definitions.py `e.wait_for_pending_event()`: block this wasm frame // until the other end shows up. if (mode === "jspi" && inst !== undefined) { - // SITE 4 (lit). `hasSyncWaiter` marks the end as having a blocked + // `hasSyncWaiter` marks the end as having a blocked // synchronous reader/writer, which is what makes a concurrent // `cancel-copy` on it a trap (see `cancelCopy`). Setting it only now // is correct: before this point nothing was actually waiting. @@ -361,11 +338,7 @@ function finishCopy( traceCopy(`${what} sync copy i=${i} RESUME -> 0x${p.toString(16)}`); return p; }, - // #106: `abandon` never runs `produce`; without the backstop the - // flag stayed set forever and a later `cancel-copy` on this end - // trapped "sync waiter" against a waiter that no longer exists. - // Idempotent, so the success path's clear-before-`take` ordering - // inside `produce` is untouched. + // Abandonment skips produce; clear the claim on that path too. onSettled: () => { end.hasSyncWaiter = false; }, @@ -390,8 +363,7 @@ function finishCopy( /** * `cancel_copy`'s reporting tail, shared by its immediate and blocking exits. - * Kept verbatim (including the wasmtime divergence below) so the blocking form - * cannot drift from the non-blocking one. + * Both exits apply the CM-3 completion-superseding exception below. */ function takeCancelEvent( end: CopyEnd, @@ -404,24 +376,13 @@ function takeCancelEvent( !end.copying() && code === eventCode && index === i, `unexpected event delivered by ${what}`, ); - // UPSTREAM DIVERGENCE (definitions.py is wrong here; wasmtime is right). - // - // `cancel_copy` (definitions.py line 2654) returns an already-armed pending - // event verbatim, so cancelling a stream write that had been partially - // satisfied yields COMPLETED with the copied count. wasmtime instead - // *supersedes* an undelivered stream COMPLETED with CANCELLED, keeping the - // count (`futures_and_streams.rs:4004-4015`): - // - // (ReturnCode::Completed(count), Event::StreamWrite { .. }) - // => ReturnCode::Cancelled(count), - // (ReturnCode::Dropped(_) | ReturnCode::Completed(_), _) => code, - // - // and `test/async/big-interleaving-test.wast:1526-1531` asserts wasmtime's - // answer (0x42 = CANCELLED | 4<<4, not 0x40). The reasoning is sound: the - // guest never observed the completion, so reporting it as completed would - // lose the fact that the operation was cancelled. Note the two exclusions - // encoded below — DROPPED keeps its code, and a *future* COMPLETED keeps - // its code (only `Event::Stream{Read,Write}` is converted). + // CM-3 exception (upstream-component-model-repo-findings.md): adopt the + // corpus/wasmtime semantics pending upstream adjudication, rather than + // definitions.py `cancel_copy`'s verbatim pending event. An undelivered + // stream COMPLETED becomes CANCELLED with the same element count; + // DROPPED and future COMPLETED remain unchanged. See + // `test/async/big-interleaving-test.wast` and wasmtime's + // `futures_and_streams.rs` cancellation handling; docs/architecture.md §1. const isStreamEvent = eventCode === EventCode.STREAM_READ || eventCode === EventCode.STREAM_WRITE; if (isStreamEvent && (payload & 0xf) === CopyResult.COMPLETED) { @@ -433,7 +394,7 @@ function takeCancelEvent( return payload; } -/** definitions.py `cancel_copy` (line 2636). */ +/** definitions.py `cancel_copy`, with the CM-3 exception in takeCancelEvent. */ function cancelCopy(input: { EndT: EndCtor; eventCode: EventCode; @@ -465,7 +426,7 @@ function cancelCopy(input: { if (!end.hasPendingEvent()) { if (!async_) { if (mode === "jspi") { - // SITE 4b (lit): definitions.py `cancel_copy` blocks until the + // definitions.py `cancel_copy` blocks until the // cancellation settles, then reports through the same tail. return blockCurrentActivation({ store: inst.store, @@ -490,7 +451,7 @@ function cancelCopy(input: { // drop-{readable,writable} // --------------------------------------------------------------------------- -/** definitions.py `drop` (line 2670). */ +/** definitions.py `drop`. */ function dropEnd( EndT: EndCtor, elem: ValType | null, @@ -498,7 +459,7 @@ function dropEnd( hi: number, what: string, ): void { - // Guest-supplied index is u32; core wasm delivers i32 args signed (F3, R2). + // Guest-supplied index is u32; core wasm delivers i32 args signed. hi = hi >>> 0; trapIf(!inst.mayLeave, `${what}: cannot leave component instance`); removeHandleWithUnwind(inst, hi, (e) => { @@ -514,13 +475,12 @@ function dropEnd( // --------------------------------------------------------------------------- /** - * definitions.py `canon_error_context_new` (line 2778). + * definitions.py `canon_error_context_new`. * * The reference is deliberately non-committal about the message: under * `DETERMINISTIC_PROFILE` it stores the empty string, otherwise it may apply a * `host_defined_transformation`. We keep the guest's message verbatim — the - * most useful behaviour for a debugging aid, and within what the spec allows - * (the message is explicitly not semantically load-bearing). + * diagnostic policy, within what the spec allows outside that profile. */ export function createErrorContextNew( decl: { options: number }, @@ -541,7 +501,7 @@ export function createErrorContextNew( }; } -/** definitions.py `canon_error_context_debug_message` (line 2792). */ +/** definitions.py `canon_error_context_debug_message`. */ export function createErrorContextDebugMessage( decl: { options: number }, ctx: StreamTrampolineContext, @@ -565,7 +525,7 @@ export function createErrorContextDebugMessage( }; } -/** definitions.py `canon_error_context_drop` (line 2803). */ +/** definitions.py `canon_error_context_drop`. */ export function createErrorContextDrop( inst: ComponentInstanceState, ): CoreFn { @@ -823,10 +783,9 @@ void currentTask; // --------------------------------------------------------------------------- // // The fused-adapter form of `lift_async_value` + `lower_stream`/`lower_future` -// (definitions.py lines 1530 / 1828), with the source and destination tables +// in definitions.py, with the source and destination tables // named by index rather than implied by the running instance — exactly the -// arrangement `resource.transfer-own` already uses. Signature (wasmtime -// `vm/component/libcalls.rs:567,576,585`): +// arrangement `resource.transfer-own` uses. Signature: // (src_idx: i32, src_table: i32, dst_table: i32) -> i32 dst_idx // // Transferring moves the *readable* end: the writable end, if this component @@ -919,15 +878,13 @@ export function createFutureTransfer(ctx: AsyncTransferContext): CoreFn { /** * error-context transfer. Unlike stream/future ends, an `error-context` is - * shareable: `lift_error_context` (definitions.py line 1451) *reads* the handle + * shareable: definitions.py `lift_error_context` reads the handle * rather than removing it, so the source keeps its own. - */ -/** + * * `instanceOf` resolves through the plan's `errorContextTables` section * (contracts/plan-format.md schema) — the * `TypeComponentLocalErrorContextTableIndex` space these arguments actually - * live in. It replaced a resource-table lookup, which shared neither the - * index space nor (in a multi-instance composition) the answer. + * live in, distinct from resource-table indices. * * The arguments are the trampoline's own core parameters, so a missing one is * an arity fault, not a zero: no `?? 0` defaults — `instanceOf(undefined!)` diff --git a/runtime/src/intrinsics/transcode.ts b/runtime/src/intrinsics/transcode.ts index 77b67ff..4193fe3 100644 --- a/runtime/src/intrinsics/transcode.ts +++ b/runtime/src/intrinsics/transcode.ts @@ -7,12 +7,12 @@ // pointers into two linear memories and expects them to do the encoding work // in place; the surrounding allocation/realloc dance stays in the adapter. // -// Authorities used, in order: -// - signatures: wasmtime-environ 47.0.3 `fact/transcode.rs` (`Transcoder::ty`) +// FACT protocol sources: +// - signatures: wasmtime-environ `fact/transcode.rs` (`Transcoder::ty`) // - call protocol (argument order, multi-pass retries, what each result is // used for): `fact/trampoline.rs` `string_copy` / `string_deflate_to_utf8` // / `string_to_utf16` / `string_utf16_to_compact` / `string_to_compact` -// - operation semantics: wasmtime 47.0.3 +// - partial-progress operations: wasmtime // `runtime/vm/component/libcalls.rs` (the twelve `Transcode` libcalls) — // the executable reference for this layer, since definitions.py models // transcoding as whole-string `store_string_*` rather than as these @@ -215,29 +215,18 @@ function inflateLatin1Bytes( } /** - * Guard against the one case where reading and writing through the same - * `Uint8Array` would corrupt data: FACT freshly allocates every destination, - * so source and destination never overlap, but they *can* live in the same - * memory. Callers that read and write interleaved snapshot the source first. + * Snapshot a byte range before writing to a potentially shared memory. + * FACT allocates destination regions separately; a snapshot also prevents + * writes from changing unread source bytes if those regions overlap. */ function snapshot(bytes: Uint8Array, ptr: number, len: number): Uint8Array { return bytes.slice(ptr, ptr + len); } /** - * O(1) defensive counterpart to wasmtime's `assert_no_overlap` - * (libcalls.rs:166-177): traps (does not merely assert) because this - * replaces a guarantee FACT's trampoline construction is supposed to - * provide — src/dst are always independently-allocated regions — so a hit - * here means that guarantee broke, which is guest-memory-corruption-class - * severity, not an internal invariant a caller controls. - * - * Applied only where a call reads and writes through the SAME backing - * `Uint8Array` while interleaving reads and writes (byte-range comparison, - * not per-element — O(1) per call). Ops that first `snapshot()` the source - * into an independent copy (transcode.ts's `snapshot`, used by every op - * above that decodes-then-writes) already break aliasing before the first - * write, so they are exempt by construction and do not call this. + * Defensive byte-range overlap trap for the non-snapshot operation, analogous + * to wasmtime's `assert_no_overlap`. Other operations snapshot source bytes + * before writing. This is an explicit Trap, not a host-precondition assertion. */ function trapIfOverlap( src: Uint8Array, @@ -274,16 +263,8 @@ export function createTranscoder( to: TranscodeMemory, ): (...args: number[]) => unknown { const fn = createTranscoderInner(op, from, to); - // Core wasm delivers i32 params to a JS import as *signed* numbers, but - // every transcoder arg (ptr/len/flag) is a FACT-validated unsigned - // quantity (contracts/intrinsics.md §A/§B; wasmtime libcalls.rs takes - // unsigned guest pointers, and validate_guest_pointer does its bounds - // arithmetic unsigned). Normalize once here, at the single call boundary - // every one of the twelve arms shares, so none of them need to know about - // the signed/unsigned wasm calling-convention detail — matching the - // `>>> 0` normalization every other i32-taking intrinsic gets (see - // intrinsics/mod.ts resource-new/-rep/-drop, resource-transfer-*, and - // exec/boundary.ts). + // Core i32 arguments arrive signed; FACT pointers, lengths, and flags are + // unsigned. Normalize once before any address or capacity arithmetic. return (...args: number[]) => fn(...args.map((a) => a >>> 0)); } @@ -385,16 +366,8 @@ function createTranscoderInner( return (srcPtr, srcLen, dstPtr) => { const src = from.bytes(); const dst = to.bytes(); - // This op does not call `snapshot()` (unlike its siblings above): - // it reads the full `out` prefix before writing anything to `dst`, - // which is the same aliasing-safety property snapshot() buys - // elsewhere, just via a builder array instead of a byte copy. The - // overlap guard is still added here (O(1): a byte-range compare, not - // per-element) as the one op in this file that is safe by algorithm - // shape rather than by an explicit `snapshot()` call — cheap - // insurance against that reasoning becoming stale under a future - // edit (docs/architecture.md §7; wasmtime asserts overlap on every - // op unconditionally, libcalls.rs:166-177). + // This arm reads its latin1 prefix into `out` before writing, rather + // than snapshotting source bytes. It also explicitly rejects overlap. trapIfOverlap(src, srcPtr, 2 * srcLen, dst, dstPtr, srcLen); const view = new DataView(src.buffer, src.byteOffset, src.byteLength); // Note: no surrogate validation here, matching wasmtime — a surrogate @@ -464,15 +437,10 @@ function createTranscoderInner( const dst = to.bytes(); inflateLatin1Bytes(dst, dstPtr, latin1Bytes); const view = new DataView(dst.buffer, dst.byteOffset, dst.byteLength); - // Defensive dst-capacity guard: wasmtime's equivalent - // (`run_utf8_to_utf16`'s `.zip(dst)`, libcalls.rs:308-312) is bounded - // by Rust's `Iterator::zip` truncating to the shorter of the two — - // it can never overrun `dst`. FACT is supposed to size `dstLen` to - // always have room (a full re-encode of a string that was already - // partially latin1-encoded never needs more u16 units than - // `dstLen - latin1Bytes`), so this should be unreachable; trap - // rather than let a broken caller corrupt guest memory past `dst`'s - // bound or silently truncate. + // dstLen is total UTF-16 capacity in code units. The widened latin1 + // prefix occupies latin1Bytes units; the decoded suffix must fit in + // the remainder. Trap rather than overrun or silently truncate if + // the FACT caller supplies insufficient capacity. const capacity = dstLen - latin1Bytes; if (s.length > capacity) { trap("utf8-to-compact-utf16: destination capacity exceeded"); diff --git a/runtime/src/jspi/bridge.ts b/runtime/src/jspi/bridge.ts index 0578548..c04f225 100644 --- a/runtime/src/jspi/bridge.ts +++ b/runtime/src/jspi/bridge.ts @@ -1,12 +1,8 @@ // The JSPI ↔ scheduler bridge: turning a blocking canonical built-in into a // genuinely suspended wasm activation. // -// This is the phase-1 seam swap. Nothing in the task model changes: `Thread`, -// `BlockRequest` and `waitUntil` are exactly as they were. What changes is -// *who* the parked thing is. For the stackless (callback-ABI) path a parked -// thread is a JS generator; for a stackful one it is a suspended wasm -// activation, and this module is the adapter that makes the two look the same -// to `Store.tick`. +// Generator waits and suspended wasm frames share Store.tick's readiness +// interface. The latter resume when their import Promise is settled. // // =========================================================================== // THE INVARIANT @@ -16,7 +12,7 @@ // `promising`-wrapped and no import is `Suspending`-wrapped — OR // "suspension-capable": every entry that can reach a blocking built-in is // `promising`-wrapped, and every blocking built-in is `Suspending`-wrapped. -// Never a mixture. +// Explicit plain sync entries cannot use suspension-capable imports. // // This is forced by empirical fact (c), pinned in // `runtime/tests/jspi/suspending_import_test.ts`: **a `Suspending`-wrapped @@ -32,14 +28,13 @@ // so a mismatch fails loudly at instantiate time rather than as a mystery // trap much later. // -// Which entries "can reach a blocking built-in"? Exactly three: +// Suspension-capable entries include: // * a lifted export's core function, // * a callback export (the callback-ABI loop re-enters wasm), // * a FACT adapter callee invoked by `{sync,async}-start-call`. -// `realloc`, `post-return` and resource destructors are deliberately NOT -// promising-wrapped: they are guest-internal or spec-forbidden from blocking, -// they never call a canonical built-in, and wrapping them would force their -// results to become Promises where cabi needs a number synchronously. +// Host-initiated destructors may also use promising entry. Nested guest +// destructors use a plain synchronous lift and cannot suspend through their +// JS trampoline frame. Realloc and post-return remain synchronous. import { assert_ } from "../cabi/trap.ts"; import { @@ -196,9 +191,8 @@ export function planNeedsSuspension(plan: { /** * Wrap a JS→wasm entry according to the mode. * - * In `plain` mode this is the identity. In `jspi` mode the returned function - * always yields a Promise (empirical fact (e)), which is why the mode is an - * embedder opt-in: it changes the shape of every lifted export. + * In plain mode this is the identity; in JSPI mode it always returns a + * Promise. Mode may be explicitly requested or inferred from the plan. */ export function enterWasm unknown>( fn: T, @@ -238,46 +232,12 @@ export function enterWasm unknown>( // Continuation-chunk attribution sentinels (issue #24) // --------------------------------------------------------------------------- // -// PROBLEM. Engine continuation chunks — the segments of a promising wasm -// activation between suspension/hop points — begin as promise REACTIONS, -// with no synchronous signal to this runtime. When several activations have -// pending continuations (a settled real suspension racing a fast-path hop, -// or two fast-path hops from nested entries), the chunks interleave at an -// empty bracket stack, and every ambient read in a later chunk — a hop's -// `owner` capture at `claimingFn` entry, or an unsafe intrinsic like -// `context.set`, which has no hop at all — inherits whatever claim the -// previous chunk left on top. Claim-stack ordering alone cannot repair -// this: the release edges are themselves promise reactions. Measured -// consequence (issue #24): wit-bindgen's callback epilogue restored one -// task's state pointer into another thread's context slots, and the next -// invocation of the starved thread's callback hit -// `assert!(!state.is_null())` (async_support.rs:578) -> unreachable. -// Reachable only with enough concurrently-suspended sibling activations -// (first corpus: polymorph-tls' webcrypto-composed suite, three async -// wit-bindgen components deep). -// -// FIX. Exploit the one ordering guarantee the platform does give us: -// microtasks run FIFO, and between our code queueing a microtask and the -// engine queueing the continuation reaction there is only synchronous -// engine-internal promise machinery. So at EVERY point where an engine -// continuation is about to be queued, queue a SENTINEL first that claims -// the chunk's owner (move-to-top): -// -// * fast-path hop: sentinel queued synchronously in `claimingFn` before -// returning the plain value — the engine queues the hop reaction while -// processing that return, so the queue reads [sentinel, chunk]. -// * genuine suspension: the wrapper attached to the import's thenable -// queues the sentinel inside the settle reaction, before returning the -// value — the engine (attached to the WRAPPED promise) queues the -// resumption when that wrapper returns, so again [sentinel, chunk]. -// This holds even when several promises settle in one drain: each -// pair is queued contiguously from within its own settle reaction. -// -// Nothing is delayed or reordered — unlike a serializing gate, which -// measurably shifted the deterministic-profile backpressure-admission -// order (async-calls-sync.wast caught it). This is the JSPI substitute for -// what fibers give wasmtime for free: identity travels with the -// resumption, here as a claim planted one microtask ahead of it. +// Engine continuation chunks can run after synchronous ambient brackets unwind. +// Queue an owner-claim sentinel before returning a plain value, or from the +// thenable's settlement reaction before the wrapped Promise settles. Promise +// reactions are FIFO, but engine continuation timing can interleave other +// claims between the sentinel and wasm chunk; adjacency is not guaranteed. +// Instance-scoped ambient lookup also filters sibling-instance candidates. function sentinelFor(owner: unknown): void { if (owner === null || owner === undefined) return; @@ -288,8 +248,9 @@ function sentinelFor(owner: unknown): void { } const SENTINEL_TICK = Promise.resolve(); -/** Wrap a suspending import's thenable so the eventual resumption chunk is - * preceded contiguously by its attribution sentinel. */ +/** Queue attribution before the wrapped Promise settles. Engine continuation + * timing need not make the sentinel and wasm chunk adjacent; instance-scoped + * ambient lookup also filters sibling-instance claims. */ function attributeContinuation( owner: unknown, r: PromiseLike, @@ -335,13 +296,9 @@ export function suspendingImport unknown>( throw e; } if (r === null || typeof (r as { then?: unknown })?.then !== "function") { - // Fast path (jspi pin (j)): the value still returns to wasm through an - // engine microtask hop, so the rest of the caller's frame is an engine - // continuation chunk like any other. The synchronous claim covers any - // reads before the hop; the sentinel re-claims contiguously ahead of - // the hop reaction (see the header above — issue #24's second shape - // was exactly a fast-path hop chunk misattributed after a sibling's - // claim intervened). + // Plain values still return through an engine hop. Claim synchronously + // for pre-hop reads and queue a sentinel before returning; other claims + // may interleave before wasm resumes, as described above. claimActivationAmbient(owner); sentinelFor(owner); return r; diff --git a/runtime/src/jspi/mechanics.ts b/runtime/src/jspi/mechanics.ts index 15a7331..4eaec1c 100644 --- a/runtime/src/jspi/mechanics.ts +++ b/runtime/src/jspi/mechanics.ts @@ -2,8 +2,7 @@ // `WebAssembly.Suspending`. This module is intentionally standalone: it has // no knowledge of the task/scheduler model (runtime/src/task,exec,intrinsics, // plan) and must not import from those directories. It is the mechanics -// layer that the upcoming JSPI scheduler phase (docs/architecture.md §6) will consume — -// not the scheduler itself. +// layer consumed by bridge.ts, not the scheduler itself. // // # The frame rule (docs/architecture.md §5) // @@ -12,10 +11,8 @@ // call to a `promising`-wrapped export and any call to a `Suspending`-wrapped // import. A JS frame anywhere in between traps. // -// This has been empirically pinned in `runtime/tests/jspi/frame_rule_test.ts` -// against Deno 2.9.5 / V8 15.0.245.2-rusty: see that file for the exact -// error constructor, message, and timing observed. Consequences (also -// findings, not just theory, per that test): +// `runtime/tests/jspi/frame_rule_test.ts` checks the error and timing. +// Consequences: // // - Host-boundary JS glue is safe: a `Suspending`-wrapped import's JS body // runs to completion and returns a Promise; the actual suspension @@ -28,11 +25,10 @@ // // # Reentrancy and concurrency (empirical, not mechanics-layer policy) // -// The engine permits things the Component Model forbids (e.g. reentering an -// instance while one of its exports is suspended) — docs/architecture.md §6 flags this as -// the scheduler's job to gate, not the engine's. See -// `reentry_test.ts`/`concurrent_activations_test.ts` for what the engine -// actually allows; this module does not enforce CM invariants. +// JSPI does not enforce task backpressure, callback exclusivity or poison +// refusal. Live-instance reentry is allowed; the scheduler owns the narrower +// admission and ordering rules. See `reentry_test.ts` and +// `concurrent_activations_test.ts` for engine behavior. import { jspiApi } from "./types.ts"; @@ -103,10 +99,10 @@ export function makePromising< /** * Wrap a JS function as a `Suspending` import: when a `promising`-suspendable * wasm activation calls it (per the frame rule above) and it returns a - * genuine Promise, the wasm activation suspends until that Promise settles; - * if it returns a non-Promise value (or the call site is not suspension- - * eligible), the value passes straight through — the "fast path", pinned in - * `suspending_import_test.ts::non_promise_return_is_fast_path`. + * genuine Promise, the wasm activation suspends until that Promise settles. + * Non-Promise results avoid that wait but still cross the engine's return + * hop. A non-eligible call is not a fallback path: it can trap even when the + * import returns a plain value (`suspending_import_test.ts`). * * The returned value is an opaque `WebAssembly.Suspending` instance; hand it * directly to the instantiation `imports` object in the slot the wasm module diff --git a/runtime/src/jspi/mod.ts b/runtime/src/jspi/mod.ts index 50018ba..72f00ae 100644 --- a/runtime/src/jspi/mod.ts +++ b/runtime/src/jspi/mod.ts @@ -1,5 +1,5 @@ -// JSPI mechanics module (docs/architecture.md §5/§6). Standalone: no imports from -// runtime/src/{task,exec,intrinsics,plan}. +// JSPI mechanics, scheduler bridge and protocol import markers +// (docs/architecture.md §5-6). Only mechanics.ts is scheduler-independent. export * from "./mechanics.ts"; export * from "./bridge.ts"; diff --git a/runtime/src/jspi/suspending.ts b/runtime/src/jspi/suspending.ts index 613932f..4d51103 100644 --- a/runtime/src/jspi/suspending.ts +++ b/runtime/src/jspi/suspending.ts @@ -1,23 +1,6 @@ -// The per-declaration suspendability marker (contracts/embedder-api.md -// §"Functions and async"; docs/architecture.md §5). -// -// The canonical definitions live in `@polyengine/protocol` since §"Module identity and @polyengine/protocol": -// the mark is a process-global `Symbol.for("polyengine.suspending/1")` brand, so -// a function marked by ANY runtime copy is honored by every other copy -// (issue #83 — a module-local symbol made a copy-B mark invisible to copy A's -// `anySuspendingImport`, silently downgrading the calling convention and -// surfacing far away as `NeedsJspi`). -// -// Layering: this module was import-free on purpose (jspi/ stays standalone); -// module identity relaxes that to "imports `@polyengine/protocol` only" — the protocol package -// is itself dependency-free, so jspi/ still pulls in no runtime machinery. - -// cancellation discard (`deferCancel`/`isDeferCancel`) and abortable() (`abortable`/`isAbortable`) -// ride the same re-export: they are the other per-declaration host-import -// marks, they live in the same dependency-free package, and -// `exec/executor.ts` reads all three through `jspi/mod.ts`. -// (Host modules import the marks from `@polyengine/protocol` directly — -// the embedder surface stopped re-exporting the vocabulary at host-ABI version.) +// Protocol-owned host-import markers. Their Symbol.for brands work across +// runtime copies. Host modules import @polyengine/protocol directly; the +// executor consumes these re-exports through jspi/mod.ts. export { abortable, anySuspendingImport, diff --git a/runtime/src/jspi/types.ts b/runtime/src/jspi/types.ts index 436882d..4ee8b83 100644 --- a/runtime/src/jspi/types.ts +++ b/runtime/src/jspi/types.ts @@ -1,20 +1,8 @@ // Module-scoped types + accessor for the JS Promise Integration (JSPI) API. // -// JSPI is phase 4 (docs/architecture.md §3) and not yet part of TypeScript's/Deno's -// built-in `lib.dom`/`lib.deno` typings. These declarations match the shape -// implemented by V8 (observed via Deno 2.9.5 / V8 15.0.245.2-rusty) and -// described by the js-promise-integration proposal Overview -// (https://github.com/WebAssembly/js-promise-integration). -// -// Deliberately NOT a `declare global` augmentation (it was one until JSR's -// server-side validation refused the package: "modifying global types is -// not allowed"). The registry policy and the truth agree here: JSPI is an -// OPTIONAL engine capability this runtime probes at runtime, and a global -// augmentation asserted it unconditionally for every downstream consumer. -// The module-scoped view keeps the assertion where the evidence is — -// `jspiApi()` returns the surface only when the engine actually has it. -// (Tests keep their own ambient declarations in -// runtime/tests/jspi/global_types.ts; test files are never published.) +// Module-scoped declarations avoid global augmentation in published packages. +// JSPI is optional: jspiApi returns this surface only after probing both APIs. +// Proposal: https://github.com/WebAssembly/js-promise-integration. /** * The engine's JSPI surface, as probed. diff --git a/runtime/src/plan/format.ts b/runtime/src/plan/format.ts index 368c7d5..bc09598 100644 --- a/runtime/src/plan/format.ts +++ b/runtime/src/plan/format.ts @@ -1,4 +1,4 @@ -// Plan v0 wire format (contracts/plan-format.md) — TypeScript mirror of the +// Plan wire format (contracts/plan-format.md) — TypeScript mirror of the // shim's serde schema (crates/translator-shim/src/plan.rs). Field names and // shapes must track the Rust side tag-for-tag; the shim is the producer of // record. @@ -25,30 +25,24 @@ export interface WirePlan { types: WireTypeDecl[]; resourceTables: WireResourceTable[]; /** - * Stream-table metadata (plan v2), index space == wasmtime's + * Stream-table metadata, index space == wasmtime's * `TypeStreamTableIndex`; referenced by the `streamTable` field of every * `stream.*` trampoline. `element` is the `T` of `stream`, `null` for the * zero-width payload. * - * ISSUE #94(2): the shim never `skip_serializing_if`s this field - * (crates/translator-shim/src/plan.rs), so every v2 plan the producer - * emits carries it (`[]` when empty). Required, not optional: the loader - * only ever accepts `formatVersion === 2` (strict equality, - * `SUPPORTED_FORMAT_VERSION`), so there is no live v1-compat path that - * needs this to be absent. + * Required even when empty, matching the producer's serialization. */ streamTables: WireAsyncTable[]; - /** Future-table metadata (plan v2); see `streamTables`. */ + /** Future-table metadata; see `streamTables`. */ futureTables: WireAsyncTable[]; /** - * Error-context-table metadata (plan v3), index space == wasmtime's + * Error-context-table metadata, index space == wasmtime's * `TypeComponentLocalErrorContextTableIndex` — the space the * `error-context-transfer` trampoline's `srcTable`/`dstTable` *runtime* * arguments live in. No element type: wasmtime's `TypeErrorContextTable` * is `{ instance }` and nothing else. * - * Required for the same reason as `streamTables`/`futureTables`: the shim - * always serializes it and the loader accepts only `formatVersion === 3`. + * Required even when empty, like streamTables/futureTables. */ errorContextTables: WireErrorContextTable[]; @@ -57,9 +51,8 @@ export interface WirePlan { * `ResourceIndex = importedResources.length + DefinedResourceIndex` * (wasmtime `Component::resource_index`). * - * The `importedResources` field (contracts/plan-format.md schema); optional - * here so plans produced by an older shim still load — absent is read as - * "no imported resources". + * The loader accepts absence as no imported resources, although the + * current producer serializes the list even when empty. */ importedResources?: WireImportedResource[]; imports: WireImport[]; @@ -113,12 +106,12 @@ export type WireCoreDef = | { kind: "instance-flags"; instance: number } | { kind: "trampoline"; index: number } /** - * `CoreDef::UnsafeIntrinsic` (plan v1 / contracts/plan-format.md v0.3). + * `CoreDef::UnsafeIntrinsic`. * `intrinsic` is wasmtime's stable symbol name * (`UnsafeIntrinsic::name()`), not an enum ordinal. The executor * materializes `context-{get,set}-i32-{0,1}` as host functions over the * current thread's context storage (definitions.py `canon_context_get` / - * `canon_context_set`, lines 2348/2358) and fails at instantiate time on + * `canon_context_set`) and fails at instantiate time on * every other symbol. */ | { kind: "unsafe-intrinsic"; intrinsic: string }; @@ -137,9 +130,9 @@ export interface WireExportItem { /** * Trampoline declarations, tag-for-tag with the wasmtime `Trampoline` enum. - * Only the variants the executor implements are given precise field types; - * the rest are matched by `kind` and rejected at instantiate time with - * capability-aware errors (contracts/intrinsics.md §B). + * Selected variants have precise field types; the catch-all also covers + * implemented intrinsics whose fields are interpreted by the factory. + * Unsupported kinds fail when resolved (contracts/intrinsics.md §B). * @internal */ export type WireTrampoline = @@ -181,8 +174,7 @@ export type WireTrampoline = instance: number; /** * The **raw** wasmtime `TypeTupleIndex` of the task's declared results - * (plan v3; in v2 this field held the interned `plan.types` index that - * `resultType` now carries). It is the key FACT's `prepare-call` passes + * It is the key FACT's `prepare-call` passes * as `task_return_type` at runtime, so it is what lets a FACT callee task * find its own declared result type. */ @@ -270,7 +262,6 @@ export type WireResourceTable = | { kind: "concrete"; resource: number; instance: number } | { kind: "abstract"; id: number }; -/** One stream or future table (plan v2). */ /** * One error-context table: the owning component instance, nothing else. * @internal @@ -279,7 +270,8 @@ export interface WireErrorContextTable { instance: number; } -/** @internal */ +/** One stream or future table: element type and owning component instance. + * @internal */ export interface WireAsyncTable { element: WireValType | null; instance: number; @@ -334,12 +326,11 @@ export type WireTypeExport = export interface WireEnvelope { plan?: WirePlan; adapters?: { file: string; wasm: string }[]; - /** Failure message (v0.1 shape; unchanged meaning). */ + /** Failure message. */ error?: string; /** - * Structured verdict accompanying `error` (contracts v0.2 proposal). Absent - * from v0.1 producers; consumers must tolerate that (treat as phase - * `"internal"`, i.e. "not a statement about the component"). + * Structured verdict accompanying `error`. If absent, consumers treat the + * phase as internal, not a validation verdict about the component. */ errorDetail?: WireErrorDetail; } @@ -347,7 +338,7 @@ export interface WireEnvelope { /** * Structured translation failure. * - * `phase` is the load-bearing field: only `"validation"` means *the component + * Only phase `"validation"` means *the component * is invalid/malformed* — the verdict the official suite's `assert_invalid` / * `assert_malformed` commands require. `"unsupported"` means the component is * valid but uses a shape this plan-format version cannot express, and diff --git a/runtime/src/plan/loader.ts b/runtime/src/plan/loader.ts index 8065ece..0983bf2 100644 --- a/runtime/src/plan/loader.ts +++ b/runtime/src/plan/loader.ts @@ -8,7 +8,7 @@ // (types.ts drops ABI-irrelevant names; labels are preserved separately // for bindgen/digest use) // - own/borrow `resource: ` (wire) -> `ResourceTypeInfo` -// identity tokens created per resource table at load time +// identity tokens shared by concrete tables naming one resource import { type FuncType, @@ -63,34 +63,8 @@ export class TranslateError extends Error { } /** - * The single formatVersion this executor understands. - * - * v5 (wasmtime `main` bump, polyengine translator-shim update): CoreDef lost - * `"task-may-block"` (wasmtime #14146 dropped the FACT-visible may-block - * global; sync-blocking is now enforced lazily by the scheduler). The `trap` - * trampoline gained a `code` field (now one nullary import per trap code, - * `runtime.trap`, rather than a call argument). The thread trampoline set - * was renamed/expanded: `thread-suspend-to-suspended`, `thread-suspend-to`, - * `thread-unsuspend`, `thread-yield-to-suspended` are gone; `thread-index` - * gained an `instance` field; `thread-resume-later`, - * `thread-suspend-then-resume`, `thread-yield-then-resume`, - * `thread-suspend-then-promote`, `thread-yield-then-promote` are new. - * v4 (2026-08-17, polyengine#13): `exports[]` gained the `"module"` kind — a - * component exporting one of its own embedded core modules, surfaced as the - * already-compiled `WebAssembly.Module`. - * v3 (2026-08-10, polyengine#89): `errorContextTables` — the index space the - * `error-context-transfer` trampoline actually uses (it was resolved through - * the *resource*-table mapping before, a different space) — and - * `task-return`'s `resultType` / raw `results` split, which lets a FACT - * callee task carry its declared result type. - * v2: `streamTables` / `futureTables` — the element types the - * stream and future built-ins need to size their copy buffers. - * v1 (contracts/plan-format.md v0.3): `CoreDef` gained `"unsafe-intrinsic"`. - * The change is purely additive, but the contract's compat rule is a strict - * equality check ("Validate `formatVersion` and fail fast on mismatch", - * producer and consumer bumped in the same commit), so v0 plans are refused - * rather than best-effort accepted — a stale cached artifact must be a loud - * failure, not a subtly different execution. + * The only accepted formatVersion. Strict equality prevents stale plans + * from being interpreted with different wire semantics. * @internal */ export const SUPPORTED_FORMAT_VERSION = 5; @@ -109,9 +83,9 @@ export interface LoadedPlan { /** Converted types table, index-aligned with `wire.types`. */ types: LoadedType[]; /** - * Identity tokens for resource tables, index-aligned with - * `wire.resourceTables`. The executor fills `impl`/`dtor` while running - * `resource` initializers. + * Index-aligned with wire.resourceTables. Concrete tables naming the same + * ResourceIndex share a token; abstract tables get distinct tokens. + * The executor fills implementation/destructor state during instantiation. */ resourceTokens: ResourceTypeInfo[]; /** @@ -120,21 +94,21 @@ export interface LoadedPlan { * (the `importedResources` field; contracts/plan-format.md schema). */ numImportedResources: number; - /** Element type per stream table (plan v2); `null` = zero-width payload. */ + /** Element type per stream table; `null` = zero-width payload. */ streamElems: (ValType | null)[]; - /** Element type per future table (plan v2). */ + /** Element type per future table. */ futureElems: (ValType | null)[]; - /** Owning component instance per stream/future table (plan v2). */ + /** Owning component instance per stream/future table. */ streamTableInstances: number[]; futureTableInstances: number[]; /** - * Owning component instance per error-context table (plan v3), index space + * Owning component instance per error-context table, index space * == `TypeComponentLocalErrorContextTableIndex`. */ errorContextTableInstances: number[]; /** * Raw wasmtime `TypeTupleIndex` -> `plan.types` index, collected from the - * `task-return` trampolines (plan v3). The key is what FACT's + * `task-return` trampolines. The key is what FACT's * `prepare-call` passes as `task_return_type` at runtime; the value is the * interned tuple type. A callee with no `task.return` trampoline of its own * (a sync-lifted callee) contributes no entry, and the lookup then reports @@ -163,19 +137,9 @@ export function loadPlan(wire: WirePlan): LoadedPlan { "canonicalOptions", "types", "resourceTables", - // ISSUE #94(2): the shim (crates/translator-shim/src/plan.rs) has no - // `skip_serializing_if` on `stream_tables`/`future_tables` — a real - // emitted v2 plan always serializes these as arrays (`[]` when empty, - // never absent). Requiring presence here keeps the loader consistent - // with what the producer actually emits, rather than silently - // tolerating an absent field via `?? []` (which would also mask a - // genuinely malformed/truncated envelope). + // The producer emits empty arrays, not absent table fields. "streamTables", "futureTables", - // Same reasoning at v3 (the shim has no `skip_serializing_if` on - // `error_context_tables` either): presence is what the producer - // guarantees, so absence is a malformed envelope, not an empty table - // space. "errorContextTables", "imports", "exports", @@ -186,14 +150,8 @@ export function loadPlan(wire: WirePlan): LoadedPlan { } } - // ISSUE #94(3): deep-schema strictness. `initializers` / `trampolines` / - // `canonicalOptions` / `CoreDef`s reach `runInitializers` unchecked today; - // a malformed op object (e.g. `{"op":"instantiate-module"}` missing - // `args`) dies as a raw `TypeError` deep in the executor rather than a - // typed `PlanError` here at load time. Proportionate check: a - // discriminated-union switch per op/trampoline kind verifying required - // fields are present and primitively typed — not a full JSON-schema - // engine. + // Validate required fields and primitive shapes before execution. These + // checks are not a complete schema or cross-reference validator. wire.initializers.forEach((init, i) => validateInitializer(init, `initializers[${i}]`) ); @@ -203,20 +161,11 @@ export function loadPlan(wire: WirePlan): LoadedPlan { wire.canonicalOptions.forEach((o, i) => validateCanonicalOptions(o, `canonicalOptions[${i}]`) ); - // plan v3: `errorContextTables` entries are `{ instance }` and nothing - // else; the executor routes real handle-table lookups through them, so a - // malformed entry must fail here rather than as an undefined index later. wire.errorContextTables.forEach((t, i) => { const where = `errorContextTables[${i}]`; expect(isRecord(t), where, `must be an object, got ${describeValue(t)}`); expectNumber(t as unknown as Record, "instance", where); }); - // ISSUE #187: `modules[]` / `exports[]` / `imports[]` get the same - // deep-schema treatment as initializers/trampolines/canonicalOptions - // (#94(3)) — a malformed entry must die here as a typed `PlanError`, - // never as a raw TypeError in `Executor.buildExport` or (worse) as a - // silent negative-offset slice of the wrong component bytes in - // `compileModules` (executor.ts's only guard was an upper-bound check). wire.modules.forEach((m, i) => validateModule(m, `modules[${i}]`)); wire.imports.forEach((imp, i) => validateImport(imp, `imports[${i}]`)); wire.exports.forEach((exp, i) => validateExport(exp, `exports[${i}]`)); @@ -234,22 +183,9 @@ export function loadPlan(wire: WirePlan): LoadedPlan { } } - // Identity tokens: one per RESOURCE, aliased through every table that - // names it — NOT one per table. plan-format.md "Type exports index into `resourceTables`": "one - // resource type can be reachable through several distinct table indices … - // Consumers keying per-resource state must key by `resourceTables[n] - // .resource`, treating table indices as aliases." Minting per-table broke - // exactly the way that warning predicts (found by the #18 polymorph-tls - // smoke): in a wac-composed component the source and destination future - // tables of a FACT transfer resolve `own` through different table - // indices, and `valTypeEqual`'s documented reference-identity comparison - // (cabi/types.ts) saw two tokens for one resource — "future: destination - // element mismatch" on every resource-bearing element type. wasmtime - // interns identity at the `ResourceIndex` level and its transfer libcall - // never re-compares element types at runtime (47.0.3 - // futures_and_streams.rs `guest_transfer`); unifying here restores parity - // for every structural-equality site at once. Abstract tables keep - // per-table tokens (no `resource` to key by; none in the current corpus). + // Nominal identity is per ResourceIndex, not table index. Alias concrete + // tables so resource-bearing type comparisons agree across component hops. + // Abstract tables have no ResourceIndex and retain per-table tokens. const tokenByResource = new Map(); const resourceTokens = wire.resourceTables.map((table) => { if (table.kind !== "concrete") return new ResourceTypeInfo(null, null); @@ -263,7 +199,7 @@ export function loadPlan(wire: WirePlan): LoadedPlan { const types = wire.types.map((t, i) => loadTypeDecl(t, resourceTokens, `types[${i}]`) ); - // plan v3: the `task-return` decls double as the `TypeTupleIndex` -> + // The `task-return` decls double as the `TypeTupleIndex` -> // `plan.types` dictionary (see `LoadedPlan.resultTupleTypes`). Two decls // naming the same raw tuple must agree — they are interned from one // wasmtime type, so disagreement means a hand-edited/corrupt plan. @@ -312,9 +248,8 @@ export function loadPlan(wire: WirePlan): LoadedPlan { /** * Component-wide `ResourceIndex` for a `DefinedResourceIndex` (the `index` - * field of a `resource` initializer). Mirrors wasmtime - * `Component::resource_index` (wasmtime-environ 47.0.3 - * `component/info.rs:222`). + * field of a `resource` initializer). Imported resources precede definitions, + * matching wasmtime's `Component::resource_index`. * @internal */ export function resourceIndexOfDefined( @@ -342,8 +277,7 @@ export function loadEnvelope(json: string): { throw new PlanError(`envelope is not valid JSON: ${e}`); } if (envelope.error !== undefined) { - // v0.1 producers send only `error`; treat the missing structured verdict - // as "internal" — an unknown phase must never be read as a validation + // Without a structured phase, the error cannot count as a validation // verdict (see TranslateError). throw new TranslateError( envelope.errorDetail ?? @@ -366,7 +300,7 @@ function base64Decode(s: string): Uint8Array { return out; } -// --- ISSUE #94(3): deep-schema validation -------------------------------- +// --- Wire shape validation ---------------------------------------------- // // Proportionate shape-checking for the wire-format ops the executor runs // strictly: required-field presence + primitive-type checks per @@ -424,11 +358,8 @@ function expectString( ); } -// ISSUE #187: `offset`/`len` reach `Uint8Array.slice` unchecked today; a -// negative or non-integer value is not merely "wrong type" (expectNumber -// would pass NaN and negatives through) but silently slices the *wrong* -// component bytes (`slice(-100, -8)` reads from the tail). Reject anything -// that is not a non-negative safe integer. +// Slice offsets/lengths need safe non-negative integers: negative offsets +// would silently select bytes relative to the component's end. function expectNonNegativeInt( o: Record, field: string, @@ -580,10 +511,9 @@ function validateInitializer(init: unknown, where: string): void { // Trampoline kinds with precise wire shapes (format.ts's non-catch-all // arms). Everything else falls to the `{ kind: string; index: number; -// [field: string]: unknown }` catch-all — milestone-aware unsupported -// kinds the executor rejects at instantiate time (contracts/intrinsics.md -// §B), so only `kind` (string) and `index` (number) are load-time -// invariants for those. +// [field: string]: unknown }` catch-all. Some catch-all kinds are implemented +// by the intrinsic factory; only their common fields are validated here. +// Unsupported kinds fail when the executor resolves them. function validateTrampoline(t: unknown, where: string): void { expect(isRecord(t), where, `must be an object, got ${describeValue(t)}`); const tr = t as Record; @@ -604,7 +534,7 @@ function validateTrampoline(t: unknown, where: string): void { case "task-return": expectNumber(tr, "instance", where); expectNumber(tr, "results", where); - // plan v3: required, `number | null`. + // Required, `number | null`. expectNumberOrNull(tr, "resultType", where); expectNumber(tr, "options", where); return; @@ -615,8 +545,7 @@ function validateTrampoline(t: unknown, where: string): void { expectNumber(tr, "resource", where); return; default: - // Catch-all: unknown/milestone-gated kind, only the common fields - // above are required. + // Catch-all: only common fields are checked here. return; } } @@ -658,12 +587,7 @@ function validateCanonicalOptions(o: unknown, where: string): void { } } -// ISSUE #187: `modules[]` — mirrors format.ts's `WireModule` union exactly. -// `embedded`'s `offset`/`len` are the fields the negative-offset walk in -// the issue exploits (executor.ts's only guard was `end > length`, which a -// negative `offset` sails through); `adapter`'s `file`/`len`/`intrinsics` -// are what `compileModules`/intrinsic wiring dereference unchecked -// downstream. +// Validate WireModule fields used by compilation and intrinsic wiring. function validateModule(m: unknown, where: string): void { expect(isRecord(m), where, `must be an object, got ${describeValue(m)}`); const mm = m as Record; @@ -702,9 +626,7 @@ function validateIntrinsicEntry(entry: unknown, where: string): void { validateCoreDef(e.def, `${where}.def`); } -// ISSUE #187: `imports[]` — mirrors format.ts's `WireImport`. `type` is -// optional on the wire (present only for imports that carry an interned -// type-table index), so it is checked only when present. +// Only imports carrying an interned type-table index have a `type` field. function validateImport(imp: unknown, where: string): void { expect(isRecord(imp), where, `must be an object, got ${describeValue(imp)}`); const i = imp as Record; @@ -721,9 +643,7 @@ function validateImport(imp: unknown, where: string): void { if (i.type !== undefined) expectNumber(i, "type", where); } -// ISSUE #187: `exports[]` — mirrors format.ts's `WireExport` union, -// recursing into `instance`'s nested `exports[]` (the "each kind's fields -// shape-checked … recursive for nested instance export lists" requirement). +// Recurse through nested instance exports as well as checking each leaf. function validateExport(exp: unknown, where: string): void { expect(isRecord(exp), where, `must be an object, got ${describeValue(exp)}`); const e = exp as Record; diff --git a/runtime/src/plan/mod.ts b/runtime/src/plan/mod.ts index 497e0d9..3fe0e35 100644 --- a/runtime/src/plan/mod.ts +++ b/runtime/src/plan/mod.ts @@ -22,7 +22,7 @@ * @module */ -// Plan format + loader (contracts/plan-format.md v0). +// Plan format + loader (contracts/plan-format.md). export * from "./format.ts"; export * from "./loader.ts"; diff --git a/runtime/src/shim/translator.ts b/runtime/src/shim/translator.ts index 6aad7f7..bffdbd3 100644 --- a/runtime/src/shim/translator.ts +++ b/runtime/src/shim/translator.ts @@ -33,7 +33,7 @@ export class Translator { * `null` when constructed from a pre-compiled `WebAssembly.Module` with no * bytes available (module identity can't be recovered post-compile). * - * This is the honest translator "build hash" for the artifact cache + * This is the translator build identity for the artifact cache * (docs/architecture.md §10): the wire envelope's `producer` block records * `{shimVersion, wasmtimeEnviron, features}`, which does NOT change when * the shim wasm is rebuilt from the same source versions (e.g. a local @@ -76,8 +76,8 @@ export class Translator { } /** - * Wrap an ALREADY-INSTANTIATED shim — the ESM wasm-module import path - * (issue #16 delivery design): `import * as shim from ".../translator_shim.wasm"` + * Wrap an already-instantiated shim, including an ESM wasm namespace: + * `import * as shim from ".../translator_shim.wasm"` * hands back an instantiated namespace (the shim imports nothing, so the * ESM integration instantiates it trivially), and this wraps it with no * further compile or copy. @@ -91,7 +91,7 @@ export class Translator { * `buildHash` (hex sha-256 of the shim wasm bytes) cannot be recovered * from an instance; pass it when known — a published package can ship the * hash of the exact asset it carries — or leave it absent and the - * artifact cache politely refuses to key on translator identity + * artifact cache refuses to key on translator identity * (cache/core.ts). */ static fromExports( @@ -101,7 +101,7 @@ export class Translator { return new Translator(exports, opts.buildHash ?? null); } - /** Translate a component binary into plan v0 + adapter artifacts. */ + /** Translate a component binary into a validated plan and adapter artifacts. */ translate(componentBytes: Uint8Array): TranslationResult { const json = this.translateRaw(componentBytes); const { wire, adapters } = loadEnvelope(json); diff --git a/runtime/src/task/mod.ts b/runtime/src/task/mod.ts index f29479d..552da6a 100644 --- a/runtime/src/task/mod.ts +++ b/runtime/src/task/mod.ts @@ -1,16 +1,5 @@ -// The 0.3 task model (docs/architecture.md §6): `ComponentInstance`, `Task`, and the -// re-export surface of the task core. Thread, Waitable/WaitableSet, Subtask -// and the scheduler live in sibling modules; see ./scheduler.ts for the -// scheduling-policy rationale and the generator-based thread model. -// -// Structural correspondence to definitions.py is the design constraint here: -// where this file diverges, the divergence is called out in a comment with -// the reference's line number. The two systematic divergences are -// -// 1. threads are generators, not OS threads (./scheduler.ts header), and -// 2. the shared-everything-threads built-ins (`thread.suspend-then-resume` -// and friends, 🧵) are absent rather than approximated — https://github.com/polymorph-components/polyengine/issues/12 -// defers that feature with memory64. +// Component-instance and task state, following definitions.py `ComponentInstance` +// and `Task`. Scheduler policy and JSPI ordering live in ./scheduler.ts. import { Table } from "../cabi/handles.ts"; import { COMPONENT_INSTANCE } from "../cabi/context.ts"; @@ -42,25 +31,11 @@ export * from "./streams.ts"; export type HandleTableEntry = unknown; /** - * Per-component-instance runtime state (definitions.py `ComponentInstance`, - * line 191). - * - * `mayLeave` is backed by a real `WebAssembly.Global(i32, mutable)` because - * FACT adapters import that global (`flags` namespace) and read/write it as - * the may_leave boolean (wasmtime 47 FACT treats the whole flags global as - * may_leave; there is no bitmask). Initial value 1 (true). - * - * There is no `may_enter` counterpart and no instance tree: at the pinned - * reference (definitions.py @ 2f13265, CM#705) there is no `may_enter`, - * `parent`, `entering_set`, `enter_from` or `leave_to`, so nothing gates - * entry into a live instance. What polyengine adds beyond the reference is - * per-instance POISONING — a named divergence - * living entirely in ./scheduler.ts (`isInstancePoisoned`, `entryRefusal`), - * not in any state on this class. - * - * `COMPONENT_INSTANCE` brands this class as a real component instance for - * the layers that only see the structural `ComponentInstanceLike` - * (cabi/handles.ts `isComponentInstance`; cabi must not import task/). + * Per-instance state. FACT shares mayLeave through a mutable i32 global + * containing a boolean, not a bitmask. Task admission uses backpressure and + * exclusiveThread, not a general reentry lock. Poison refusal lives separately + * in scheduler.ts. COMPONENT_INSTANCE lets cabi identify real instances + * without importing the task layer. */ export class ComponentInstanceState implements ComponentInstanceLike { readonly index: number; @@ -93,7 +68,7 @@ export class ComponentInstanceState implements ComponentInstanceLike { } } -/** definitions.py `Task.State` (line 445). */ +/** definitions.py `Task.State`. */ export type TaskState = | "initial" | "started" @@ -105,14 +80,13 @@ export type OnStart = () => ComponentValue[]; export type OnResolve = (result: ComponentValue[] | null) => void; /** - * Canonical options as the task model needs to see them (definitions.py - * `Task.opts`): only the two flags that change task *semantics*. + * Task execution flags and the lift-option identity checked by task.return. */ export interface TaskOptions { async_: boolean; callback: boolean; /** - * The two fields definitions.py's `LiftOptions.equal` (line 643) compares. + * The two fields definitions.py's `LiftOptions.equal` compares. * `canon_task_return` requires the options at the `task.return` site to * equal the ones the task was lifted with, so the task has to remember * them. @@ -121,7 +95,7 @@ export interface TaskOptions { memory: unknown | null; } -/** definitions.py `LiftOptions.equal` (line 643): encoding + memory identity. */ +/** definitions.py `LiftOptions.equal`: encoding + memory identity. */ export function liftOptionsEqual( a: { stringEncoding: string; memory: unknown | null }, b: { stringEncoding: string; memory: unknown | null }, @@ -129,10 +103,6 @@ export function liftOptionsEqual( return a.stringEncoding === b.stringEncoding && a.memory === b.memory; } -/** - * One export activation (definitions.py `class Task`, line 444). Also the - * task-side borrow scope: `numBorrows` satisfies cabi's `TaskBorrowScope`. - */ const ADMIT_TRACE = (() => { try { return Deno.env.get("CE_SP_TRACE") === "1"; @@ -141,6 +111,8 @@ const ADMIT_TRACE = (() => { } })(); +/** One call and its threads (`Task` in definitions.py), also a cabi borrow + * scope. Resolution delivers the result; remaining threads may keep running. */ export class Task { state: TaskState = "initial"; /** TaskBorrowScope (cabi/context.ts): live borrows lowered into this task. */ @@ -149,54 +121,18 @@ export class Task { implicitThread: Thread | null = null; readonly threads: Thread[] = []; /** - * True for a task created by a FACT cross-component call - * (`prepare-call`, see intrinsics/fact_calls.ts). - * - * Such a task's `onStart` / `onResolve` carry **flat core values**, not - * lifted component values: FACT fuses the caller-side lift and callee-side - * lower into a pair of adapter functions (`[async-start]` / `[async-return]`) - * that run *in wasm*, so the host only shuttles the core values between - * them. definitions.py has no analogue because it has no fused adapters — - * there, `canon_lift` lowers the params and `canon_lower`'s `on_resolve` - * lifts the results, both in the host. The observable semantics are - * identical; only which side of the boundary performs the copy differs. - * - * `canon_task_return` consults this to decide whether to lift its flat - * arguments (host-boundary task) or pass them straight through (FACT task). + * FACT tasks shuttle flat core values through onStart/onResolve; wasm + * adapters perform the conversions. canon_task_return passes those values + * through rather than applying the host-boundary result lift. */ factPassthrough = false; /** - * Plan v3: does `ft.results` hold this FACT task's *declared* result type? - * - * A FACT callee task's result type arrives as the raw wasmtime - * `TypeTupleIndex` `prepare-call` passes as `task_return_type`; v3's - * `task-return.results` / `resultType` pair is the dictionary for it - * (the task-return trampoline's raw `results` key + interned `resultType`; - * contracts/plan-format.md schema). It resolves for every callee - * that has a `task.return` trampoline of its own — which is every callee - * that can call `task.return` — but a callee with none (sync-lifted, - * reached through an async-to-sync adapter) contributes no entry, and then - * `ft.results` is the empty placeholder it was before v3. Only when this is - * true may `canon_task_return` compare against it. + * Whether ft.results is the declared FACT result type rather than an + * empty placeholder. prepare-call's raw tuple index resolves through the + * plan's task-return results/resultType mapping. Callees without such a + * trampoline may lack a mapping; only known types may be compared. */ factResultTypesKnown = false; - /** - * In-flight FACT sync-call brackets for THIS task - * (`enter-sync-call`/`exit-sync-call`). - * - * MOVED to `Thread` (see `Thread.syncCallStack`). Per-task was already an - * improvement on per-executor, but it is still not the right unit: a task - * can own several threads, so one activation's `exit-sync-call` could pop a - * sibling activation's scope. Tracing big-interleaving showed exactly that - * -- tasks whose `enter` count exceeded their `exit` count by one, and other - * tasks taking an `exit` at depth 0, with the `ctx` fallback never firing. - * - * The bracket belongs to the ACTIVATION that opened it: FACT emits the - * matching `enter-sync-call` and `exit-sync-call` from the same wasm - * activation by construction, so riding the activation identity makes the - * exit find the same stack the enter used no matter which task the scheduler - * considers current in between (the 3i bracket-spans-suspension ruling). - */ constructor( public ft: FuncType, @@ -207,9 +143,9 @@ export class Task { ) {} /** - * definitions.py `Task.needs_exclusive` (line 473): an async-typed task + * definitions.py `Task.needs_exclusive`: an async-typed task * needs the instance's exclusive thread unless it is a *stackful* async - * lift. Sync-lowered (`not opts.async_`) and callback-ABI tasks both do. + * lift. Sync canonical lifts (`not opts.async_`) and callback-ABI tasks both do. */ needsExclusive(): boolean { assert_(this.ft.async === true, "needs_exclusive on a sync-typed task"); @@ -217,7 +153,7 @@ export class Task { } /** - * definitions.py `Task.enter_implicit_thread` (line 477) — the backpressure + * definitions.py `Task.enter_implicit_thread` — the backpressure * and exclusivity gate, in full. * * Returns false when the task was cancelled while waiting to enter, in @@ -264,7 +200,7 @@ export class Task { return true; } - /** definitions.py `Task.register_thread` (line 497). */ + /** definitions.py `Task.register_thread`. */ registerThread(thread: Thread): void { assert_( !this.threads.includes(thread) && thread.task === this, @@ -275,16 +211,13 @@ export class Task { thread.index = this.inst.threads.add(thread); } - /** definitions.py `Task.exit_implicit_thread` (line 503). */ + /** definitions.py `Task.exit_implicit_thread`. */ exitImplicitThread(thread: Thread): void { assert_(thread === this.implicitThread, "exit of a non-implicit thread"); this.unregisterThread(thread); if (this.ft.async === true && this.needsExclusive()) { - // definitions.py lines 506-508, verbatim shape: assert-held, then - // release. The former release-if-held tolerance existed only for the - // removed release-at-BLOCK divergence (issue #43); under the hold rule - // the implicit thread of a needs-exclusive task holds the slot from - // `enter_implicit_thread` to here, without exception. + // Callback waits release and retake the slot between invocations; + // the final invocation must still own it when exiting. assert_( this.inst.exclusiveThread === thread, "exit_implicit_thread without holding the exclusive thread", @@ -293,7 +226,7 @@ export class Task { } } - /** definitions.py `Task.unregister_thread` (line 510). */ + /** definitions.py `Task.unregister_thread`. */ unregisterThread(thread: Thread): void { const i = this.threads.indexOf(thread); assert_(i !== -1 && thread.task === this, "unregister of a foreign thread"); @@ -311,13 +244,13 @@ export class Task { } /** - * definitions.py `Task.request_cancellation` (@ 2f13265). Delivered to a + * definitions.py `Task.request_cancellation`. Delivered to a * cancellable thread if one exists; otherwise recorded as pending, to be * picked up at the next cancellable block point (`deliverPendingCancel`). * * `caller` is retained for the call-site shape (fact_calls.ts's * `subtask.onCancel`) and for diagnostics; no condition here consults it - * (CM#705: entry into a live instance is ungated). + * (live-instance reentry is allowed). */ requestCancellation(caller: ComponentInstanceState | null): void { void caller; @@ -330,20 +263,9 @@ export class Task { this.state === "started", `request_cancellation in state ${this.state}`, ); - // Candidates are CANCELLABLE BLOCK POINTS of this task. The reference - // only ever finds them among `self.threads`, because its threads block - // *in place* (`wait_until` marks the thread itself cancellable). Under - // jspi the same block point is a `SuspensionPoint` parked in - // `store.waiting` — the wasm frame is suspended mid-built-in and the - // Thread that owns the activation sits non-cancellably on its - // `awaitValue` — so a scan of `threads` alone finds nothing and a - // cancellation the reference delivers synchronously was silently - // deferred to `pending-cancel` (cancellable.wast:322, test 1: a - // cancellable `waitable-set.wait` must observe TASK_CANCELLED). - // A resumed SuspensionPoint hands `cancelled` to its `produce`, which - // every cancellable built-in already translates (TASK_CANCELLED for - // waits, 1 for thread.yield), so delivery works unchanged once the - // point is simply *found*. + // Include JSPI SuspensionPoints: their owning generator waits on a + // non-cancellable awaitValue, while the actual cancellable park is in + // store.waiting. Resume delivers the flag to that point's produce callback. type Cancellable = { cancellable: boolean; resume(cancelled?: boolean): void; @@ -353,10 +275,7 @@ export class Task { if (excludeImplicit) { candidates = candidates.filter((t) => t !== this.implicitThread); } - // Suspension points of this task's activation are frames OF the implicit - // thread, so they obey the same exclusion (definitions.py line 526: with - // another thread holding the exclusive slot, the implicit thread may not - // run). + // The implicit thread's SuspensionPoints obey the same exclusivity test. if (!excludeImplicit) { const store = this.inst.store as unknown as { waiting: ({ task?: unknown } & Cancellable)[]; @@ -370,28 +289,13 @@ export class Task { } } } - // Merged reference (definitions.py @ 2f13265): `if candidates: deliver`, - // full stop — no enterability condition, no bracket (CM#705). - // - // ONE divergence conjunct survives: a POISONED instance is a corpse whose - // threads never resume, so the request parks as pending-cancel forever — - // which is the honest state, since a corpse can never reach a cancellable - // suspension to deliver at. The reference never faces this because a trap - // there kills the whole store. The marker is the authoritative input. + // Poisoned instances cannot run a cancellation recipient. if (candidates.length > 0 && !isInstancePoisoned(this.inst)) { this.state = "cancel-delivered"; try { chooseCandidate(candidates).resume(CANCELLED_TRUE); } catch (e) { - // A trap escaping the delivery poisons the callee instance - // (polyengine#164/#212) — polyengine's per-instance corpse divergence; - // the reference wraps this `resume(Cancelled.TRUE)` in no handler at - // all and simply ends the world. - // - // Capability signals are the exception, exactly as in `tick`: they - // mark this RUNTIME incomplete, not the component faulted, and in the - // reference the blocking operation they stand in for completes - // normally. + // Escaping delivery failures poison the recipient, except capability signals. if (!(e instanceof NeedsJspi) && !(e instanceof PendingCapability)) { notifyInstancePoisoned( this.inst as unknown as { handles: Iterable }, @@ -406,15 +310,10 @@ export class Task { } /** - * Is the implicit thread cancellable *right now*? - * - * The reference makes cancellability a live predicate — the callback loop - * passes `cancellable = lock_available` (definitions.py 2167/2175), false - * while a sibling activation of the instance holds the exclusive slot. We - * carry a static flag per block point instead, so this is where the "and - * the lock is free" conjunct lives: both `request_cancellation`'s candidate - * filter and `Thread.wait_until`'s pending-cancel wakeup disjunct consult - * it. + * Live exclusivity conjunct for cancellability. `canon_lift`'s callback + * waits use lock_available; static park flags alone cannot represent a + * sibling taking the slot. Both delivery selection and pending-cancel + * readiness consult this predicate. */ implicitThreadCancellable(): boolean { return !(this.ft.async === true && this.needsExclusive() && @@ -422,12 +321,12 @@ export class Task { this.inst.exclusiveThread !== this.implicitThread); } - /** definitions.py `Task.has_pending_cancel` (line 533). */ + /** definitions.py `Task.has_pending_cancel`. */ hasPendingCancel(): boolean { return this.state === "pending-cancel"; } - /** definitions.py `Task.deliver_pending_cancel` (line 536). */ + /** definitions.py `Task.deliver_pending_cancel`. */ deliverPendingCancel(cancellable: boolean): boolean { if (cancellable && this.hasPendingCancel()) { this.state = "cancel-delivered"; @@ -436,18 +335,17 @@ export class Task { return false; } - /** definitions.py `Task.start` (line 542). */ + /** definitions.py `Task.start`. */ start(): ComponentValue[] { assert_(this.state === "initial", "start on a started task"); this.state = "started"; return this.onStart(); } - /** definitions.py `Task.return_` (line 547). */ + /** definitions.py `Task.return_`: deliver the result before setting state. + * Resolution does not unregister threads or release callback exclusivity. */ return_(result: ComponentValue[]): void { trapIf(this.state === "resolved", "task.return on a resolved task"); - // Wording parity with wasmtime's exit-time check, pinned by - // drop-cross-task-borrow.wast:309. trapIf( this.numBorrows > 0, "borrow handles still remain at the end of the call", @@ -456,7 +354,7 @@ export class Task { this.state = "resolved"; } - /** definitions.py `Task.cancel` (line 554). */ + /** definitions.py `Task.cancel`. */ cancel(): void { trapIf( this.state !== "cancel-delivered", diff --git a/runtime/src/task/scheduler.ts b/runtime/src/task/scheduler.ts index aa55e2a..eb5f2ae 100644 --- a/runtime/src/task/scheduler.ts +++ b/runtime/src/task/scheduler.ts @@ -1,55 +1,19 @@ -// The 0.3 task scheduler (docs/architecture.md §6) — the `Store` of definitions.py plus -// the current-thread context that every canonical built-in reads. +// Scheduler and canonical built-in ambient (docs/architecture.md §6). // -// =========================================================================== -// SCHEDULING POLICY (orchestrator decision, docs/architecture.md §6) -// =========================================================================== +// The default chooses ready threads in waiting-list order and events in join +// order. These are deterministic choices within definitions.py `Store.tick` +// and `WaitableSet.get_pending_event`'s allowed nondeterminism, not ordering by +// the instant a readiness predicate became true. POLYENGINE_SCHED_SEED selects +// reproducible pseudo-random candidates; `Thread.waitUntil` always takes the +// reference's deterministic-profile blocking path. // -// definitions.py makes two explicitly nondeterministic choices: -// -// * `Store.tick` (line 597): `random.choice(list(candidates))` over ready -// threads; -// * `WaitableSet.get_pending_event` (line 821): `random.shuffle(self.elems)` -// before picking a waitable with a pending event; -// * `Thread.wait_until` (line 396): `if ready_func() and not -// DETERMINISTIC_PROFILE and random.randint(0,1): return` — an optional -// "don't block even though you could" fast path. -// -// All three are *allowed* nondeterminism, not required: any single consistent -// choice is a conforming schedule. This scheduler therefore runs a -// **deterministic FIFO ready queue** by default — candidates are resumed in -// the order they became ready, waitable sets deliver events in join order, -// and `wait_until` always blocks (the reference's `DETERMINISTIC_PROFILE` -// branch). Reproducible schedules are worth a great deal when debugging a -// concurrency bug, and FIFO is also the fairest of the cheap policies. -// -// Setting `POLYENGINE_SCHED_SEED=` switches to a **seeded shuffle**: the same -// choice points become pseudo-random but reproducible from the seed, which is -// how we explore the schedule space that the FIFO default deliberately pins. -// A test that passes under FIFO but fails under some seed has found a real -// order-dependence — in our runtime or in the guest. The seed is read once at -// module load; `schedulerSeedForTesting` exists so tests can drive both modes -// without a subprocess. -// -// =========================================================================== -// THREADS WITHOUT STACK SWITCHING -// =========================================================================== -// -// definitions.py implements `Thread` on real OS threads with lock handoff -// (`cont_new`/`resume`/`block`, lines 270-305) purely to get one-shot -// continuations. We get the same structure from **JS generators**: a thread -// body is a generator function that `yield`s a block request and is resumed -// by `next(cancelled)`. That is a faithful model precisely because the -// stackless (callback-ABI) path never blocks *inside* a wasm frame — every -// wasm call returns a callback code before the host decides to wait. Blocking -// inside a wasm frame (stackful async lifts; a sync lower on an unresolved -// subtask) genuinely requires JSPI; those sites fail loudly -// rather than pretending (see `needsJspi`). +// Generator bodies handle waits between wasm calls. Mid-wasm blocking requires +// JSPI; the bridge presents suspended frames to the same scheduler. import { assert_, trapIf } from "../cabi/trap.ts"; import type { ComponentInstanceLike } from "../cabi/context.ts"; -/** definitions.py `Cancelled` (line 248). */ +/** definitions.py `Cancelled`. */ export const CANCELLED_FALSE = false; export const CANCELLED_TRUE = true; export type Cancelled = boolean; @@ -64,18 +28,8 @@ export interface BlockRequest { /** Resumable once this returns true; `null` = only an explicit resume. */ readyFunc: (() => boolean) | null; cancellable: boolean; - /** - * JSPI seam. When present the thread is not waiting on a scheduler - * condition at all — it is waiting for a **Promise**, namely the one a - * `promising`-wrapped wasm entry returned. The driving loop awaits it and - * resumes the body with the resolved value (or throws the rejection into - * the body, so a post-resume trap unwinds exactly like a synchronous one — - * jspi pin (e)). - * - * This is what lets one generator body serve both modes: in plain mode the - * core call returns a value and the body never yields such a request, so - * the synchronous path is bit-for-bit what it was before JSPI existed. - */ + /** Promise park, separate from scheduler readiness. Settlement resumes the + * generator with a value or throws the rejection into its unwind path. */ awaitValue?: Promise; } @@ -88,13 +42,9 @@ export interface BlockRequest { export type ThreadBody = Generator; /** - * Failure raised where the reference genuinely needs to suspend a wasm frame. - * - * This is deliberately *not* a `Trap`: the component is not at fault and the - * program is not ill-formed — our runtime is incomplete. Reporting it as a - * trap would let a conformance run score a missing capability as a correct - * rejection, which is the exact failure mode contracts/plan-format.md's - * error-phase split exists to prevent. + * Missing wasm-frame suspension capability, not a component `Trap`. + * Keep capability failures distinct from valid conformance rejections + * (contracts/plan-format.md's error-phase split). */ export class NeedsJspi extends Error { constructor(what: string) { @@ -108,21 +58,10 @@ export function needsJspi(what: string): never { } /** - * Failure raised when a synchronous entry into an instance would race a - * pending lift (contracts/embedder-api.md §"Functions and async", - * failure-ladder arm 2). - * - * In jspi mode a promising-wrapped entry settles through a microtask hop even - * when nothing suspended, and the hop-quiescence gate (exec/boundary.ts) - * defers Promise-surface calls that would enter during that window. A - * synchronous caller — a resource constructor, or the embedder's `sync()` - * adapter — cannot be deferred, so it refuses instead. - * - * Deliberately *not* a `Trap`, and deliberately raised BEFORE the instance is - * entered: nothing was entered, so there is nothing to poison. The refusal is - * transient — the instance stays enterable, and the call succeeds on retry - * once the in-flight activity settles, or immediately through the - * Promise-shaped surface, which defers rather than refusing. + * Transient, non-poisoning refusal before synchronous entry would race a + * pending JSPI result lift. Constructors and `sync()` cannot defer; retry + * after the hop settles or use the Promise-shaped surface, which waits + * (contracts/embedder-api.md §"Functions and async"). */ export class SyncEntryBusy extends Error { constructor(what: string) { @@ -146,12 +85,8 @@ export class PendingCapability extends Error { } /** - * Hook invoked when a trap breaks an instance's enter/leave bracket in - * `Store.tick` (instance poisoning — see the comment at the call site). - * task/streams.ts registers the stream/future-end retirement walk here - * (#66). An injection seam rather than an import: streams.ts (via - * waitable.ts) already imports this module, and a scheduler → streams import - * would make `CopyEnd extends Waitable` evaluation-order-sensitive. + * Stream/future-end retirement on instance poisoning. Registration avoids a + * scheduler -> streams -> waitable -> scheduler evaluation-order cycle. */ let onInstancePoisoned: | ((inst: { handles: Iterable }, cause: unknown) => void) @@ -195,17 +130,9 @@ export function setOnInstancePoisoned( } /** - * Additional poisoning observers, appended to the single `onInstancePoisoned` - * hook above (#292). Separate from it for the same evaluation-order reason - * the hook exists at all — the registrant (exec/boundary.ts, rejecting the - * pending Promises of async-typed lifts whose task will now never finish) - * already imports this module, and we must not import it back — but a Set - * rather than a second single slot, because "the" poisoning action is - * streams.ts's and this is strictly extra. - * - * Ordering is deliberate: the primary hook (stream/future-end retirement, - * #66) runs FIRST, so a listener that settles host-visible Promises observes - * ends already retired rather than ends about to be. + * Additional observers, including pending-lift rejection in exec/boundary.ts. + * They run after stream/future retirement returns. A throwing retirement hook + * or listener stops notification; the diagnostic map still retains the first cause. */ const instancePoisonedListeners = new Set< (inst: { handles: Iterable }, cause: unknown) => void @@ -227,21 +154,13 @@ export function notifyInstancePoisoned( inst: { handles: Iterable }, cause: unknown, ): void { - // First cause wins: a poisoned instance can collect follow-on failures - // (late settles retired against it, repeated bracket breaks), and the - // original trap is the one worth reporting on later entry refusals - // (polyengine#145 ask 1). + // Preserve the original cause across follow-on failures. if (!poisonedInstances.has(inst)) poisonedInstances.set(inst, cause); onInstancePoisoned?.(inst, cause); for (const f of instancePoisonedListeners) f(inst, cause); } -/** Poisoned instances → poisoning cause, for late-settle retirement - * (`Thread.resumeWith`) and entry-refusal diagnostics (`withPoisonCause`, - * polyengine#145). A WeakMap mirror of streams.ts's `retiredInstances`, kept - * here because thread.ts cannot import streams.ts (the same - * evaluation-order constraint that made `setOnInstancePoisoned` an - * injection seam). */ +/** Poison causes shared by late-settle retirement and entry diagnostics. */ const poisonedInstances = new WeakMap(); export function isInstancePoisoned(inst: object): boolean { @@ -249,25 +168,15 @@ export function isInstancePoisoned(inst: object): boolean { } /** - * The recorded cause of an instance's poisoning: the original trap that - * broke the enter/leave bracket (polyengine#145). `undefined` when the instance - * is not poisoned — and, degenerately, when the poisoning cause itself was - * a thrown `undefined`; use `isInstancePoisoned` for the predicate. + * Original poison cause. Use `isInstancePoisoned` to distinguish an unmarked + * instance from one whose cause was a thrown `undefined`. */ export function instancePoisonCause(inst: object): unknown { return poisonedInstances.get(inst); } /** - * Append the recorded poison cause to an entry-refusal trap message - * (polyengine#145 ask 1). "cannot enter component instance" has exactly one - * cause — a permanently poisoned instance, the corpse of an earlier trap — - * and this - * suffix names the trap that made it one. The call is kept unconditional at - * the refusal sites (returning `base` unchanged for an unmarked instance) so - * the message construction stays in one place; the suffix is - * conformance-safe because the official suite matches trap messages by - * substring (harness/src/runner.ts). + * Append the original poison cause, or leave `base` unchanged if unmarked. */ export function withPoisonCause(inst: object, base: string): string { if (!poisonedInstances.has(inst)) return base; @@ -276,26 +185,11 @@ export function withPoisonCause(inst: object, base: string): string { } /** - * The entry-refusal decision, in one place: may `caller` enter `callee` right - * now, and if not, what does the refusal trap say? Returns `null` when entry - * is allowed, otherwise the exact trap message for `base`. - * - * POISONING IS THE WHOLE MECHANISM (CM#705). There is no transient - * reentrance gate: at the pinned reference (definitions.py @ 2f13265) - * `may_enter`, `entering_set`, `enter_from`, `leave_to` and - * `ComponentInstance.parent` do not exist — `Store.lift` runs `canon_lift` - * with no gate at all, so host-mediated reentrance into a live instance is - * simply VALID. - * - * Against that, per-instance poisoning is polyengine's NAMED DIVERGENCE. A - * trapped instance is a corpse — entry is refused permanently, with the - * recorded cause appended (polyengine#145 ask 1) — where wasmtime instead - * kills the whole store. The reference never faces the question because a - * trap there is the end of the world. - * - * The `caller !== callee` guard keeps a self-call out of the refusal: a dtor - * invoked from inside its own instance (cabi/handles.ts) is the live case — - * it must not be refused by its own instance's marker. + * Return a refusal message for a poisoned callee, otherwise `null`. + * Live-instance reentry is allowed by definitions.py `Store.lift`; this + * runtime's per-instance poisoning policy is separate from task exclusivity + * and JSPI hop serialization. Same-instance calls, including guest self-drop + * destructors, bypass the poison refusal. */ export function entryRefusal( callee: object, @@ -347,11 +241,7 @@ export function schedulerSeedForTesting(value: number | null): void { } /** - * Test hook: snapshot the module's current seed without mutating it. Used by - * the `just sched-seeds` regression guard (sched_seed_guard_test.ts) to - * confirm `readSeed()` actually picked up `POLYENGINE_SCHED_SEED` from the - * environment at import time, rather than silently falling back to FIFO for - * lack of `--allow-env`. + * Test hook: inspect the seed, including whether environment access succeeded. */ export function schedulerSeedSnapshotForTesting(): number | null { return seed; @@ -374,9 +264,7 @@ function nextRandom(): number { } /** - * Pick one candidate. FIFO (index 0 — candidates are supplied in - * ready-order) unless a seed is configured, in which case a seeded uniform - * choice, mirroring the reference's `random.choice`. + * Pick the first supplied candidate, or a seeded pseudo-random candidate. */ export function chooseCandidate(candidates: readonly T[]): T { assert_(candidates.length > 0, "chooseCandidate on an empty candidate set"); @@ -385,17 +273,12 @@ export function chooseCandidate(candidates: readonly T[]): T { } // --------------------------------------------------------------------------- -// Current-thread context (definitions.py `current_thread`, line 306) +// Current-thread context (definitions.py `current_thread`) // --------------------------------------------------------------------------- /** - * The reference keeps the running thread in a thread-local - * (`thread_local_handler`). A JS generator has no such ambient slot, so the - * scheduler maintains an explicit stack: `resume()` pushes, and every - * canonical built-in reads the top. It is a stack rather than a single slot - * because a *host* import called from a guest can lift into another component - * instance, nesting one activation inside another exactly as the reference's - * recursive `store.lift` does. + * Synchronous execution brackets, innermost last. Nested host-mediated calls + * require a stack rather than a single current-thread slot. */ // deno-lint-ignore no-explicit-any const threadStack: any[] = []; @@ -419,13 +302,9 @@ export function popCurrentThread(t: CurrentThreadLike): void { } /** - * Run `fn` with `t` as the ambient, for `fn`'s SYNCHRONOUS extent. - * - * This is the wasm-entry bracket (`awaitCore`). It is the same `threadStack` - * the scheduler's own `resume()` bracket uses, deliberately: a wasm entry made - * from *inside* an engine-driven resumption (a FACT callee reached from a - * resumed activation — fact_calls.ts) has an empty scheduler bracket, and the - * entry itself is then the most specific statement of who is running. + * Run `fn` with `t` as ambient for its synchronous extent only. Wasm entries + * and built-in bodies use this even during engine-driven resumptions, when + * no scheduler `resume()` bracket remains. */ // deno-lint-ignore no-explicit-any export function withActivation(t: any, fn: () => T): T { @@ -441,26 +320,15 @@ export function withActivation(t: any, fn: () => T): T { } /** - * The WASM-ENTRY brackets alone — a subset of `threadStack`. - * - * Kept separately because it is the exact analogue of what the retired - * async-context store held: the store was written by `withActivation` and by - * nothing else, so a built-in reached under a scheduler `resume()` bracket - * that had not (yet) entered wasm saw NO store, even though `threadStack` - * named a thread. `Store.consumePendingIfRunning` — the driver-gate - * release whose scheduling effects the corpus pins precisely — asked exactly - * that question, so it must keep asking exactly that question — measured: - * routing it through the full `threadStack` instead moved 64 conformance - * commands. Ambient *resolution* is a different question and uses the full - * `threadStack`. + * `withActivation` brackets only. Consuming a pending resumption requires + * evidence of activation execution, not merely a scheduler generator step. + * Ambient resolution uses the broader `threadStack` instead. */ // deno-lint-ignore no-explicit-any const entryStack: any[] = []; /** - * "Whose wasm frame are we lexically inside, or running on behalf of?" — the - * async-context store's replacement, used only by - * `Store.consumePendingIfRunning`. + * Activation execution evidence used only by `consumePendingIfRunning`. */ // deno-lint-ignore no-explicit-any function activationOf(): any { @@ -473,93 +341,22 @@ function activationOf(): any { // --------------------------------------------------------------------------- /** - * ACTIVATIONS THE ENGINE IS RUNNING OUTSIDE OUR FRAMES — innermost last. - * - * =========================================================================== - * WHAT REPLACED THE ASYNC-CONTEXT STORE, AND WHY IT NEEDS NO ENGINE MAGIC - * =========================================================================== - * - * A wasm activation under JSPI does not stay inside our JS frames. Two - * distinct mechanics take it outside them, and BOTH are ours to observe: - * - * (i) A GENUINE SUSPENSION. A `Suspending`-wrapped built-in returned a - * Promise; the engine parks the activation and resumes it in a - * microtask of its own when that Promise settles. There is exactly one - * source of such a Promise in this runtime — `blockCurrentActivation` - * mints it, `SuspensionPoint.resume`/`.abandon` settle it — so the - * moment of resumption is ours, including for a **background - * activation** whose lifted call already returned (that resumption - * still runs through `SuspensionPoint.resume`, from `Store.tick`). - * - * (ii) THE MICROTASK HOP ON EVERY `Suspending` CALL — jspi pin (j), - * `tests/jspi/fastpath_hop_test.ts`. Even when the built-in produced - * its value synchronously and nothing suspended, the guest's frame - * resumes through a microtask, i.e. AFTER our `withActivation` bracket - * (and `callCore`, and the whole driving frame) has unwound. This one - * is easy to overlook because nothing looks asynchronous at the call - * site; it is nonetheless the dominant case, and the one that produced - * `exit-sync-call with an empty sync-call stack` when it was missed - * (trap-if-done.wast:448, big-interleaving-test.wast). - * - * Both are claimed explicitly — (i) in `SuspensionPoint.resume`, (ii) in the - * wrapper `suspendingImport` puts around every blocking-capable trampoline — - * naming the activation captured from the ambient while its bracket was still - * live. That is exactly the value the async-context store used to - * reproduce: the store was set by `withActivation` around the wasm entry, and - * the engine restored it because it had captured the context when it - * registered the continuation. We now record the same activation ourselves, - * at the same instant, by construction — no Node `async_hooks` builtin, no - * `AsyncContext` proposal, nothing beyond Promises (docs/architecture.md §4.3; M3A-1). - * - * NOTE ON ORDINARY `await`s. Nothing here needs a context to survive a plain - * `await` any more, and nothing ever did on its own merits: the driving loops - * (`drive`/`driveAsync` in exec/boundary.ts, the host-stream pump) run outside - * every activation and read no ambient. What they do is *resume* threads, and - * every resumption re-establishes the ambient explicitly — a scheduler-driven - * one through `Thread.#resumeInternal`'s `pushCurrentThread` bracket, an - * engine-driven one through this queue. - * - * LIFO, TOP-IS-CURRENT — and that direction is load-bearing, not incidental. - * Activations NEST: an outer activation's built-in can synchronously enter an - * inner activation's wasm (`async-start-call` running its callee through - * `awaitCore`), and the inner one is the one executing. Reading the OLDEST - * claim instead of the newest was measured at 45 conformance failures. - * - * The opposite shape — A settles B's suspension so B runs AFTER A — is - * deliberately NOT represented here: `SuspensionPoint.resume` pushes only when - * nothing is currently running, so B never shadows A. B is picked up by its - * own first `Suspending` call. A driver's settle-time claim would name B - * here, which is exactly why it is not an ambient tier — see `resolveAmbient` - * and `Store.pendingResumptions`. - * - * An activation leaves this stack when it parks again - * (`blockCurrentActivation`) or finishes (its `awaitValue` promise settles — - * `Store.noteAwaiting`). + * Engine-driven activation claims, innermost last. JSPI resumptions and + * even plain-value `Suspending` returns can run after synchronous brackets + * unwind (`tests/jspi/fastpath_hop_test.ts`). The bridge captures the owner + * before the hop and reclaims it with continuation sentinels. + * + * Nested execution moves the running owner to the top. Settling B while A + * still runs must not shadow A: `SuspensionPoint.resume` claims B immediately + * only when no ambient exists. Parking releases the claim; `noteAwaiting` + * releases it on completion or rejection. Driver awaits carry no ambient. */ // deno-lint-ignore no-explicit-any const activationClaims: any[] = []; /** - * Record that the engine will run `t`'s wasm outside our frames. - * - * Idempotent in MEMBERSHIP but not in POSITION: re-claiming MOVES an - * existing claim to the top. The stack's contract is "top = the innermost - * activation the engine is running outside our frames", and a re-claim is - * direct evidence that `t` is running RIGHT NOW (its Suspending import just - * returned into its wasm). The previous early-return kept stale order: a - * nested callee's claim whose release edge is a promise reaction - * (`Store.noteAwaiting` -> `Store.releasePendingOf`) outlives the callee by a - * microtask, and an outer activation's continuation chunk that resumed in - * that window re-claimed itself as a NOOP — leaving the finished callee on - * top, so every ambient read in the rest of the chunk (the next hop's - * `owner` capture, and any unsafe intrinsic like `context.set`, which has - * no hop to re-anchor on) answered the wrong thread. Found as issue #24: - * wit-bindgen's callback epilogue restored its task pointer into another - * thread's context slots, and the next disciplined callback invocation - * panicked on a null slot (async_support.rs:578). - * - * A null/undefined activation is "no claim" — the instantiation-time shape - * that has no thread at all. + * Move `t` to the top, including on re-claim: a nested callee's release + * reaction may lag behind the caller's next chunk. Nullish owners do nothing. */ // deno-lint-ignore no-explicit-any export function claimActivationAmbient(t: any): void { @@ -571,10 +368,10 @@ export function claimActivationAmbient(t: any): void { activationClaims.push(t); } -// #24 probe. +// Optional ambient tracing. // deno-lint-ignore no-explicit-any function traceAmbient(what: string, t: any): void { - // Lazy import avoidance: reuse context.ts's ids via a local map. + // Local diagnostic identities avoid another dependency on context.ts. console.error( `[amb] ${what} ${dbgId(t)} | stack=[${threadStack.map(dbgId).join(",")}] ` + `claims=[${activationClaims.map(dbgId).join(",")}]` + @@ -594,13 +391,8 @@ export function dbgId(t: unknown): string { } /** - * Drop `t`'s activation-ambient claim, if it holds one. - * - * The two closing edges: the activation PARKS on a fresh suspension - * (`blockCurrentActivation`), or it FINISHES — its `awaitValue` promise - * settles, normally or by rejection, and `Store.noteAwaiting`'s eager settle - * continuation calls this. The `task.implicitThread` indirection covers the - * second edge for claims taken against a task's implicit thread. + * Release on park or activation settlement, with an implicit-thread fallback + * for claims recorded through the task rather than this exact thread. */ // deno-lint-ignore no-explicit-any export function releaseActivationAmbient(t: any): void { @@ -618,14 +410,8 @@ export function releaseActivationAmbient(t: any): void { } // --------------------------------------------------------------------------- -// The resumed-but-not-yet-run gate (a SEPARATE concern from the ambient above) +// Ambient diagnostics // --------------------------------------------------------------------------- -// -// The gate answers one question for the DRIVER: "was a suspension settled -// whose activation has not run yet — must I refrain from scheduling anything -// else?" That is per-Store SET semantics, not a global identity slot, and it -// is not an input to ambient resolution: see `Store.pendingResumptions` -// below. const AMBIENT_TRACE = (() => { try { @@ -635,7 +421,7 @@ const AMBIENT_TRACE = (() => { } })(); -/** Diagnostic (#24 probe): the full ambient state, for tracing. */ +/** Full ambient state for tracing. */ export function ambientDebug(): { stack: unknown[]; claims: unknown[]; @@ -647,9 +433,7 @@ export function ambientDebug(): { } /** - * Diagnostic: module-scope AMBIENT state that must NOT survive a completed - * call. The scheduling gate is no longer module-scope — a store's - * `pendingResumptions` set is the per-Store analogue and is checked there. + * Module-scope ambient residue; scheduling gates live on each Store. */ export function ambientResidue(): { stack: number; claim: boolean } { return { @@ -659,48 +443,9 @@ export function ambientResidue(): { stack: number; claim: boolean } { } /** - * THE ambient precedence, in one place. Every reader goes through this. - * - * 1. `threadStack` -- a synchronous bracket we pushed ourselves: either - * `Thread.#resumeInternal`'s `resume()` bracket, `withActivation`'s - * wasm-entry bracket, or `suspendingImport`'s built-in-call bracket. - * Most specific: we are literally inside that activation's execution. - * 2. the TOP of `activationClaims` -- the innermost activation the engine - * is running outside our frames (a `Suspending` hop or a resumption). - * LIFO, because activations nest: an outer activation's built-in can - * synchronously enter an inner one's wasm. - * - * What tiers 1+2 state directly is "the innermost wasm activation currently - * executing, across the engine's hops and resumptions" -- the same quantity - * an async-context store written around the wasm entry would carry, without - * depending on the engine to restore it on every continuation captured inside - * that extent (M3A-1). The equivalence is not asserted from the armchair: it - * was established differentially, against such a store, over the whole - * conformance corpus, comparing at every read (zero disagreements over 1395 - * commands), and the corpus pins the result. - * - * Having TWO readers with different precedence orders is not a hypothetical - * hazard: they disagree silently at exactly the sites that matter -- the FACT - * bracket sites read `maybeCurrentThread`, so a divergent order there - * attributes the bracket to the driver's claim instead of its own activation - * (`exit-sync-call with an empty sync-call stack`), and a precedence fix - * applied to the other reader measures as "no change" because the failing - * sites never call it. Do not add a third reader; extend this one. (`activationOf` above is not a - * second reader -- it answers a different question, "whose wasm frame are we - * running on behalf of", and is used only by - * `Store.consumePendingIfRunning`.) - * - * TWO TIERS ARE ENOUGH, and specifically a driver's settle-time claim is NOT - * a third: such a claim names whichever activation was settled or claimed - * across an await -- right for that one and wrong for every other in-flight - * activation. It is not needed, because the sentinel discipline (tier 2's - * claim/release edges, #24) always answers first: instrumented reads where - * tiers 1-2 were empty and a settle-time claim was live decided NOTHING - * across the conformance corpus, both seeded shuffles - * (`POLYENGINE_SCHED_SEED` 1 and 4242), test-runtime and the smoke-tls - * three-async-component #24 corpus. A driver's settle-time claim is a - * SCHEDULING gate only, and lives as the per-Store `Store.pendingResumptions` - * set (issue #158). + * Ambient precedence: innermost synchronous bracket, then newest engine + * activation claim. `pendingResumptions` is never an ambient source: it + * names work owed a turn, not necessarily the activation executing now. */ function resolveAmbient(): CurrentThreadLike | undefined { return threadStack[threadStack.length - 1] ?? @@ -717,20 +462,9 @@ export function currentThread(): T { } const t = resolveAmbient(); if (t === undefined) { - // Reaching this is not an internal invariant violation, so it must not be - // an `AssertionError`: it is a *known incompleteness*. wasmtime lets a - // core module's start function call canonical built-ins during - // instantiation, before any task exists, and definitions.py has no model - // for that — `current_thread()` (line 306) simply presumes a running - // task, because in the reference a built-in is only ever reached from - // inside one. - // - // Instance-scoped built-ins already avoid this by taking their instance - // from the trampoline declaration (see intrinsics/async_builtins.ts). What - // lands here is a *task*-scoped built-in (task.return, task.cancel, - // thread.yield, subtask.*) called at instantiation time, which needs the - // instantiation-time task context the spec implies but does not spell out. - // Exercised by test/async/dont-block-start.wast:3. + // Task-scoped built-ins during core start need an instantiation-time + // task context this runtime does not implement. Instance-scoped built-ins + // can instead use their trampoline declaration. throw new PendingCapability( "instantiation-time task context — a task-scoped canonical built-in " + "ran outside any task (a core start function calling task.return / " + @@ -746,48 +480,14 @@ export function maybeCurrentThread(): CurrentThreadLike | undefined { } /** - * THE ambient, NARROWED BY THE INSTANCE WHOSE CORE FRAME IS EXECUTING. - * - * For a built-in whose declaration names a component instance, "who is - * running" is not an open question about the whole store: the call arrived - * from a core frame OF THAT INSTANCE, so the running activation is one of - * that instance's. This narrows `resolveAmbient` accordingly — same tiers, - * same order, candidates filtered — and falls back to the unscoped answer - * when the instance has no candidate at all (the instantiation-time shape, - * and any built-in reached before its instance has a task). - * - * WHY IT IS NEEDED (polyengine#24's residue; polyvisor#49 trap 1, - * `runtime/tests/context_attribution_test.ts`). A JSPI continuation chunk — - * the tail of a suspended activation, e.g. wit-bindgen's callback epilogue - * restoring its task pointer with `context.set` (rt/async_support.rs:592) — - * runs with an EMPTY `threadStack` and, unlike a hop, has no re-anchoring - * edge of its own. Tier 2 then answers the newest claim, which is whichever - * SIBLING activation suspended most recently. The attribution sentinels - * (jspi/bridge.ts) plant that claim one microtask ahead of the chunk, which - * is exact when the engine queues the resumption while the settle reaction - * returns (measured so in Deno's V8) — and NOT exact in Chromium, where a - * wider gap lets a sibling's sentinel land in between. Measured there 3/3: - * one task's epilogue wrote its state pointer into another task's slots, and - * the starved task's next callback entry hit `assert!(!state.is_null())` - * (async_support.rs:578) -> unreachable. - * - * Ordering discipline cannot fix that class — engine chunk boundaries are not - * observable, so every microtask-ordering scheme is a hope. Instance identity - * is not a hope: it is static (the declaration), and it is decisive because - * ONE INSTANCE CAN ONLY HAVE ONE ACTIVATION MID-FRAME AT A TIME — a callback - * invocation holds `inst.exclusiveThread` for its whole extent, suspensions - * included (definitions.py line 2187 / `runCallbackLoop`), and a sync or - * stackful-async lift holds the entry gate. Two activations that can race for - * an unbracketed read are therefore necessarily of different instances, which - * is exactly what this discriminates. - * - * SPEC BASIS. `canon_context_get`/`canon_context_set` (definitions.py 2348 / - * 2358) read `current_thread().storage`, and in the reference a built-in is - * only ever reached from inside the activation that called it — the identity - * is exact by construction, never inferred. This runtime has to reconstruct - * it; narrowing the reconstruction to the declaring instance moves it TOWARD - * the reference (it can only ever remove candidates the reference would never - * have named), never away. + * Resolve using the declaring instance: a matching top synchronous bracket, + * then the newest matching activation claim, then the unscoped fallback. + * Engine continuation timing can interleave sibling-instance sentinels; + * static instance identity removes those candidates + * (`runtime/tests/context_attribution_test.ts`). It does not distinguish + * concurrent activations of the same instance; their order still depends on + * the brackets and claims. In definitions.py `canon_context_get` and + * `canon_context_set`, identity comes directly from `current_thread`. */ // deno-lint-ignore no-explicit-any export function currentThreadForInstance( @@ -818,7 +518,7 @@ function instOf(t: any): unknown { return t?.task?.inst; } -/** definitions.py `current_task()` (line 309). */ +/** definitions.py `current_task`. */ // deno-lint-ignore no-explicit-any export function currentTask(): any { return currentThread().task; @@ -833,14 +533,14 @@ export function maybeCurrentTask(): any | null { return maybeCurrentThread()?.task ?? null; } -/** definitions.py `current_instance()` (line 312). */ +/** definitions.py `current_instance`. */ // deno-lint-ignore no-explicit-any export function currentInstance(): any { return currentTask().inst; } // --------------------------------------------------------------------------- -// Store (definitions.py `class Store`, line 562) +// Store (definitions.py `Store`) // --------------------------------------------------------------------------- /** Structural view of a Thread, as the store's ready queue needs it. */ @@ -853,14 +553,8 @@ export interface SchedulableThread { } /** - * The embedder-visible scheduler state (definitions.py `Store`). One per - * instantiated component in this runtime — the reference shares one `Store` - * across component instances of a linked graph, and so do we: `Executor` - * creates a single `Store` and hands it to every `ComponentInstanceState`. - * - * `waiting` is kept as an **array, in insertion order**, which is what makes - * the default policy FIFO: `readyCandidates()` preserves the order in which - * threads started waiting. + * Scheduler state shared by the component instances of an Executor. + * `waiting` preserves insertion order for the default candidate policy. */ export class Store { readonly waiting: SchedulableThread[] = []; @@ -874,54 +568,23 @@ export class Store { readonly pendingHostCalls: Set> = new Set(); /** - * An exception raised by a host import's promise (a rejection, or a trap - * thrown while lowering its results). It cannot propagate out of the - * microtask that produced it, so it is parked here and rethrown by whoever - * is driving the store — which is the call the guest is blocked in. + * Asynchronous host or background-driver failure, parked for a driver to + * surface. The consuming driver need not belong to the originating call. */ hostFailure: unknown = undefined; /** - * Resumed-but-not-yet-run activations of THIS store — the driver's - * scheduling gate, not an ambient. - * - * Keeping this distinct from `activationClaims` matters. This set answers - * "may I schedule something else right now?" (`Store.tick` and both driving - * loops refuse while it is non-empty, which is what forces a microtask yield - * so the resumed activation actually runs). `activationClaims` answers - * "whose code is this?". Conflating them — driving off the ambient queue — - * wedges the loops, because an activation that merely hopped legitimately - * holds an ambient while the scheduler is free to proceed. - * - * PER-STORE and MULTI-ENTRY (issues #158 mechanism B, #210), both load - * bearing. MULTI-ENTRY because two engine resumptions can legitimately be - * pending at once: a running activation X may deliver a resume to Z while - * Y's resumption is still outstanding, and a one-claimant gate cannot - * represent that. PER-STORE because a claim held store-wide makes every - * driver on EVERY store yield: an idle store's `driveStoreAsync` dies at - * the 10,000-hop assert (~311ms) while another store merely dwells on a - * slow host import. - * - * Cross-store de-serialization is safe by disjointness: an activation - * belongs to exactly one store. Same-store the set is conservative — the - * gate keeps refusing until EVERY pending entry has died. - * - * Release edges, per entry: the activation PARKS again - * (`blockCurrentActivation` -> `consumePendingIfRunning`), it FINISHES (its - * `awaitValue` promise settles -> `noteAwaiting` -> `releasePendingOf`), or - * the driver drops its own speculative entry (`removePendingResumption`). + * Per-store scheduling gate, not an ambient source. Several resumptions + * can be outstanding; `tick` waits until all entries are released, without + * blocking independent stores. An entry ends when its activation runs or + * parks (`consumePendingIfRunning`), finishes (`noteAwaiting`), or the + * driver removes its own speculative entry. */ readonly pendingResumptions: Set = new Set(); /** - * Record that a suspension of this store has been settled and its - * activation has not run yet. Idempotent; a null/undefined activation is - * "no entry" (the instantiation-time shape that has no thread at all). - * - * No one-claimant assert: two entries are legitimate (see - * `pendingResumptions`). Two SuspensionPoints of ONE task cannot be pending - * simultaneously — a task's single activation suspends at one point at a - * time — so collapsing entries by identity loses nothing. + * Record an activation owed a turn. Idempotent by identity; nullish values + * do not create entries. */ addPendingResumption(t: unknown): void { if (t === null || t === undefined) return; @@ -941,19 +604,8 @@ export class Store { } /** - * Drop the pending entry iff its activation is demonstrably RUNNING — i.e. - * the entry names the same thread the ACTIVATION AMBIENT names for the code - * calling us. An entry exists to cover the window between settling a - * suspension and the resumed activation running; once that activation's own - * code is on the stack the window is closed, and holding the entry would - * gate the store on an activation that has already had its turn — while a - * running activation's built-in settles ANOTHER activation's suspension - * (`subtask.cancel` delivering a cancellation to a parked callee, - * cancellable.wast) that other entry must legitimately stay. - * - * The comparison is against `activationOf()` — the wasm-ENTRY brackets, - * deliberately not the full `threadStack` (see `entryStack`: routing it - * through the full stack moved 64 conformance commands). + * Release only the executing activation's entry, not another activation + * it may have just resumed. `activationOf` excludes scheduler-only brackets. */ consumePendingIfRunning(): void { const a = activationOf(); @@ -961,15 +613,8 @@ export class Store { } /** - * Drop the pending entry naming `t` — the settle-side half: an entry taken - * when `t`'s suspension was settled dies when `t`'s activation finishes (its - * `awaitValue` promise settles; `noteAwaiting` calls this from the eager - * settle continuation) or parks again (`blockCurrentActivation` consumes via - * `consumePendingIfRunning`). - * - * The `task.implicitThread` indirection covers entries taken against a - * task's implicit thread. `t` FINISHING also ends its activation ambient, - * so both are dropped here. + * Activation settlement ends both ambient and scheduling claims, including + * claims recorded against the task's implicit thread. */ // deno-lint-ignore no-explicit-any releasePendingOf(t: any): void { @@ -994,15 +639,8 @@ export class Store { } /** - * Ready waiting threads, in wait order (the FIFO of the default policy). - * - * A POISONED instance's threads are not candidates: they are a corpse's and - * must never resume (polyengine's per-instance poisoning divergence). The - * filter lives here, not in `tick` alone, because the answer is also a - * VERDICT elsewhere — the drivers' deadlock probe asks "did anything become - * ready?" and must get an answer that agrees with what `tick` will actually - * run, or it re-arms forever on a thread `tick` refuses (exec/boundary.ts's - * probe, against `canon_lift`'s `trap_if(not candidates)`). + * Ready, non-poisoned threads in wait order. Drivers use this same filter + * for deadlock probes so readiness agrees with what `tick` can resume. */ readyCandidates(): SchedulableThread[] { return this.waiting.filter((t) => @@ -1019,20 +657,11 @@ export class Store { readonly awaiting: Set = new Set(); /** - * Settled-but-unserviced activation tails, in settle order. - * - * A settled `awaitValue` is the rest of an activation that already finished - * its wasm: result shaping, the callback loop, `exit_implicit_thread` (and - * with it the exclusive-thread release). The reference runs all of that - * atomically inside `Thread.resume`; under jspi it lands a few engine - * microtasks after the observable effects of the activation (`task.return` - * flips `resolved` DURING the wasm, the settle only afterwards — jspi - * pin (j)). Any scheduling decision taken in that window sees phantom - * state — a finished callee still "holding" its exclusive slot made - * cancellable.wast report STARTING for an entry the reference admits. So - * settlement is recorded EAGERLY (at park time, below), `tick` refuses to - * run anything while a tail is unserviced, and the driving loop services - * this queue first. + * Settled activation tails in settlement order. Result lifting, callback + * dispatch and implicit-thread exit follow the wasm call; JSPI separates + * them by microtasks where definitions.py `Thread.resume` keeps them in one + * step. Record settlement eagerly and service tails before `tick` so later + * threads do not observe unfinished bookkeeping or unreleased exclusivity. */ readonly settled: { // deno-lint-ignore no-explicit-any @@ -1042,14 +671,8 @@ export class Store { }[] = []; /** - * Park `t` on `promise` (jspi `awaitValue`), with EAGER settle tracking. - * - * The `.then` here is also what closes the claim discipline for - * resumptions the driver did not settle itself (a guest built-in resolving - * another activation's suspension — `subtask.cancel` delivering a - * cancellation): the claim taken at settle time must survive until the - * resumed activation parks again or finishes, and "finished" is exactly - * this continuation firing. See `releasePendingOf`. + * Track settlement at park time, including resumptions initiated by guest + * built-ins rather than a driver. Completion or rejection releases claims. */ // deno-lint-ignore no-explicit-any noteAwaiting(t: any, promise: Promise): void { @@ -1067,25 +690,10 @@ export class Store { } /** - * Service settled activation tails. Returns whether anything ran. EVERY - * driving loop must call this before (and interleaved with) `tick` — the - * queue gates `tick`, so a driver that never services it wedges the store - * (observed: host-stream pumping between export calls). A `resumeWith` may - * throw (trap unwinding); callers propagate or park it exactly as they do - * for `tick`. - * - * Every non-stale tail is dispatched immediately, in queue order: there is - * no enterability condition to defer on (CM#705). - * - * The ordering discipline is therefore settle order, full stop — and it is - * the reason this queue exists rather than a direct resumption from the - * settle continuation: in definitions.py the tail runs atomically inside - * the entered bracket, so the phantom-state gate (`tick` refuses while an - * unserviced tail is queued, see `hasServiceableSettled`) is what keeps a - * parked activation's tail from being observed out of order. - * - * A POISONED instance's tail is dispatched like any other: `resumeWith`'s - * poison early-return retires it, so it drains rather than leaking. + * Dispatch tails in queue order without an entry-lock test. Every driver + * must service this queue before and between ticks; exceptions propagate + * to that driver. Stale entries are discarded, and `resumeWith` retires + * poisoned-instance tails without running their bodies. */ serviceSettled(): boolean { let did = false; @@ -1095,8 +703,7 @@ export class Store { scan: for (;;) { for (let i = 0; i < this.settled.length; i++) { const s = this.settled[i]; - // Stale: the thread was resumed elsewhere (driveAsync's race-winner - // path). Drop it regardless of enterability; it is not progress. + // Another driver already resumed this thread. if (!this.awaiting.has(s.t)) { this.settled.splice(i, 1); continue scan; @@ -1114,60 +721,20 @@ export class Store { } /** - * "Would a `serviceSettled` call make progress right now?" — i.e. is any - * entry queued at all. Every entry either dispatches or is dropped as - * stale, so a non-empty queue always makes progress. - * - * It exists to gate `tick` (and to keep the driving loops from parking) - * behind unserviced tails: resuming some other thread while a settled tail - * waits would expose the out-of-order state the queue is there to prevent. + * Every queued entry can dispatch or be discarded as stale. A non-empty + * queue therefore gates `tick` and prevents drivers from parking. */ hasServiceableSettled(): boolean { return this.settled.length > 0; } /** - * "Does component instance `inst` still have runnable work?" — the - * drain-to-quiescence predicate behind the **deferred entry decision** - * (issue #43). - * - * wasmtime decides an async-lowered call's initial status only after the - * executor has drained the work queued ahead of it: a queued - * `GuestCall(StartImplicit)` is popped, and if `is_ready` is false - * (`do_not_enter || backpressure`) the caller is told STARTING - * (concurrent.rs :1497-1522, :3040-3160). That formulation is FIFO-order - * dependent; polyengine uses the order-robust restatement (issue #43): *the - * call reports STARTING only if the callee is still unstarted after the - * instance's runnable work has been exhausted* — drain to quiescence, not - * pop-one. That is what keeps `sync-streams.wast` green under - * `POLYENGINE_SCHED_SEED` shuffles, which wasmtime's own rule would not be. - * Adjudicated 2026-08-10 (issue #43): entry-status timing is NOT - * normative — this predicate implements a scheduler *policy*, picked so - * the suite's schedule-overfitted STARTED assertion holds under any - * seed; the hold-rule gate itself is the spec semantics. - * - * "Runnable work of `inst`" is, exhaustively: - * - * (a) a settled-but-unserviced activation tail (`settled`) — bookkeeping - * the reference runs atomically inside `Thread.resume`, so the - * instance is mid-step, not quiescent; - * (b) a waiting entry (thread or `SuspensionPoint`) of `inst` that is - * `ready()` — the scheduler will resume it on the next tick. A gate - * holder parked mid-frame on an un-rendezvous'd operation is NOT - * ready and therefore contributes nothing: that is the "holder - * cannot be drained" case, whose answer is STARTING; - * (c) a thread of `inst` in `awaiting` whose promise is not a scheduler - * park — i.e. genuinely in flight across an engine microtask hop. - * A JSPI-parked activation appears in `awaiting` *and* owns a - * `SuspensionPoint` in `waiting` (`SuspensionPoint.owner`), and is - * accounted for by (b) instead; counting it here would make the - * instance permanently non-quiescent. - * - * `excludeTask` is the CALLER's task, and is excluded everywhere: the - * caller cannot be drained — it is the activation asking the question. - * This is what makes the "only obstacle is the current running activation" - * shape (a nested lower from inside the gate holder's own invocation) - * answer STARTING immediately, with no park at all. + * Work to drain before deciding an async callee is still STARTING: + * queued tails, ready waiters, and awaiting threads crossing an engine + * hop. A thread owning a SuspensionPoint is genuinely blocked, so only + * that point's readiness counts. Exclude the caller's task throughout: + * it cannot be drained while asking this question. This is the runtime's + * entry-status scheduling policy, not an additional spec entry gate. */ hasRunnableWork(inst: unknown, excludeTask: unknown): boolean { // deno-lint-ignore no-explicit-any @@ -1195,71 +762,25 @@ export class Store { } /** - * definitions.py `Store.tick` (@ 2f13265): resume one ready thread. There - * is no bracket and no gate — the reference body is exactly "pick a ready - * thread, resume it" (CM#705). - * - * Returns false when no thread was ready, so callers can distinguish - * "made progress" from "stuck" without inspecting the queue themselves. + * Resume one ready thread, following definitions.py `Store.tick` with + * JSPI ordering and poison filters. False means no step ran, including + * when a pending resumption or queued tail must be serviced first. */ tick(): boolean { - // One suspension resolved per turn. - // - // Settling a suspension hands control to wasm in a *microtask*, not - // synchronously — so `tick` returns with the resumed activation not yet - // run and its pending entry still outstanding. Resolving a second one - // before that happens would let the first activation's built-ins - // attribute themselves to the wrong task (observed as `exit-sync-call` - // popping another task's bracket). Refusing to make progress while an - // entry is pending forces the caller to yield to the microtask queue - // first, which is exactly what `driveAsync` does. - // - // THIS STORE's entries only (issue #210): activations never cross stores, - // so another store's pending resumption says nothing about what this one - // may schedule. + // Let this store's settled suspensions reach their engine continuations + // before scheduling another thread. if (this.pendingResumptions.size > 0) return false; - // Same discipline, other edge: a settled-but-unserviced activation tail - // (see `settled`) is mid-"atomic resume" from the reference's point of - // view; scheduling anything before servicing it acts on phantom state. - // That is settle-order discipline and has nothing to do with reentrance. - // `hasServiceableSettled` (rather than - // "queue non-empty") only because a tail whose thread was already resumed - // elsewhere must not wedge the store. + // Finish queued bookkeeping before observing readiness. if (this.hasServiceableSettled()) return false; - // Ready is sufficient — almost. Nothing filters this set for reentrance: - // at the pinned reference (definitions.py @ 2f13265) `Store.tick` resumes - // any ready thread with no gate and no bracket (CM#705), so a sibling - // instance's thread going ready while another instance is entered from - // the host is simply resumable. - // - // What is added is polyengine's per-instance poisoning divergence: a - // poisoned instance is a corpse, its threads must never resume, and the - // MARKER is the whole test. That filter lives in `readyCandidates` (so - // the drivers' deadlock probe reads the same candidate set this does). - // `Thread.resumeWith` makes the same call on the tail path. const candidates = this.readyCandidates(); if (candidates.length === 0) return false; const thread = chooseCandidate(candidates); const inst = thread.task.inst; - // A trap out of the resumption poisons the instance (polyengine's named - // divergence: a per-instance corpse where wasmtime kills the whole store). - // Capability signals are the exception: a `NeedsJspi`/`PendingCapability` - // marks an operation this runtime cannot perform, not a component fault — - // in the reference that operation blocks and then completes, so poisoning - // here would turn one unsupported operation into a permanently dead - // instance. + // Capability failures do not poison; other escaping failures do. try { thread.resume(); } catch (e) { if (!(e instanceof NeedsJspi) && !(e instanceof PendingCapability)) { - // Poisoned: its live stream/future ends can never rendezvous again — - // retire them so parked host peers settle instead of hanging (#66). - // - // Routed through `notifyInstancePoisoned` (not the raw hook) so the - // poison MARKER is recorded too (polyengine#145): `Thread.resumeWith`'s - // quiet-retire of late settled tails (#156) and `entryRefusal` both - // read it; without the marker a settled tail of this dead instance - // would be resumed as if healthy. notifyInstancePoisoned( inst as unknown as { handles: Iterable }, e, @@ -1276,14 +797,9 @@ export class Store { // --------------------------------------------------------------------------- /** - * Host-activity "arm" promises, by identity: entries a driver parks in - * `Store.pendingHostCalls` purely to say "the embedder may still act". They - * are NOT outstanding work — treating them as such is the "activity keeps - * `pendingHostCalls` non-empty forever" hazard documented in - * exec/host_streams.ts — so the between-calls drivers filter them out via - * `hasRealHostCall`/`realHostCalls`. The registry lives here (rather than in - * exec/host_streams.ts, which mints the arms) so exec/boundary.ts's - * settlement pump can share the classification without an import cycle. + * Activity arms mean the embedder may still act, not that it owes a result. + * Between-call pumps exclude them from outstanding-work checks. Shared here + * to avoid a boundary/host_streams import cycle. */ const hostActivityArms = new WeakSet>(); @@ -1310,15 +826,8 @@ export function realHostCalls(store: Store): Promise[] { } /** - * Is there anything left that only a turn of the event loop could advance? - * Activity arms do not count: they say "the embedder may still act", which is - * precisely the state in which a between-calls driver should stop and let the - * operation's promise stay pending (the documented hang, exec/host_streams.ts - * module header). - * - * `store.settled` (settled-but-unserviced activation tails) DOES count: it - * gates `tick`, so exiting with a tail queued is a lost wakeup — the store is - * wedged until some other driver appears. + * No queued tails, awaiting activations, or real host calls. Activity arms + * and ready waiting threads do not count; callers must drain ticks first. */ export function storeQuiescent(store: Store): boolean { return store.settled.length === 0 && store.awaiting.size === 0 && @@ -1326,19 +835,9 @@ export function storeQuiescent(store: Store): boolean { } /** - * The reference's `canon_lift` sync driving loop (definitions.py, lines - * 2190-2192, post-CM#705): - * - * ```python - * while task.state != Task.State.RESOLVED: - * candidates = { t for t in inst.threads if t.ready() } - * trap_if(not candidates) - * random.choice(list(candidates)).resume() - * ``` - * - * Note the candidate set is `inst.threads` — threads *of the callee instance* - * — with no exclusion (CM#705 dropped the prior `exclusive_thread` carve-out), - * and that an empty set is a **trap** (the spec's deadlock trap), not a hang. + * definitions.py `canon_lift`'s sync loop: drive ready threads of the callee + * instance until resolution, trapping if none are ready. Unlike the host + * driver, this neither drains the whole store nor awaits JSPI microtasks. */ export function driveSyncLift( task: { diff --git a/runtime/src/task/streams.ts b/runtime/src/task/streams.ts index a133aba..07a501d 100644 --- a/runtime/src/task/streams.ts +++ b/runtime/src/task/streams.ts @@ -1,48 +1,12 @@ // Streams, futures and error-context: the async *value* types // (definitions.py `### Stream State`, `### Future State`, `class ErrorContext`). // -// =========================================================================== -// THE RENDEZVOUS, AND HOW IT MAPS ONTO OUR TASK CORE -// =========================================================================== -// -// A stream is not a buffer. `SharedStreamImpl` (definitions.py line 997) holds -// at most **one pending side** — a reader waiting for data, or a writer -// waiting for a reader — and a copy happens only when the second side arrives. -// That is the whole model: -// -// * first side to call `read`/`write` finds `pending_buffer == None` and -// parks itself via `set_pending(...)`; -// * second side finds a pending buffer, copies `min(remain, remain)` -// elements directly between the two guests' linear memories, and notifies -// *both* sides; -// * either side may be partially satisfied — that is not an error, it is the -// normal case, and the progress count is what the guest is told. -// -// Nothing here needs a scheduler: the copy is synchronous inside whichever -// call arrives second. What the scheduler provides is the *waiting*: a parked -// side has `CopyState.COPYING` and its `CopyEnd` (a `Waitable`) carries the -// pending event that wakes the guest. -// -// So the mapping to our task core is small and mechanical: -// -// reference here -// ------------------------------------------------------------------ -// CopyEnd(Waitable) CopyEnd extends our Waitable — the same -// base the SUBTASK path already uses, so -// waitable sets, `waitable.join` and the -// callback loop's WAIT code all work unchanged -// set_pending_event(thunk) the same thunk indirection: the event payload -// (progress, result) is computed at *delivery* -// time, exactly as `Subtask.setSubtaskPendingEvent` -// already does. This is why the phase-1 decision -// to keep events as thunks rather than values -// pays off here with no generalization at all. -// STREAM_READ / STREAM_WRITE EventCode values, already defined -// FUTURE_READ / FUTURE_WRITE -// -// The one genuinely new thing is `Buffer`: a cursor over a guest's linear -// memory that can be *partially* consumed, which is what makes partial copies -// expressible. +// A shared stream holds at most one pending side, not a queue of values. +// The opposite side copies min(source.remain(), destination.remain()) +// synchronously on arrival. Partial progress is normal; the pending side's +// onCopy decides when to reclaim its buffer. CopyEnd waitables carry guest +// notifications, whose progress/result payloads are evaluated at delivery. +// Host buffers and direct byte sessions use the same rendezvous. import { defineBrand, ERROR_CONTEXT } from "@polyengine/protocol"; import { assert_, Trap, trapIf } from "../cabi/trap.ts"; @@ -56,10 +20,8 @@ import type { ComponentValue, ValType } from "../cabi/types.ts"; import { Waitable } from "./waitable.ts"; import { isInstancePoisoned, setOnInstancePoisoned } from "./scheduler.ts"; -/** Structural element-type equality (`null` = the zero-width payload). - * Delegates to `valTypeEqual`: naive `JSON.stringify` comparison throws on - * resource-bearing element types (cabi/types.ts `valTypeEqual` contract - * note; found by the #18 polymorph-tls smoke). */ +/** Structural element-type equality, retaining nominal resource identity. + * `null` denotes the zero-width payload. */ export function sameElemType(a: ValType | null, b: ValType | null): boolean { if (a === null || b === null) return a === b; return valTypeEqual(a, b); @@ -69,22 +31,19 @@ export function sameElemType(a: ValType | null, b: ValType | null): boolean { export const BUFFER_MAX_LENGTH = 2 ** 28 - 1; /** - * One rendezvous chunk. u8 payloads travel as `Uint8Array` — the lift out of - * guest memory and the conventions layer's lowering both produce typed - * chunks, and every buffer in the copy path keeps them whole (issue #54: the - * typed shape is what makes both the guest-memory store and a host→host - * hand-off bulk). Every other element type travels as a plain array. + * One rendezvous chunk. u8 payloads stay `Uint8Array` through the bulk-copy + * path; other element types travel as plain arrays. */ export type PayloadChunk = ComponentValue[] | Uint8Array; -/** definitions.py `CopyResult` (line 977). */ +/** definitions.py `CopyResult`. */ export enum CopyResult { COMPLETED = 0, DROPPED = 1, CANCELLED = 2, } -/** definitions.py `CopyState` (line 1075). */ +/** definitions.py `CopyState`. */ export enum CopyState { IDLE = 1, COPYING = 2, @@ -97,7 +56,7 @@ export type OnCopy = (reclaim: ReclaimBuffer) => void; export type OnCopyDone = (result: CopyResult) => void; // --------------------------------------------------------------------------- -// Buffers (definitions.py `class BufferGuestImpl`, line 930) +// Buffers (definitions.py `BufferGuestImpl`) // --------------------------------------------------------------------------- /** @@ -171,19 +130,13 @@ export class GuestBuffer { this.progress += vs.length; } - // --- direct-access byte edges (embedder-api.md §"Streams and futures" (polyengine#128)) --- - // - // `ByteWindow`, implemented for the `stream` case only. The two methods - // together are the copy `read`/`write` would have done, split so that the - // *peer's* callback performs it: `byteView` hands out the range, and - // `advanceBytes` records the bytes that actually moved. They are role-blind - // (destination or source) because `this.ptr` already advances on BOTH - // `read` and `write` above, and `elemSize(u8) === 1`. + // ByteWindow for stream: the peer callback moves bytes through a view, + // then advanceBytes records the acknowledged progress in either direction. /** * A fresh view over the next `n` bytes of this buffer's remaining range. * - * Fresh on every call, via `bytesOf` (cabi/memory.ts:195) over the + * Fresh on every call, via `bytesOf` over the * `LiveMemory` getters — so a `memory.grow` between two rendezvous of one * parked direct session never yields a view onto the detached buffer. */ @@ -214,23 +167,13 @@ export class GuestBuffer { // The direct-access seam (embedder-api.md §"Streams and futures" ("Direct-access byte edges") (polyengine#128)) // --------------------------------------------------------------------------- // -// direct-access byte edge lets ONE side of a rendezvous be a *direct session*: instead of handing -// the rendezvous a buffer to copy out of / into, the host parks a callback -// that runs synchronously inside the rendezvous and performs the canonical -// copy itself, against a scoped view of the peer's memory. -// -// The seam below is the whole of it inside this file. It exists so that the -// rendezvous keeps mirroring definitions.py `SharedStreamImpl.read`/`.write` -// (lines 1032/1050) line for line for every non-direct path: when neither -// side is direct, `rendezvousCopy` IS `dst.write(src.read(n))`, unchanged. -// -// Nothing here imports from `exec/`: a direct session is recognised -// structurally (`direct === true`) and driven through two small optional -// protocols — `DirectBuffer` (the session) and `ByteWindow` (the peer). +// One side may supply a synchronous callback instead of a buffer. It moves +// bytes through the peer's scoped ByteWindow; ordinary copies still use +// dst.write(src.read(n)). Structural interfaces avoid an import from exec/. /** - * The buffer surface the rendezvous actually uses (definitions.py `Buffer`, - * line 918). `GuestBuffer` and the host layer's `HostBuffer` both satisfy it. + * The buffer surface used by the rendezvous (definitions.py `Buffer`). + * Both `GuestBuffer` and the host layer's `HostBuffer` satisfy it. */ export interface RendezvousBuffer { remain(): number; @@ -308,10 +251,8 @@ function isDirectBuffer(b: RendezvousBuffer): b is DirectBuffer { /** * The one copy site, shared by `SharedStreamImpl.read` and `.write`. * - * Collapses to definitions.py's `dst_buffer.write(src_buffer.read(n))` - * whenever neither side is a direct session — which is every guest↔guest, - * guest↔host-chunk and host-chunk↔host-chunk rendezvous, i.e. everything - * that existed before direct-access byte edge. + * Uses the reference's `dst_buffer.write(src_buffer.read(n))` unless one + * side supplies a direct callback. Two direct sessions have no backing buffer. */ function rendezvousCopy( src: RendezvousBuffer, @@ -341,7 +282,7 @@ function bothDirectError(): TypeError { } /** - * definitions.py `none_or_number_type` (line 1070). Guards the "temporary" + * definitions.py `none_or_number_type`. Guards the "temporary" * same-instance restriction below. */ function noneOrNumberType(t: ValType | null): boolean { @@ -364,7 +305,7 @@ function noneOrNumberType(t: ValType | null): boolean { } // --------------------------------------------------------------------------- -// The shared stream (definitions.py `class SharedStreamImpl`, line 997) +// The shared stream (definitions.py `SharedStreamImpl`) // --------------------------------------------------------------------------- /** Common shape of the object a `stream`/`future` *value* refers to. */ @@ -407,13 +348,9 @@ export class SharedStreamImpl implements SharedBase { pendingOnCopyDone: OnCopyDone | null = null; /** - * Observers fired once, when this stream becomes dropped — by EITHER - * side, including the loud component fault teardown walk (`dropSharedForTeardown`). The - * embedder's producer pump uses this to cancel a producer parked on an - * external event (§"Streams and futures"'s cancellation companion: an - * accept-shaped producer holds a live platform resource while parked, - * and the reader dropping is its only stop signal). `null` = already - * fired. + * Fired once on either end's drop, including teardown. Releases host + * activity and notifies producers parked on external events, where no + * short write can signal reader loss. `null` means already fired. */ #onDropped: (() => void)[] | null = []; @@ -478,7 +415,7 @@ export class SharedStreamImpl implements SharedBase { } } - /** definitions.py `SharedStreamImpl.read` (line 1032). */ + /** definitions.py `SharedStreamImpl.read`. */ read( inst: unknown, dstBuffer: GuestBuffer, @@ -495,7 +432,6 @@ export class SharedStreamImpl implements SharedBase { if (this.pendingBuffer.remain() > 0) { if (dstBuffer.remain() > 0) { const n = Math.min(dstBuffer.remain(), this.pendingBuffer.remain()); - // direct-access byte edge seam (#128). `"chunk"` is the reference line verbatim. const pendingIsDirect = isDirectBuffer(this.pendingBuffer); const out = rendezvousCopy(this.pendingBuffer, dstBuffer, n); if (out === "both-direct") { @@ -528,7 +464,7 @@ export class SharedStreamImpl implements SharedBase { } } - /** definitions.py `SharedStreamImpl.write` (line 1050). */ + /** definitions.py `SharedStreamImpl.write`. */ write( inst: unknown, srcBuffer: GuestBuffer, @@ -545,7 +481,6 @@ export class SharedStreamImpl implements SharedBase { if (this.pendingBuffer.remain() > 0) { if (srcBuffer.remain() > 0) { const n = Math.min(srcBuffer.remain(), this.pendingBuffer.remain()); - // direct-access byte edge seam (#128). `"chunk"` is the reference line verbatim. const pendingIsDirect = isDirectBuffer(this.pendingBuffer); const out = rendezvousCopy(srcBuffer, this.pendingBuffer, n); if (out === "both-direct") { @@ -573,9 +508,8 @@ export class SharedStreamImpl implements SharedBase { } else if ( srcBuffer.isZeroLength() && this.pendingBuffer.isZeroLength() ) { - // Zero-length rendezvous: both sides are empty, which is a *completed* - // handshake rather than a parked write (definitions.py line 1064 — - // the case `test/async/zero-length.wast` exists to pin). + // Two empty buffers complete the arriving write's handshake. + // See the reference's test/async/zero-length.wast. onCopyDone(CopyResult.COMPLETED); } else { this.resetAndNotifyPending(CopyResult.COMPLETED); @@ -585,29 +519,11 @@ export class SharedStreamImpl implements SharedBase { } /** - * direct-access byte edge (#128): route a rendezvous whose direct session did NOT copy. - * - * Two outcomes land here, and both share one invariant: the peer's parked - * operation survives, no event is delivered, and the stream is not dropped - * — a runtime never emits a zero-progress COMPLETED copy, which is - * unreachable in definitions.py for a nonzero-capacity operation. - * - * * `"retracted"` — `"done"` with zero marked. The session ends and - * resolves with its running total, through the ordinary - * `on_copy_done(COMPLETED)` channel. - * * `"failed"` — misuse or a throwing callback. The session has ALREADY - * rejected (`DirectSession.#fail`), so it must be retired silently: - * its rejection is its notification. - * - * Which side was the session decides where each goes, and both shapes are - * states definitions.py already produces: - * - * * PARKED session ⇒ the "the parked side had nothing left" branch - * (definitions.py:1043/1063): retire it and park the arriving - * operation, which gets no event either way. - * * ARRIVING session ⇒ the "arriving buffer of zero capacity" state - * (definitions.py:1041/1057): the pending side is left untouched with - * its `on_copy` unfired, and the arriving side completes. + * Retire a direct session without reporting a copy to its peer. Retraction + * resolves the session with its total; failure has already rejected it. + * If the session was pending, replace it with the arriving peer. Otherwise + * leave the pending peer untouched. Neither path drops the stream or emits + * a zero-progress completion to the peer's nonzero-capacity operation. */ #routeDirectNoCopy( out: "retracted" | "failed", @@ -650,7 +566,7 @@ export class SharedStreamImpl implements SharedBase { } } -/** definitions.py `class SharedFutureImpl` (line 1119). Exactly one element. */ +/** definitions.py `SharedFutureImpl`. Exactly one element. */ export class SharedFutureImpl implements SharedBase { /** * Optional hook fired when this shared object is lowered into a component @@ -678,26 +594,11 @@ export class SharedFutureImpl implements SharedBase { dropped = false; /** - * Set when the future's **writable** side went away without ever delivering - * its one value (#84 teardown of a trap-poisoned instance, #90 host - * `drop()` on a lowered-but-unwritten future). - * - * definitions.py keeps this state unreachable: `WritableFutureEnd.drop` - * traps unless the end is DONE (definitions.py:1183-1184), so a readable - * future end can never observe DROPPED (`future_copy`'s `on_copy_done` - * assertion, definitions.py:2607). Our two teardown paths deliberately - * bypass that trap — a poisoned instance cannot be asked to trap again, and - * the host `drop()` is a public API door — so the state exists here and has - * to be *total*: an unwritten future whose writer died can never satisfy - * its reader, so the reader is told at its rendezvous point, with a - * **trap**, never a DROPPED/COMPLETED answer and never a silent hang. - * - * Consumers of the flag: - * * `read` below, for a reader that has not parked yet (trap on the spot); - * * intrinsics/stream_builtins.ts `futureCopy`, for a parked guest reader - * (the pending event's thunk throws instead of producing a tuple); - * * exec/host_streams.ts leaves host readers on their existing DROPPED - * path — the conventions layer already brands that outcome. + * The writer disappeared without delivering its value. Unlike the + * reference's WritableFutureEnd.drop, host drop and poisoned-instance + * teardown permit this state. Guest readers trap at read or event delivery + * (`futureCopy`), never receive a value-less completion. Low-level host + * readers receive DROPPED; the conventions layer turns it into an error. */ abandonReason: Error | null = null; pendingInst: unknown = null; @@ -705,13 +606,8 @@ export class SharedFutureImpl implements SharedBase { pendingOnCopyDone: OnCopyDone | null = null; /** - * Observers fired once, when this future becomes dropped — by EITHER side, - * including the loud component fault teardown walk (`dropSharedForTeardown`). Streams grew - * this for resource stream producer cancellation; futures need it as the release hook - * for a host wrapper's activity arm (#162, §"Streams and futures"): a guest dropping - * its end with no host operation parked, and the `readResult()` - * already-dropped fast path, both bypass every other close site. `null` = - * already fired. + * Fired once on either end's drop, including teardown. Releases host + * activity even with no host operation parked. `null` means already fired. */ #onDropped: (() => void)[] | null = []; @@ -775,11 +671,7 @@ export class SharedFutureImpl implements SharedBase { } read(inst: unknown, dstBuffer: GuestBuffer, onCopyDone: OnCopyDone): void { - // #84 leg (c): the reader arrives AFTER the writable side was abandoned. - // definitions.py:1154 (SharedFutureImpl.read) asserts `not self.dropped` - // here because the drop - // trap keeps that unreachable; for our abandoned state the honest answer - // is the same trap the parked reader gets, delivered synchronously. + // A reader arriving after abandonment gets the same trap as a parked one. if (this.dropped && this.abandonReason !== null) { throw futureAbandonTrap(this.abandonReason); } @@ -816,7 +708,7 @@ export class SharedFutureImpl implements SharedBase { } // --------------------------------------------------------------------------- -// Copy ends (definitions.py `class CopyEnd`, line 1081) +// Copy ends (definitions.py `CopyEnd`) // --------------------------------------------------------------------------- /** @@ -834,11 +726,8 @@ export abstract class CopyEnd extends Waitable { /** "stream" | "future" — trap-wording parity with wasmtime. */ abstract readonly kind: "stream" | "future"; /** - * Which end this is. Wasmtime words a busy READABLE-end drop as a table - * removal ("cannot remove busy stream") and a busy WRITABLE-end drop as a - * drop ("cannot drop busy stream") — the suite pins both spellings side by - * side (drop-stream.wast:158 read end vs :160 / builtin-trap-poisons- - * instance.wast:38 write end). + * Busy readable-end removal and writable-end drop use distinct trap text, + * as required by the conformance corpus. */ abstract readonly side: "readable" | "writable"; @@ -905,7 +794,7 @@ export class WritableFutureEnd extends CopyEnd { } /** - * definitions.py `WritableFutureEnd.drop` (line 1183): a future's writable + * definitions.py `WritableFutureEnd.drop`: a future's writable * end may only be dropped once it has actually delivered its one value — * `test/async/futures-must-write.wast` is the case this exists for. */ @@ -971,8 +860,8 @@ export function abandonReasonOf(shared: unknown): Error | null { * through `dropSharedForTeardown` instead, which adds the dead-guest * discipline). * - * Never throws, and idempotent: a second call on an already-dropped future is - * a no-op, so `drop()`/`Symbol.dispose` at the layers above are total. + * Idempotent on an already-dropped future. Peer notification can throw; + * drop observers still run through `dropSharedForTeardown`'s finally block. */ export function abandonSharedFuture( shared: SharedFutureImpl, @@ -992,45 +881,14 @@ interface PoisonedInstanceLike { /** * Drop a shared stream/future as *teardown*, without waking a doomed guest. * - * Same outcome as `drop()` for host ends and healthy guest peers (a DROPPED - * notification), with one difference: a parked side belonging to a - * **poisoned** guest instance is retired silently via `resetPending`. - * Notifying it would queue a phantom event into the corpse's waitables, and - * a later driving loop servicing it would resume machinery whose instance - * can no longer be entered (`tick` excludes poisoned instances). Host sentinels are - * not instances at all, so they are always notified. - * - * #100: THE HEALTH TEST IS "POISONED", NOT "BUSY". Deadness must be judged - * by the poison marker and nothing weaker: any liveness proxy that also - * covers a merely mid-call instance is unsound in one direction and strands - * healthy tasks. Under such a proxy a caller mid cross-component (FACT) call - * into an instance that trapped would drag its *other*, healthy tasks down - * with it — one parked on an end of a stream/future the trapping callee also - * held would be classified dead here and retired silently, the exact outcome - * #66 exists to prevent. - * - * So the test consults the poison marker itself. It is per-instance and - * recorded at the single seam every poisoning site routes through - * (`notifyInstancePoisoned`, task/scheduler.ts: exec/boundary.ts `poison`, - * `Store.tick`, `Thread.resumeWith`, the FACT cross-component catches in - * intrinsics/fact_calls.ts, and cabi/handles.ts's gated destructor call), - * and it is recorded *before* the retirement walk runs, so an instance's own - * parked ends still see it during its own walk. `retiredInstances` is - * consulted alongside it because the walk is also reachable directly (it is - * set at walk entry, so the two agree). + * Notify host ends and healthy guest peers with DROPPED. Silently retract + * only a parked end whose instance is already poisoned or retired; being + * mid-call is not evidence of poisoning. The poison marker is set before + * the retirement walk, and direct walks set `retiredInstances` on entry. + * Drop observers run even if peer notification throws. * - * Why this does not re-open review B2 (phantom events into a corpse): the - * concern is that a DROPPED event queued onto a waitable of an instance that - * can never be entered again would be serviced by a later driving loop and - * resume machinery `tick` deliberately excludes. "Can never be entered - * again" is precisely poisoning — a merely mid-call instance is entirely - * ordinary, and its parked task resumes normally and consumes the event. - * The narrowed predicate - * therefore excludes exactly the population B2 is about, and admits only - * peers that will run again. - * - * Used by the poisoning walk below and by the trapping-import abandonment - * path (embedder/instantiate.ts `releaseAsyncArgs`). Idempotent. + * This does not poison an instance itself. It is also used for abandoned + * import arguments and failed handle removals. Idempotent. */ export function dropSharedForTeardown( shared: SharedStreamImpl | SharedFutureImpl, @@ -1052,43 +910,15 @@ export function dropSharedForTeardown( } /** - * Retire every live stream/future end in a trap-poisoned instance's handle - * table (#66). - * - * Rationale: a trapped instance is a corpse (polyengine's per-instance - * poisoning divergence), so no task of this instance can ever rendezvous - * again. Its - * table's `CopyEnd`s are therefore unreachable-forever — leaving their shared - * objects live strands the peers: a parked HOST operation never settles (its - * promise hangs), and a LATER host operation would "succeed" against the - * corpse (a copy into memory nothing will ever read — silent data loss). - * Dropping the shared object now converts both into the spec-shaped DROPPED - * outcome, and the recorded failure lets the embedder layer brand it. - * - * Called from every poisoning site — exec/boundary.ts `poison()` (the - * sync-lift path), scheduler.ts `Store.tick` and thread.ts - * `Thread.resumeWith` (traps during a resumed thread), and the FACT - * cross-component catches (intrinsics/fact_calls.ts, callee side) — with the - * trap as `cause`. Idempotent per instance. The parked-side notification - * discipline lives in `dropSharedForTeardown` above. - * - * Two refinements over the original #66 walk, both from #84: - * - * 1. FUTURES ARE NOT STREAMS. A `stream`'s reader may legitimately observe - * DROPPED (that is end-of-stream), but a `future`'s reader may not - * (definitions.py:2607) — the reference keeps the state unreachable by - * trapping an early writable-end drop (definitions.py:1183-1184), which - * a poisoned instance can no longer be made to do. So an unwritten - * writable future end in this table marks its shared object *abandoned* - * (first pass below) and its reader traps instead. A writable end that - * already reached `CopyState.DONE` delivered its value; nothing is owed. + * Retire stream/future ends still in a poisoned instance's handle table. + * Peers cannot rendezvous with that instance again. Record their failure + * before notification, marking unwritten writable futures abandoned so + * guest readers trap rather than observe an invalid value-less completion. * - * 2. ONE END'S FAILURE MUST NOT STRAND THE REST. The notification of a - * retired end runs arbitrary peer callbacks (host settlers, event - * thunks); a throw used to abort the loop mid-table, leaving the - * remaining ends live and their peers hanging — exactly the outcome this - * walk exists to prevent. The walk now always completes and rethrows the - * first failure afterwards. + * Snapshot before invoking peer code; attempt every retirement and rethrow + * the first notification failure afterwards. Idempotent per instance. This + * is async-end retirement, not general resource or host-operation cleanup; + * ends already transferred out are no longer in this table. */ export function retireInstanceAsyncEnds( inst: PoisonedInstanceLike, @@ -1164,20 +994,16 @@ setOnInstancePoisoned(retireInstanceAsyncEnds, (inst, entry, cause) => { }); // --------------------------------------------------------------------------- -// error-context (definitions.py `class ErrorContext`, line 2775) +// error-context (definitions.py `ErrorContext`) // --------------------------------------------------------------------------- /** - * definitions.py models the debug message as a `String` triple; we keep the - * decoded JS string plus its encoding, which is all `store_string` needs. + * The decoded debug message. Encoding is chosen when the message is lowered. */ export class ErrorContext { constructor(readonly debugMessage: string) {} } -// module identity brand (contracts/embedder-api.md §"Module identity"): error-contexts are -// STATEFUL — they live in a component instance's handle table — so the brand -// exists to make a foreign one diagnosable at the lowering sites, never -// usable. Both this internal class and the embedder-facing wrapper -// (embedder/streams.ts) carry it, because either shape can be handed back to -// a lowering site by embedder code. +// Both internal and facade forms are recognizable. Cross-copy reconstruction +// requires the facade's public string `message`; this internal shape has only +// `debugMessage` and is not a portable carrier (embedder/values.ts). defineBrand(ErrorContext.prototype, ERROR_CONTEXT); diff --git a/runtime/src/task/subtask.ts b/runtime/src/task/subtask.ts index a41fad0..97da462 100644 --- a/runtime/src/task/subtask.ts +++ b/runtime/src/task/subtask.ts @@ -1,12 +1,11 @@ -// definitions.py `### Subtask State` (line 858): one in-progress call from -// this component to an import. +// definitions.py `Subtask`: one in-progress call to an import. import { assert_, trapIf } from "../cabi/trap.ts"; import type { ResourceHandle } from "../cabi/handles.ts"; import type { CoreValue } from "../cabi/types.ts"; import { EventCode, Waitable } from "./waitable.ts"; -/** definitions.py `Subtask.State` (line 859). */ +/** definitions.py `Subtask.State`. */ export enum SubtaskState { STARTING = 0, STARTED = 1, @@ -16,9 +15,7 @@ export enum SubtaskState { } /** - * Anything that can be lent to a callee. `ResourceHandle` is the only - * implementor today; typed structurally so cabi's borrow-scope interfaces - * keep working unchanged. + * Structural lender interface shared with cabi's borrow scopes. */ export interface Lendable { numLends: number; @@ -35,26 +32,20 @@ export class Subtask extends Waitable { flatResults: CoreValue[] = []; /** - * The callee TASK behind this subtask, when there is one (FACT - * cross-component calls; host-import subtasks have none). `subtask.cancel` - * needs it under jspi: a cancellation delivered to a suspended activation - * resumes it on a MICROTASK (the engine's, not ours), so the async form - * must wait until the callee's state is determinate before choosing - * between BLOCKED and the resolved state — the same determinacy question - * `async-start-call` answers, and it needs the same object to ask it of. + * FACT callee task; host imports have none. Async subtask.cancel uses it + * to wait for a JSPI-delivered cancellation to reach a determinate state + * before choosing BLOCKED versus a resolved status. */ // deno-lint-ignore no-explicit-any calleeTask: any = null; /** - * Handles lent to the callee for the duration of the call. `null` once - * `deliverResolve` has run — the reference uses exactly this - * `lenders is None` sentinel to mean "resolve delivered" (line 908), so the - * nullability is semantic, not an optimization. + * Lenders remain live until resolution is delivered, not merely recorded. + * Null is `Subtask.resolve_delivered`'s sentinel. */ lenders: Lendable[] | null = []; - /** definitions.py `Subtask.resolved` (line 880). */ + /** definitions.py `Subtask.resolved`. */ resolved(): boolean { switch (this.state) { case SubtaskState.STARTING: @@ -65,7 +56,7 @@ export class Subtask extends Waitable { } } - /** definitions.py `Subtask.add_lender` (line 890). */ + /** definitions.py `Subtask.add_lender`. */ addLender(h: Lendable): void { assert_( !this.resolveDelivered() && !this.resolved(), @@ -75,7 +66,7 @@ export class Subtask extends Waitable { this.lenders!.push(h); } - /** definitions.py `Subtask.resolve` (line 895). */ + /** definitions.py `Subtask.resolve`. */ resolve(state: SubtaskState, flatResults: CoreValue[]): void { assert_( state === SubtaskState.RETURNED || flatResults.length === 0, @@ -86,7 +77,7 @@ export class Subtask extends Waitable { this.flatResults = flatResults; } - /** definitions.py `Subtask.deliver_resolve` (line 902). */ + /** definitions.py `Subtask.deliver_resolve`. */ deliverResolve(): void { assert_( !this.resolveDelivered() && this.resolved(), @@ -96,7 +87,7 @@ export class Subtask extends Waitable { this.lenders = null; } - /** definitions.py `Subtask.resolve_delivered` (line 908). */ + /** definitions.py `Subtask.resolve_delivered`. */ resolveDelivered(): boolean { assert_( this.lenders !== null || this.resolved(), @@ -106,20 +97,10 @@ export class Subtask extends Waitable { } /** - * Release a never-delivered subtask's lenders after its call broke off a - * non-poisoning exit — trap-rethrow past the CALLEE, capability bail, or - * an abandoned park (contracts/intrinsics.md §A's trap-unwind/lender-release - * obligation; the park legs are #102/#106). - * - * The reference has no analogue because it never resumes after a trap: - * the store dies with the lent handles inside it. The resolution state - * mirrors `canon_lower`'s `on_resolve(None)` branch (definitions.py - * line 2267): CANCELLED_BEFORE_STARTED if the callee never started, - * CANCELLED_BEFORE_RETURNED otherwise. - * - * Idempotent, and a no-op when the resolution was already delivered — a - * settled hook can call it unconditionally without disturbing the success - * path's own `deliverResolve`. + * Idempotent lender cleanup for abandoned calls and non-poisoning unwind + * (contracts/intrinsics.md's trap-unwind/lender-release obligation). + * Unresolved calls take `canon_lower`'s cancellation state according to + * whether they started. Already-delivered resolutions are unchanged. */ unwindLenders(): void { if (!this.resolved()) { @@ -133,7 +114,7 @@ export class Subtask extends Waitable { if (!this.resolveDelivered()) this.deliverResolve(); } - /** definitions.py `Subtask.drop` (line 912). */ + /** definitions.py `Subtask.drop`. */ override drop(): void { trapIf( !this.resolveDelivered(), @@ -143,17 +124,9 @@ export class Subtask extends Waitable { } /** - * definitions.py `canon_lower`'s `on_progress`/`subtask_event` closure - * (lines 2297-2298). The event payload is computed **at delivery time** and - * delivering it is what runs `deliver_resolve` — so the lent handles are - * released exactly when the guest observes the resolution, not when it - * happens. - * - * The `!this.resolveDelivered()` guard has no reference analogue: it exists - * so this can coexist with `unwindLenders()` (which may itself have already - * delivered the resolve on an abandoned path). The reference's - * `subtask_event` calls `deliver_resolve()` unconditionally and would - * assert on a double delivery. + * `canon_lower`'s event thunk reads status at delivery and releases lenders + * when the guest observes resolution. The delivered guard also tolerates + * an earlier unwindLenders cleanup on an abandoned path. */ setSubtaskPendingEvent(subtaski: number): void { this.setPendingEvent(() => { @@ -164,8 +137,8 @@ export class Subtask extends Waitable { } /** - * Pack a `canon_lower` async return value: `state | (subtaski << 4)` - * (definitions.py line 2306, with the accompanying asserts on the ranges). + * Pack a `canon_lower` async result: low four bits hold state, upper 28 the + * nonzero subtask handle index. */ export function packSubtaskResult( state: SubtaskState, diff --git a/runtime/src/task/thread.ts b/runtime/src/task/thread.ts index 80eefe9..8d74903 100644 --- a/runtime/src/task/thread.ts +++ b/runtime/src/task/thread.ts @@ -1,4 +1,4 @@ -// definitions.py `class Thread` (line 317), reimplemented over JS generators. +// definitions.py `Thread`, implemented over JS generators. // // Mapping to the reference, state for state: // @@ -10,11 +10,8 @@ // thread.storage[2] storage: [0, 0] (context.{get,set}) // thread.index index (inst.threads table slot) // -// The reference's `resume()` drives a chain of `switch_to` handoffs -// (`suspend_then_resume` and friends, lines 408-437). Those are the 🧵 -// shared-everything-threads built-ins, which https://github.com/polymorph-components/polyengine/issues/12 defers along with -// memory64; `resume()` here therefore handles a single thread, and the -// switch-to variants are absent rather than approximated. +// Shared-everything thread switching is not implemented (#12); resume drives +// one generator, while JSPI suspension is represented by the bridge. import { assert_ } from "../cabi/trap.ts"; import { @@ -37,25 +34,15 @@ type ThreadState = "running" | "suspended" | "waiting" | "done"; export class Thread implements SchedulableThread { /** - * Per-thread context slots (definitions.py `Thread.storage`, line 323 — - * initialised `[0,0]`). `canon_context_{get,set}` (lines 2348/2358) read and - * write *this*, not per-task state: two threads of the same task have - * independent context. wit-bindgen 0.60 keeps its async task pointer in - * slot 0. - * - * Slots are plain JS numbers: a `context.set` of an i64 value above - * 2^53-1 would lose precision. Moot while memory64/threads support is - * deferred (issue #12) — revisit this when that issue's closure lands. + * Per-thread slots for `canon_context_get` / `canon_context_set`, not + * task-shared state. Number storage is for the supported i32 context; + * full-width i64 context would need a different representation. */ readonly storage: number[] = [0, 0]; /** - * The FACT sync-call bracket stack for THIS activation. - * - * `enter-sync-call` pushes and `exit-sync-call` pops; FACT emits both from - * the same activation, so the activation is the continuity that makes this a - * stack. See the note on `Task.syncCallStack` for why per-task was not - * enough. + * FACT brackets belong to the activation that emitted enter-sync-call and + * exit-sync-call. A task can own several threads, so this is not task-shared. */ // deno-lint-ignore no-explicit-any readonly syncCallStack: any[] = []; @@ -64,12 +51,8 @@ export class Thread implements SchedulableThread { index: number | null = null; /** - * definitions.py `Thread.cancellable` — set at each block point, cleared - * while the thread runs. The reference evaluates it as a live predicate - * (`cancellable = lock_available` in the callback loop, line 2167), so a - * thread that is not parked is never a `request_cancellation` candidate; - * clearing on resume gives the same answer for the only shape that differs - * (a running implicit thread that still holds the exclusive slot). + * Cancellability of the current park, cleared on resume. Callback lock + * availability is checked separately by `Task.implicitThreadCancellable`. */ cancellable = false; @@ -100,12 +83,12 @@ export class Thread implements SchedulableThread { return this.#state === "done"; } - /** definitions.py `Thread.ready` (line 334). */ + /** definitions.py `Thread.ready`. */ ready(): boolean { return this.waiting() && this.#readyFunc !== null && this.#readyFunc(); } - /** definitions.py `Thread.start_waiting_internal` (line 350). */ + /** definitions.py `Thread.start_waiting_internal`. */ #startWaiting(readyFunc: () => boolean): void { assert_(!this.waiting() && this.#readyFunc === null); this.#readyFunc = readyFunc; @@ -113,7 +96,7 @@ export class Thread implements SchedulableThread { this.#store.startWaiting(this); } - /** definitions.py `Thread.stop_waiting_internal` (line 355). */ + /** definitions.py `Thread.stop_waiting_internal`. */ #stopWaiting(cancelled: Cancelled): void { assert_(this.waiting() && this.#readyFunc !== null); assert_( @@ -125,20 +108,12 @@ export class Thread implements SchedulableThread { this.#store.stopWaiting(this); } - /** definitions.py `Thread.resume_later` (line 361). */ + /** definitions.py `Thread.resume_later`. */ resumeLater(): void { assert_(this.suspended(), "resume_later on a non-suspended thread"); this.#startWaiting(() => true); } - /** - * definitions.py `Thread.resume` (line 366): run the body until it blocks - * again or finishes. - * - * The reference's loop over `switch_to` targets is omitted (see the module - * header). What remains is: leave the waiting list if we were on it, become - * the current thread, and step the generator with the cancelled flag. - */ /** Pending `awaitValue` promise, if this thread is parked on one. */ awaiting: Promise | null = null; @@ -151,37 +126,16 @@ export class Thread implements SchedulableThread { this.awaiting = null; this.#store.awaiting.delete(this); this.#state = "suspended"; - // Not a bracketed resumption: post-CM#705 (definitions.py @ 2f13265) - // `Store.tick` resumes a ready thread with no enter/leave bracket at all, - // and this path — the same thread body, woken by a Promise instead of a - // ready-condition — matches it. - // - // What the catch preserves is polyengine's per-instance poisoning, which - // must be MARKER-recorded here specifically: a trap delivered as an - // `awaitValue` rejection is how EVERY guest trap in a suspended - // activation arrives under jspi (pin (e)), and if it unwound silently the - // second call of `builtin-trap-poisons-instance.wast` would re-run the - // guest and report "cannot drop busy stream" where the suite demands the - // poisoned-instance "cannot enter component instance". - // - // Capability signals do not poison, for the same reason as in `tick`: - // they mark the RUNTIME incomplete, not the component faulted. + // Remove awaiting membership before running code that can re-park, so + // overlapping drivers cannot consume this settlement twice. const inst = this.task.inst; - // A poisoned instance's parked segments never run again: this settle - // belongs to an activation that was in flight when a SIBLING activation - // trapped (#66 retired the handle tables). Resuming would re-enter the - // corpse, and asserting turned one legible trap into an assert cascade - // (the classic double-fault shape: `list too long`, then this assert as second - // victim). Retire quietly: the abandoned call's own driver reports, via - // its deadlock trap naming the export. + // Retire late tails of poisoned instances without executing their bodies. if (isInstancePoisoned(inst)) return; try { this.#resumeInternal(value, failure); } catch (e) { if (!(e instanceof NeedsJspi) && !(e instanceof PendingCapability)) { - // Retire the poisoned table's stream/future ends so parked host peers - // settle instead of hanging (#66), and record the marker — the whole - // entry-refusal mechanism since #251's re-key. + // Rejected JSPI activations poison just like synchronous failures. notifyInstancePoisoned( inst as unknown as { handles: Iterable }, e, @@ -191,6 +145,7 @@ export class Thread implements SchedulableThread { } } + /** `Thread.resume`: run until the next block or completion. */ resume(cancelled: Cancelled = CANCELLED_FALSE): void { assert_( !this.running() && !this.done(), @@ -231,10 +186,8 @@ export class Thread implements SchedulableThread { const req = step.value; this.cancellable = req.cancellable; if (req.awaitValue !== undefined) { - // Parked on a Promise, not on a scheduler condition. The driving loop - // owns it from here (exec/boundary.ts `drive`); parking through - // `noteAwaiting` arms the eager settle tracking the scheduler's - // phantom-state gate depends on (see `Store.settled`). + // Promise parks are driver-owned, with eager settlement tracking to + // order their bookkeeping before later scheduler ticks. this.#state = "suspended"; this.awaiting = req.awaitValue; this.#store.noteAwaiting(this, req.awaitValue); @@ -250,18 +203,8 @@ export class Thread implements SchedulableThread { } /** - * definitions.py `Thread.wait_until` (line 396), as a generator-side helper. - * - * Call it from a thread body with `yield*`: - * `const cancelled = yield* thread.waitUntil(() => cond, true);` - * - * Deviation from the reference, deliberate: the reference may return - * immediately when `ready_func()` already holds - * (`if ready_func() and not DETERMINISTIC_PROFILE and random.randint(0,1)`). - * We always take the blocking path, i.e. we behave as the reference's - * `DETERMINISTIC_PROFILE`. Blocking-then-immediately-ready is observably - * equivalent (the scheduler will find this thread ready on the next - * candidate scan) and it removes a coin flip from every wait. + * Generator form of `Thread.wait_until`; call with `yield*`. Uses the + * reference's deterministic-profile blocking path even if already ready. */ *waitUntil( readyFunc: () => boolean, @@ -269,27 +212,21 @@ export class Thread implements SchedulableThread { ): Generator { assert_(this.running(), "waitUntil on a non-running thread"); if (this.task.deliverPendingCancel(cancellable)) return CANCELLED_TRUE; - // definitions.py `ready_or_cancelled` (line 369): a cancel that arrived - // while this task was not cancellable (parked as `pending-cancel`) makes - // the thread ready on its own — otherwise the wakeup is lost until some - // unrelated event happens to satisfy `readyFunc`. The reference's - // `cancellable()` is a live predicate; ours is the static flag AND - // `Task.implicitThreadCancellable` (the "lock is free" conjunct the - // callback loop's `lock_available` supplies there). + // Pending cancellation is itself a wakeup, but the implicit callback + // thread cannot receive it while another thread holds exclusivity. const readyOrCancelled = () => readyFunc() || (cancellable && this.task.hasPendingCancel() && (this !== this.task.implicitThread || this.task.implicitThreadCancellable())); const cancelled = yield { readyFunc: readyOrCancelled, cancellable }; - // AFTER the block (line 372): converts a plain wakeup taken through the - // pending-cancel disjunct into Cancelled.TRUE, and wins over any event - // that became pending in the meantime. + // As in Thread.wait_until, pending cancellation wins over a ready event + // after the block as well as before it. if (this.task.deliverPendingCancel(cancellable)) return CANCELLED_TRUE; return cancelled; } - /** definitions.py `Thread.suspend` (line 390). */ + /** definitions.py `Thread.suspend`. */ *suspend( cancellable: boolean, ): Generator { @@ -299,7 +236,7 @@ export class Thread implements SchedulableThread { return cancelled; } - /** definitions.py `Thread.yield_` (line 405): `wait_until(lambda: True)`. */ + /** definitions.py `Thread.yield_`: wait with an always-ready predicate. */ *yield_(cancellable: boolean): Generator { return yield* this.waitUntil(() => true, cancellable); } diff --git a/runtime/src/task/waitable.ts b/runtime/src/task/waitable.ts index 29a16fb..e3082af 100644 --- a/runtime/src/task/waitable.ts +++ b/runtime/src/task/waitable.ts @@ -1,12 +1,12 @@ -// definitions.py `### Waitable State` (line 754) — the event protocol shared -// by subtasks and (later) stream/future ends. +// definitions.py `Waitable` / `WaitableSet`: events shared by subtasks and +// stream/future ends. import { assert_, trapIf } from "../cabi/trap.ts"; import { chooseCandidate } from "./scheduler.ts"; import type { BlockRequest, Cancelled } from "./scheduler.ts"; import type { Thread } from "./thread.ts"; -/** definitions.py `EventCode` (line 756). */ +/** definitions.py `EventCode`. */ export enum EventCode { NONE = 0, SUBTASK = 1, @@ -21,13 +21,9 @@ export enum EventCode { export type EventTuple = [code: EventCode, p1: number, p2: number]; /** - * definitions.py `class Waitable` (line 767). - * - * The pending event is a **thunk**, not a value: the reference computes the - * payload at delivery time (`get_pending_event` calls it), which is what lets - * a subtask report its *final* state even if it advanced between the event - * being set and being read. Keeping the thunk is load-bearing — see - * `Subtask.setPendingEvent`. + * `Waitable.get_pending_event` computes the payload at delivery time, so a + * subtask reports its current state even if it advanced after notification. + * See `Subtask.setSubtaskPendingEvent` for delivery-time lender release. */ export class Waitable { pendingEvent: (() => EventTuple) | null = null; @@ -47,7 +43,7 @@ export class Waitable { } /** - * definitions.py `Waitable.wait_for_pending_event` (line 786): a + * definitions.py `Waitable.wait_for_pending_event`: a * *non-cancellable* block until this waitable has an event, used by the * synchronous `subtask.cancel` path. */ @@ -70,7 +66,7 @@ export class Waitable { return pendingEvent(); } - /** definitions.py `Waitable.join` (line 797). */ + /** definitions.py `Waitable.join`. */ join(wset: WaitableSet | null): void { assert_(!this.hasSyncWaiter, "join on a waitable with a sync waiter"); if (this.wset) { @@ -82,7 +78,7 @@ export class Waitable { if (wset) wset.elems.push(this); } - /** definitions.py `Waitable.drop` (line 805). */ + /** definitions.py `Waitable.drop`. */ drop(): void { assert_( !this.hasPendingEvent(), @@ -101,7 +97,7 @@ const EV_TRACE = (() => { } })(); -/** definitions.py `class WaitableSet` (line 810). */ +/** definitions.py `WaitableSet`. */ export class WaitableSet { readonly elems: Waitable[] = []; numWaiting = 0; @@ -111,10 +107,8 @@ export class WaitableSet { } /** - * definitions.py `WaitableSet.get_pending_event` (line 821). The reference - * shuffles `elems` before scanning; we scan in **join order** under the - * default FIFO policy (`chooseCandidate` over the ready elements), which is - * within the same allowed nondeterminism — see scheduler.ts's policy note. + * `WaitableSet.get_pending_event` permits choosing any pending member. + * Default to join order; seeded scheduling chooses among ready members. */ getPendingEvent(): EventTuple { const ready = this.elems.filter((w) => w.hasPendingEvent()); @@ -132,7 +126,7 @@ export class WaitableSet { return ev; } - /** definitions.py `WaitableSet.wait_for_event_and` (line 829). */ + /** definitions.py `WaitableSet.wait_for_event_and`. */ *waitForEventAnd( thread: Thread, readyFunc: () => boolean, @@ -152,7 +146,7 @@ export class WaitableSet { } } - /** definitions.py `WaitableSet.wait_for_event` (line 841). */ + /** definitions.py `WaitableSet.wait_for_event`. */ *waitForEvent( thread: Thread, cancellable: boolean, @@ -161,7 +155,7 @@ export class WaitableSet { } /** - * definitions.py `WaitableSet.poll` (line 844). Never blocks, so it is a + * definitions.py `WaitableSet.poll`. Never blocks, so it is a * plain function rather than a generator. */ // deno-lint-ignore no-explicit-any @@ -173,7 +167,7 @@ export class WaitableSet { return this.getPendingEvent(); } - /** definitions.py `WaitableSet.drop` (line 852). */ + /** definitions.py `WaitableSet.drop`. */ drop(): void { trapIf(this.elems.length > 0, "cannot drop waitable set with waitables"); trapIf(this.numWaiting > 0, "cannot drop waitable set with waiters"); diff --git a/runtime/tests/deferred_test.ts b/runtime/tests/deferred_test.ts index 575e3b8..e8b962c 100644 --- a/runtime/tests/deferred_test.ts +++ b/runtime/tests/deferred_test.ts @@ -1,47 +1,5 @@ // Deferred run_tests.py areas: explicitly-ignored placeholders so `deno test` // output shows what is not yet ported and why. -// -// The task core + callback ABI landed the task/thread/waitable -// machinery, so the entries that were blocked purely on "the scheduler does -// not exist" are gone — their content now lives in real tests: -// -// test_async_callback, test_callback_interleaving, test_async_backpressure, -// test_sync_ignores_backpressure -> tests/task_test.ts -// test_async_to_async, test_async_to_sync, test_async_flat_params -// -> tests/async_lower_test.ts -// + tests/integration/e2e_async_test.ts -// test_cancel_subtask (host side) -> tests/task_test.ts (cancellation) -// test_roundtrips (driving loop) -> tests/task_test.ts (sync driving loop, -// deadlock trap) + the e2e suites -// stream/future + error-context -> the value types are implemented -// lift/lower (cabi/async_values.ts), the copy -// protocol lives in task/streams.ts, and -// the host-side ends the reference tests -// needed are exec/host_streams.ts — -// exercised by -// tests/integration/e2e_streams_test.ts -// test_handles (full port) -> host-side ends give the missing piece -// (a host that drives a component and -// holds handles across the call) -// -// What remains ignored is blocked on a *capability*, not on the scheduler: -// a host-API addition, the component instance tree, and 🧵 threads. The two -// JSPI-blocked entries retired once auto-detection landed: -// -// test_sync_using_wait -> a sync task's blocking waitable-set.wait -// is lit (jspi site 2) and exercised green -// under auto-detection by -// test/async/sync-streams.wast (sync copies -// blocking mid-frame) and -// big-interleaving-test.wast's stackful -// `await` exports -// test_thread_cancel_callback -> cancellation delivered to a thread -// parked inside a blocking built-in is -// exercised green by -// test/async/cancellable.wast (cancellable -// wait/yield + pending-cancel delivery; -// requestCancellation finds -// SuspensionPoints since the flip) const deferred: [name: string, reason: string][] = [ [ diff --git a/runtime/tests/digest_test.ts b/runtime/tests/digest_test.ts index 5baaa4d..aacd6e2 100644 --- a/runtime/tests/digest_test.ts +++ b/runtime/tests/digest_test.ts @@ -7,15 +7,7 @@ // Fixtures: runtime/tests/bindgen/fixtures/*.envelope.json are the shim's // C-ABI JSON envelope (`{plan, adapters}` — runtime/src/shim/mod.ts / // crates/translator-shim's README) for each of the three sync guest -// fixtures, checked in as static data. Regenerated (2026-08-08, REVISION -// ROUND) against a current, building `crates/translator-shim` (Track A's -// `importedResources` field (contracts/plan-format.md schema) — is now -// present in every envelope, as an empty array for all -// three fixtures; verified digest-neutral: `computeWorldDigest` on the -// regenerated envelopes matches the same `EXPECTED` values below, byte-for- -// byte identical canonical JSON, since none of these fixtures have -// component-level imports). Regenerate from a clean `crates/translator-shim` -// checkout: +// fixtures, checked in as static data. Regenerate from the translator shim: // // cargo build -p translator-shim --example dump-plan // for w in hello values resources; do @@ -138,12 +130,11 @@ Deno.test("digest: verifyWorldDigest flags a mismatch (wrong expected digest)", }); // --------------------------------------------------------------------------- -// DigestError guard paths (REVISION ROUND: review found these unexercised). +// DigestError guard paths. // --------------------------------------------------------------------------- Deno.test("digest: imported-resources guard fires (own/borrow cannot be safely aliased)", async () => { - // CONTRACT: format.ts:23-33's `importedResources` (v0.2 proposal). No - // alias map exists from an imported resource's `ResourceIndex` to a + // No alias map exists from an imported resource's `ResourceIndex` to a // qualified name yet, so any plan declaring imported resources must be // refused outright rather than risk silently aliasing an own/borrow site // to the wrong (exported) resource — see digest.ts's buildResourceNameMap. diff --git a/runtime/tests/embedder/casing_test.ts b/runtime/tests/embedder/casing_test.ts index c78a230..aceb635 100644 --- a/runtime/tests/embedder/casing_test.ts +++ b/runtime/tests/embedder/casing_test.ts @@ -1,9 +1,6 @@ // `parseLeafName` unit tests: the mangled export/import name grammar -// (contracts/embedder-api.md §"Naming and casing") plus the unknown-bracket -// refusal mandated by the getters/setters pre-ruling (§"Getters and setters -// (pre-ruling, 2026-08-30 — not yet implementable)", final paragraph): -// "the runtime refuses unknown bracket forms in mangled names loudly at -// instantiation (rather than misbinding them as plain names…)". +// (contracts/embedder-api.md §"Naming and casing"). Unknown bracket forms +// must be refused rather than bound as plain names. // // The known forms are already pinned end-to-end against real fixtures in // `version_test.ts`; this file is the focused unit suite for the parser diff --git a/runtime/tests/integration/e2e_cancel_import_test.ts b/runtime/tests/integration/e2e_cancel_import_test.ts index c78f9ca..d845023 100644 --- a/runtime/tests/integration/e2e_cancel_import_test.ts +++ b/runtime/tests/integration/e2e_cancel_import_test.ts @@ -1,18 +1,8 @@ -// Issue #239 end-to-end regression: `driveAsync` (runtime/src/exec/boundary.ts) -// used to hold a store-wide scheduling gate (`Store.pendingResumptions`) -// across an await bounded only by the HOST's answer, whenever two drivers -// were live on the same store and one was parked in its awaiting-race. The -// second driver spun at the top of its own loop and died in ~311ms with: -// -// driveAsync: a resumed-activation claim was never released (the -// activation neither parked, finished, nor trapped) -// -// `runtime/tests/same_store_driver_test.ts` pins the store-level unit shape -// of the fix. This file is the end-to-end proof against a real wit-bindgen -// guest (`examples/guests/cancel-import`), covering every shape the guest -// models: two concurrent export calls, a detached task parked mid-frame with -// no export call outstanding, and a detached task cancelling an in-flight -// async import (`subtask.cancel` via wit-bindgen's drop-to-cancel path). +// Issue #239: a pending host answer must not hold the store-wide scheduling +// gate against another driver. Complements same_store_driver_test.ts with a +// wit-bindgen guest (examples/guests/cancel-import): concurrent exports, +// detached tasks parked mid-frame, and cancellation of an in-flight import +// via wit-bindgen's drop-to-cancel path. // // Requires build artifacts (both produced from source in this repo): // - target/wasm32-unknown-unknown/release/translator_shim.wasm @@ -51,10 +41,8 @@ function delay(ms: number): Promise { return new Promise((r) => setTimeout(r, ms)); } -// Scoped per instantiation (per test), per the dispatch: abortable()'s `abortable()` -// import must observe the abort exactly once per discard and never on a -// natural-completion path, and giving each instance its own counter keeps -// concurrently-run tests from bleeding into one another. +// Per-instance counters isolate concurrent tests. The abortable import must +// observe one abort per discard and none on natural completion. async function instantiate() { let abortsObserved = 0; const imports = { @@ -62,13 +50,12 @@ async function instantiate() { // JSPI involved. sleep: (ms: bigint) => delay(Number(ms)), // Sync-typed import wrapped in `suspending()` (contracts/embedder-api.md - // §"Functions and async" §"Functions and async"): calling it parks the guest's + // §"Functions and async"): calling it parks the guest's // wasm frame mid-activation until the Promise settles — the #239 suspending mark // park shape. block: suspending((ms: bigint) => delay(Number(ms))), - // cancellation discard opt-out (contracts/embedder-api.md §"Functions and async"): branding an - // async-typed import `deferCancel` keeps the pre-cancellation discard run-to-completion - // behavior on cancel, per-declaration. + // `deferCancel` opts this declaration out of prompt cancellation discard; + // the host call must run to completion. "sleep-defer": deferCancel((ms: bigint) => delay(Number(ms))), // Interface-scoped sibling, exercising the raw executor's brand read at // an interface-member leaf (`buildLoweredImport` path walk). diff --git a/runtime/tests/lift_done_verdict_test.ts b/runtime/tests/lift_done_verdict_test.ts index a893736..f7a07f4 100644 --- a/runtime/tests/lift_done_verdict_test.ts +++ b/runtime/tests/lift_done_verdict_test.ts @@ -1,45 +1,12 @@ // A lift that exited `done` resolves even if another driver hop-parks before // its continuation runs — polyengine#310. // -// THE REGRESSION (0.6.5 -> 0.6.6). `createLiftedFunction`'s asynchronous -// completion path used to be -// -// pending.then(() => { -// if (idlePolicy === "exit" && !driveDone()) return backgroundCompletion(); -// return finishHostEntry(); -// }) -// -// which RE-DERIVES `driveDone()` a microtask after the driver already decided. -// `driveDone` is a predicate over store-wide state (`hopParked()` looks at -// every task's threads, #280), and other drivers of the same store mutate it: -// in the traced consumer the settlement pump serviced a settled host call -// belonging to another task, the activation it resumed transiently hop-parked, -// and the lift — whose own driver had exited `EXIT-done` one microtask -// earlier — took `backgroundCompletion()`. That path waits for the task's LAST -// thread to unregister, so for an export that spawns long-lived background -// futures (the consumer's `boot`: engine driver, event pump, accept loop) the -// host's Promise never settles. -// -// THE SHAPE here, store-level in the style of `parked_driver_host_call_test.ts` -// (no checked-in example guest has the participants): -// -// * a real async-typed, callback-ABI lifted export whose core returns a -// Promise, so its thread parks on it and the drive necessarily goes -// through `driveAsync` — i.e. the export completes on the `.then` path -// where the re-derivation lived; -// * the core resolves the task (`task.return`) and spawns an IMMORTAL -// second thread into the same task (`readyFunc: () => false`), the model -// of the consumer's background futures: it keeps `task.threads` non-empty -// forever, which is what makes `backgroundCompletion()` a black hole -// rather than a detour; -// * a foreign ready thread, resumed by the very tick that follows the -// export thread's resumption, queues a microtask that puts a foreign -// HOP-park into `store.awaiting` (`entryHopThreads` = awaiting with no -// `SuspensionPoint` owner). It lands after the driver's `done()` test and -// before the lift's continuation — exactly the window the trace shows. -// -// Pre-fix the export's Promise never settles; with the verdict plumbed out of -// the driver (`DriveExit`) it resolves. +// Completion must use the driver's `DriveExit`, not re-evaluate `driveDone` +// in a later microtask: another task can change the store-wide hop state. +// The callback-ABI export resolves its task but leaves a non-ready background +// thread alive. A foreign thread queues a hop-park between the driver's done +// verdict and the lift continuation. Re-evaluating there would incorrectly +// wait for the background thread and leave the host Promise unsettled. import { assertEq } from "./support/asserts.ts"; import { diff --git a/runtime/tests/plan_loader_test.ts b/runtime/tests/plan_loader_test.ts index 4848c4a..1e8363e 100644 --- a/runtime/tests/plan_loader_test.ts +++ b/runtime/tests/plan_loader_test.ts @@ -180,7 +180,7 @@ Deno.test("loader: nested structural types convert recursively", () => { }); }); -// --- structured translation verdicts (contracts v0.2 proposal) ------------- +// --- structured translation verdicts -------------------------------------- Deno.test("loader: envelope errorDetail becomes a TranslateError with phase", () => { // A `validation` phase is the shim's judgment about the *component* — the @@ -234,7 +234,7 @@ Deno.test("loader: v0.1 envelope without errorDetail is not a validation verdict } }); -// --- imported resources (contracts v0.2 proposal) -------------------------- +// --- imported resources --------------------------------------------------- Deno.test("loader: ResourceIndex = imported + defined", () => { const withImports = loadPlan(minimalPlan({ diff --git a/runtime/tests/plan_v3_test.ts b/runtime/tests/plan_v3_test.ts index a9c94b7..1312882 100644 --- a/runtime/tests/plan_v3_test.ts +++ b/runtime/tests/plan_v3_test.ts @@ -1,18 +1,14 @@ -// Plan v3 (contracts/plan-format.md schema, polyengine#89) pins. +// Plan table index spaces (contracts/plan-format.md, polyengine#89). // // 1. `errorContextTables` is a required section, and the // `error-context-transfer` trampoline resolves its table arguments -// through it — NOT through the resource-table mapping it borrowed at v2. -// The v2 arrangement was not merely "structurally wrong": the -// resource-table accessor answers successfully whenever a concrete -// resource table exists at the colliding index, so a composition with an -// ErrorContext at that slot read and wrote a *different instance's* -// handle table with no diagnostic at all. +// through it, not through resourceTables. A colliding resource-table +// index can silently route to a different instance's handle table. // 2. `task-return` decls carry the raw wasmtime `TypeTupleIndex` (`results`) // alongside its interned `plan.types` entry (`resultType`). That pair is // the dictionary a FACT callee task needs to know its own declared result -// type, which re-enables `canon_task_return`'s -// `trap_if(result_type != task.ft.result)` (definitions.py:2388) for FACT +// type, enabling `canon_task_return`'s +// `trap_if(result_type != task.ft.result)` (definitions.py) for FACT // tasks. import { assertEq } from "./support/asserts.ts"; diff --git a/runtime/tests/resume_claim_discipline_test.ts b/runtime/tests/resume_claim_discipline_test.ts index 6a8f741..de2fd89 100644 --- a/runtime/tests/resume_claim_discipline_test.ts +++ b/runtime/tests/resume_claim_discipline_test.ts @@ -1,25 +1,11 @@ // Resume-time claim/pending-resumption discipline (issue #158). // -// `SuspensionPoint.#resumeInner` (jspi/bridge.ts) has two arms — `produce` -// returned a value, or `produce` threw a resume-time trap — and BOTH hand -// control back to a wasm activation, so both record a pending resumption on -// the store (`Store.addPendingResumption`). Both also call -// `Store.consumePendingIfRunning()` first: when the code delivering the resume -// is a RUNNING guest activation that itself has a pending entry (a -// `subtask.cancel` settling a parked callee from inside its own frame), that -// entry's window is closed. Mechanism A of #158 was the trap arm missing that -// call; back then the gate was a single global slot with a one-claimant -// assert, so the same delivery shape with a trapping `produce` tripped the -// assert — and the assert preempted `#fail(e)`, so the parked guest received -// an AssertionError instead of its trap. These tests pin the fixed symmetry. -// -// Mechanism B of #158 (a second engine-driven resumption in one turn, from an -// activation that is NOT the entry holder) is RESOLVED, 2026-08-22: the gate -// became the per-Store, multi-entry `Store.pendingResumptions` set and the -// one-claimant assert is gone with the slot (the invariant it protected — -// tier-3 ambient attribution unambiguity — no longer exists; see -// `resolveAmbient`). The mechanism-B test below therefore pins the SUCCESS of -// that shape, cross-store and same-store, where it used to pin the assert. +// Both value and trap delivery in `SuspensionPoint.#resumeInner` hand control +// back to wasm. Both must consume the running caller's pending entry before +// recording the resumed activation with `Store.addPendingResumption` +// (#158 mechanism A). This includes cancellation delivered inside a guest frame. +// Multiple resumptions may be pending in one turn; the per-Store set must +// preserve other activations' entries, within and across stores (mechanism B). // // Scaffolding follows park_state_settle_test.ts. NOTE: the AMBIENT state // (activationClaims, threadStack) is still MODULE-GLOBAL, so every test cleans @@ -281,12 +267,8 @@ Deno.test("resume from an EMPTY bracket self-consumes via the claims-top fallbac }); Deno.test("resume from a DIFFERENT running activation while a resumption is pending — cross-store (#158 mechanism B)", () => { - // FLIPPED 2026-08-22. This is issue #158's mechanism B. It used to assert: - // the resumed-ambient gate was a single global slot, so a resumption - // delivered by an activation that is NOT the entry holder could not be - // reconciled by `consumeClaimIfRunning` and tripped the one-claimant assert. - // The gate is now the per-Store, multi-entry `Store.pendingResumptions`; - // both resumptions are legitimately pending and nothing asserts. + // #158 mechanism B: delivery from a different running activation must + // preserve both pending resumptions in their respective stores. const x = mkWorld(); // the activation actually running (a dispatched tail's // guest chunk, under its own wasm-entry bracket) const y = mkWorld(); // the settled-but-not-yet-run activation diff --git a/runtime/tests/same_store_driver_test.ts b/runtime/tests/same_store_driver_test.ts index 22e4c48..5f9151a 100644 --- a/runtime/tests/same_store_driver_test.ts +++ b/runtime/tests/same_store_driver_test.ts @@ -229,17 +229,10 @@ Deno.test("the speculative entry is dropped while a second driver is live on the // The release path removes ONLY the driver's own entry (issue #158's lesson, // re-asserted under #239's new wake path). // -// `Store.pendingResumptions` is a SET, not the single global slot it started -// as, precisely so the race site can "name exactly what we added": the -// `finally` used to blanket-clear, so an entry minted DURING the await by a -// guest-synchronous delivery (`SuspensionPoint.resume`, jspi/bridge.ts) was -// clobbered early, re-opening the mis-attribution window that entry exists to -// close. #239 gave the driver a brand-new reason to run that `finally` — it -// now stands down mid-await whenever another driver arrives — so the -// identity-scoped removal is worth pinning on that path specifically. Nothing -// else in this file distinguishes the two: with one entry in play, -// `removePendingResumption(chosen)` and `pendingResumptions.clear()` are the -// same function. +// When another driver arrives, the awaiting driver stands down and removes +// only its own speculative pending entry. An entry added mid-await by +// `SuspensionPoint.resume` must survive (#158). Two entries are needed to +// distinguish identity-scoped removal from clearing the whole set. // // The foreign entry is a bare sentinel object rather than a second // `awaitingThread`. `pendingResumptions` is `Set` and the property @@ -249,13 +242,8 @@ Deno.test("the speculative entry is dropped while a second driver is live on the // scheduling. A second parked thread would add awaiting-set membership the // property does not involve. // -// Teardown avoids the hop bound rather than catching it: while the foreign -// entry sits in the set the store-wide gate stays held, so BOTH drivers yield -// at their loop tops under the 10,000-hop `claimHops` assert (~311ms). The -// observation takes a few milliseconds, and teardown deletes the sentinel -// FIRST — restoring the state a real `SuspensionPoint.resume` would restore -// when its activation parks or finishes — so both drivers reach their `done()` -// and exit normally, with no AssertionError to swallow. +// Delete the sentinel first during teardown: it holds the store-wide gate, +// so neither driver can reach `done()` until it is removed. Deno.test("a driver releases only its own speculative entry, not the whole gate (#158 under #239's stand-down)", async () => { const store = new Store(); let settleThread!: (v: unknown) => void; diff --git a/tools/browser/bundle.ts b/tools/browser/bundle.ts index f361da5..cff38c9 100644 --- a/tools/browser/bundle.ts +++ b/tools/browser/bundle.ts @@ -1,12 +1,7 @@ // Builds the browser-lane bundle into `harness/browser/dist/` (gitignored). // -// Bundler: `deno bundle --platform browser` (Deno 2.9.5 ships it as an -// experimental esbuild-backed bundler; it resolves the workspace's -// `@polyengine/*` bare specifiers natively, so no alias plumbing is -// needed). The runtime is platform-neutral by contract (docs/architecture.md §4.3) — a -// `node:` specifier appearing in the bundle graph means a regression; the -// M3A-1 episode (scheduler ambient on `node:async_hooks`, fixed by explicit -// ambient threading) is pinned by runtime/tests/platform_purity_test.ts. +// `deno bundle --platform browser` resolves workspace specifiers. Platform-only +// imports in the runtime graph are regressions (runtime/tests/platform_purity_test.ts). // // Usage: deno run -A tools/browser/bundle.ts diff --git a/tools/browser/classify.ts b/tools/browser/classify.ts index e13eac4..6b2c27a 100644 --- a/tools/browser/classify.ts +++ b/tools/browser/classify.ts @@ -3,11 +3,6 @@ // overlay, classify against the Deno-lane xfail truth // (`harness/src/xfail.ts`) widened/narrowed by the overlay's deltas, and // report unexpected failures / stale overlay entries / totals drift. -// -// Extracted from `tools/browser/run-lane.ts` (byte-equivalent behavior; -// proven by that driver's gate: `chromium`/`firefox`/`webkit` still print -// "OK (matches expectation)"). `tools/shell/run-lane.ts` reuses this instead -// of forking a second copy — see issue #22. import { Summary } from "../../harness/src/summary.ts"; import { isXfail } from "../../harness/src/xfail.ts"; diff --git a/tools/browser/opfs-smoke.ts b/tools/browser/opfs-smoke.ts index fd9a001..3d0992a 100644 --- a/tools/browser/opfs-smoke.ts +++ b/tools/browser/opfs-smoke.ts @@ -10,22 +10,10 @@ // (Recipe: `just smoke-opfs `; both lanes ride `just browsers` and // the post-merge browser CI job.) // -// What it does: bundles `harness/browser/opfs_entry.ts` (and, always, the -// `opfs_worker_entry.ts` dual-mode worker module — issue #129 realm -// neutrality; the extra bundle is cheap and keeps the driver's bundling step -// single-shaped regardless of `--realm`), starts the lane's static server -// (which also serves /opfs.html and the wasip2 fixture corpus under -// /fixtures/), launches the browser (shared launcher — Firefox gets the -// JSPI pref), calls the in-page `__opfsSmoke(realm)`, and asserts every -// check passed. Two halves (see opfs_entry.ts): the direct descriptor -// battery (no wasm) and the composed fs-probe guest (std::fs through -// wasi-libc, parking through the marks contracts/embedder-api.md §"The WASI -// parking kernel" describes — JSPI required, so this is also the browser -// exercise of the suspending kernel over real async -// storage). `--realm page` (default) runs both halves on the page; `--realm -// worker` / `--realm shared-worker` run the SAME battery inside a spawned -// dedicated/shared worker (opfs_worker_entry.ts) — the OPFS × JSPI-parking -// × worker-realm intersection issue #129 exists to pin. +// Bundles the page and worker entries, serves them with the fixture corpus, +// and asserts every __opfsSmoke(realm) check passes. Covers direct descriptor +// operations and fs-probe guest I/O through the JSPI parking kernel over real +// async storage. Page (default), worker and shared-worker run the same battery. // // Prerequisites: `just shim` (the translator shim wasm) and // `just fixtures` (examples/guests/build/fs-probe.component.wasm) — both @@ -74,9 +62,7 @@ async function main(): Promise { if (!["page", "worker", "shared-worker"].includes(realm)) { fail(`unknown --realm '${realm}' (page | worker | shared-worker)`); } - // `--realm `'s value is not a flag, so exclude it from the lane guess - // (mirrors run-lane.ts's parseArgs — the sibling track hit the same - // `--realm worker chromium` misparse). + // Exclude --realm's value from the positional lane guess. const realmValIdx = realmIdx >= 0 ? realmIdx + 1 : -1; const lane = Deno.args.filter((a, i) => !a.startsWith("-") && i !== realmValIdx)[0] ?? diff --git a/tools/browser/run-lane.ts b/tools/browser/run-lane.ts index b887ca7..4f0c418 100644 --- a/tools/browser/run-lane.ts +++ b/tools/browser/run-lane.ts @@ -10,12 +10,12 @@ // // PLAYWRIGHT_BROWSERS_PATH=$PWD/.browser-cache \ // deno run -A npm:playwright@1.62.1 install chromium -// # …and `firefox` / `webkit` for the stretch lanes. +// # Repeat for firefox and, optionally, webkit. // // Then, from the repo root: // // just browser-lane chromium # required lane -// just browser-lane firefox # findings lane +// just browser-lane firefox # required lane // just browser-lane webkit # findings lane // // or directly: @@ -42,29 +42,14 @@ // lane's overlay (`harness/browser/expectations/.ts`), prints the // per-directory table, and exits non-zero when a required lane deviates. // -// Exit codes: 0 = lane matched its expectation; 1 = unexpected results in a +// Exit codes: 0 = expected results or findings-only deviations; 1 = deviations in a // required lane; 2 = infrastructure failure (no browser, no corpus, page // crash). // -// LANE NOTES (measured 2026-08-09, linux-arm64, playwright 1.62.1) -// --------------------------------------------------------------- -// chromium HeadlessChrome/151 — REQUIRED lane, ~23 s. JSPI on by default. -// firefox Firefox/153 — runs the full corpus in ~26 s. JSPI works behind -// `javascript.options.wasm_js_promise_integration`, which this -// driver sets via `firefoxUserPrefs` (launch.ts FIREFOX_PREFS). -// webkit WebKit 26.5 (WPE headless) — runs the full corpus in ~10 s. -// JSPI works unflagged. On a host that is not Ubuntu 24.04 the -// bundled build will not launch until its Ubuntu-24.04-ABI -// libraries are supplied; the exact recipe (and why exporting -// `LD_LIBRARY_PATH` around this driver does NOT work) is in -// `harness/browser/expectations/webkit.ts`. Run that lane with -// `PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1`. -// -// Historical note: all three lanes originally carried FINDING M3A-1 (the -// scheduler's ambient rode `node:async_hooks`, absent in every browser — -// 80 async/ commands). Fixed by explicit ambient threading in the -// scheduler; chromium now runs at exact Deno parity (deltas: []) and the -// stale-delta detector keeps it that way. +// launch.ts supplies JSPI flags/preferences and forwards the environment. +// WebKit may need Ubuntu-24.04-compatible libraries on other Linux hosts; +// see harness/browser/expectations/webkit.ts. Lane policy and deltas live +// in the per-engine expectations, not in measured timings here. // ============================================================================ import { startServer } from "./serve.ts"; @@ -88,10 +73,8 @@ const repoRoot = normalize( ); /** - * JS realm the corpus runs in (issue #129). `page` is the historical - * behavior; the worker realms run the SAME corpus through the SAME driver and - * are judged against the SAME per-engine expectation — same engine + same - * corpus means identical totals, so any delta at all is a realm leak. + * Execution realms share corpus and per-engine expectations, so realm-specific + * failures cannot be hidden by separate overlays. */ const REALMS = ["page", "worker", "shared-worker"] as const; type Realm = typeof REALMS[number]; diff --git a/tools/npm-build/build.ts b/tools/npm-build/build.ts index 782a289..48c176c 100644 --- a/tools/npm-build/build.ts +++ b/tools/npm-build/build.ts @@ -4,11 +4,11 @@ // // JSR is the primary registry (README §Consuming); this emits the same five // packages for npm consumers, from the same sources, at the same version AS -// THEIR JSR COUNTERPART. The build is a pure function of the workspace: +// THEIR JSR COUNTERPART unless --version overrides the lockstep set: // `name`, `version` and `exports` come from each package's deno.json, so a // version bump or a new entry point needs no edit here. // -// Versioning mirrors release.yml's "compute tag and version" step exactly: +// Versioning follows the lockstep/package policy: // runtime, translator, wasi and ct-runner are a LOCKSTEP set (one emission // version between them — either the caller's `--version` stamp, or their // agreeing manifest version when no stamp is given). @polyengine/protocol is @@ -17,8 +17,8 @@ // emits at its own manifest version, on both registries, regardless of // `--version`. Dependency edges follow the same asymmetry: a lockstep // package depending on a lockstep sibling pins the exact emission version -// (they publish atomically); a dependency on protocol is a caret of -// protocol's manifest version (`^0.1.0`), matching what `deno publish` +// (they are released together); a dependency on protocol is a caret of +// protocol's manifest version, matching what `deno publish` // itself does when it rewrites workspace cross-deps for the JSR emission — // letting a protocol bump dedup across lockstep versions built before and // after it, per contracts/embedder-api.md §"Module identity and @@ -31,14 +31,14 @@ // `import.meta.url` (runtime/src/embedder/copy.ts), neither of which survives a // CommonJS emit. // -// THE INVARIANT THIS FILE EXISTS TO PROTECT: cross-package imports become real +// Cross-package imports must become real // npm `dependencies`, never inlined source. Duplicate copies of the runtime or // the protocol package in one module graph are the latent-failure mode that // contracts/embedder-api.md §"Module identity and @polyengine/protocol" is // a response to; registry symbols // make a duplicate survivable, not correct. `mappings` below forces every // `@polyengine/*` specifier — including subpath forms like -// `@polyengine/runtime/shim` — onto the npm package at the exact same version. +// `@polyengine/runtime/shim` — onto the owning npm package and version range. // tools/npm-build/smoke.mjs asserts the property mechanically after packing. // // Build order is dependency order, and each package is built with its @@ -134,7 +134,7 @@ async function main() { // One version for the lockstep four: either the manifests' (which the // release workflow's lockstep guard already pins to agree) or the caller's - // stamp for a prerelease. A torn set would produce packages depending on + // version override. A torn set would produce packages depending on // sibling versions that were never published. protocol is NOT part of // this — it rides its own manifest version always (see header). for (const p of LOCKSTEP) { @@ -164,7 +164,7 @@ async function main() { for (const pkg of PACKAGES) { const name = manifests.get(pkg)!.name; // Dependency edges: a lockstep sibling is pinned EXACT (they publish - // atomically, and a prerelease stamp must pin exactly what it built + // together, and a version override must pin exactly what it built // alongside); protocol is pinned by CARET of its own manifest version — // JSR parity, since `deno publish` rewrites workspace cross-deps to // caret, and it lets a protocol bump dedup across mixed lockstep diff --git a/tools/npm-build/consumer/types.ts b/tools/npm-build/consumer/types.ts index 31c0745..b372bf1 100644 --- a/tools/npm-build/consumer/types.ts +++ b/tools/npm-build/consumer/types.ts @@ -37,10 +37,7 @@ export async function typeSurface(componentBytes: Uint8Array) { // A byte stream is `Stream`: `Chunk` widens a numeric element // type to `Uint8Array | number[]`, so the u8 bulk path is expressible. - // `createStream` (contracts/embedder-api.md §"The host-ABI surface and - // its version") is the application-surface spelling of - // the former `Stream.create()` static — the concrete class is no longer - // exported. + // createStream is the application factory; Stream is the returned handle type. const { stream, writer } = createStream(); await writer.write(new Uint8Array([1, 2, 3])); await writer.close(); diff --git a/tools/npm-build/stamp_check.ts b/tools/npm-build/stamp_check.ts index 669fafc..c67d89b 100644 --- a/tools/npm-build/stamp_check.ts +++ b/tools/npm-build/stamp_check.ts @@ -1,5 +1,4 @@ -// The stamp-path leg of `just test-npm`: proves the `--version` override -// works as release.yml's prerelease path relies on it — stamping the +// The stamp-path leg of `just test-npm`: checks the `--version` override stamps the // lockstep four while leaving @polyengine/protocol on its own manifest // version (contracts/embedder-api.md §"Version canonicalization"). // diff --git a/tools/release-bundle/dual_copy_test.ts b/tools/release-bundle/dual_copy_test.ts index 4bdf9d6..0a91761 100644 --- a/tools/release-bundle/dual_copy_test.ts +++ b/tools/release-bundle/dual_copy_test.ts @@ -1,34 +1,12 @@ -// THE dual-copy pin (contracts/embedder-api.md §"Module identity and -// @polyengine/protocol"; issue #83). +// Cross-copy contract tests (contracts/embedder-api.md "Module identity and +// @polyengine/protocol"). Copy A is the source tree; copy B is a separate bundle. +// Query-string cache-busting an entry would still share its dependency modules +// and classes, so it would not test this boundary. // -// Two GENUINELY distinct runtime copies in one process: -// -// copy A — the source tree (`runtime/src/embedder/mod.ts`); -// copy B — the release bundle built by ./build.ts. -// -// The bundle is the second copy on purpose, and it is the *production* shape -// of the bug: two separately-built bundles on one page each embed a copy, and -// no resolution discipline can reach that case. Note that the obvious cheap -// trick — importing the same entry twice with different query strings — does -// NOT produce a second copy: the entry module is duplicated, but every -// relative import below it resolves to the same already-cached module, so the -// classes and symbols underneath are shared and every assertion here would -// pass vacuously. -// -// What is pinned: the census sees both copies; the STATELESS contract values -// (`ComponentException`, the `suspending` mark, hand-rolled brands) are honored across -// the boundary; the STATEFUL ones (`Stream`) are refused with a named -// cross-copy error rather than silently adapted; and an unbranded throw in a -// multi-copy graph says so. -// -// contracts/embedder-api.md §"The host-ABI surface and its version" shrinks -// the SHIPPED bundle (./entry.ts) to application -// surface only — it no longer re-exports `@polyengine/protocol` vocabulary, -// on purpose (contracts/embedder-api.md §"The host-ABI surface and its -// version"). This test builds copy B from ./test_entry.ts instead: the same -// dependency graph as the shipped entry, plus protocol re-exports so this -// test can still reach copy B's OWN classes — the premise the test pins. -// The production artifact (built from entry.ts) is unaffected. +// The census sees both copies; stateless brands interoperate; stateful Stream +// handles are refused; unbranded throws include the multi-copy diagnostic. +// test_entry.ts adds protocol exports solely to expose copy B's classes to the +// test. The shipped entry.ts keeps its application-only surface. import { buildBundle } from "./build.ts"; import { COPY_URL, instantiate } from "../../runtime/src/embedder/mod.ts"; diff --git a/tools/release-bundle/entry.ts b/tools/release-bundle/entry.ts index 0c26f00..99268f5 100644 --- a/tools/release-bundle/entry.ts +++ b/tools/release-bundle/entry.ts @@ -1,16 +1,15 @@ // The consumer-facing embedder bundle: one platform-neutral ES module -// carrying the embedder API surface plus the L3 runner glue, for consumers +// carrying the embedder API surface plus component-test runner glue, for consumers // that cannot import polyengine's TS sources directly — browser pages/workers // and plain Node (the callback ABI needs no JSPI flag, so stock `node` can // import this). Built by ./build.ts with `deno bundle --platform browser` // (the same emission the browser lanes use, tools/browser/bundle.ts) and -// shipped as the `polyengine-embedder.mjs` release asset (#16 interim scheme). +// shipped as the `polyengine-embedder.mjs` release asset. // // Surface discipline: everything here is already public — the embedder API // (contracts/embedder-api.md), the shim's `Translator`, `@polyengine/ct-runner` // (runSuite + Context + import analysis + the tags inventory), and -// `@polyengine/wasi`. The bundle adds no API of its own; per the #8 -// rescope there is no runtime code generation anywhere in this graph +// `@polyengine/wasi`. The bundle adds no API of its own and generates no runtime JS // (nothing needs CSP beyond `wasm-unsafe-eval`). export * from "@polyengine/runtime/embedder"; diff --git a/tools/release-bundle/test_entry.ts b/tools/release-bundle/test_entry.ts index c8f35f3..c8fb902 100644 --- a/tools/release-bundle/test_entry.ts +++ b/tools/release-bundle/test_entry.ts @@ -1,15 +1,5 @@ -// TEST-ONLY bundle entry — never shipped as the `polyengine-embedder.mjs` -// release asset (see ./entry.ts for that surface, and build.ts's -// `buildBundle(out, entry)` for the override this file relies on). -// -// dual_copy_test.ts needs to reach copy B's OWN `@polyengine/protocol` -// instance — the class-per-copy premise the test pins — but -// contracts/embedder-api.md §"The host-ABI surface and its version" makes -// the *shipped* entry stop re-exporting protocol vocabulary, on purpose: a -// real host module never gets it from the runtime. This second entry point -// exists solely so the in-repo cross-copy test can still observe copy B's -// protocol symbols after the bundle's own dependency graph resolves them — -// it changes nothing about what a consumer's bundle exports. +// Test-only entry exposing the bundled copy's protocol classes to dual_copy_test. +// buildBundle(out, entry) selects this instead of the shipped application entry. export * from "./entry.ts"; export { ComponentException, diff --git a/tools/shell/bundle.ts b/tools/shell/bundle.ts index 6e5e8cd..6729dd0 100644 --- a/tools/shell/bundle.ts +++ b/tools/shell/bundle.ts @@ -1,8 +1,4 @@ -// Builds the shell-lane bundle into `tools/shell/dist/` (gitignored, same as -// `harness/browser/dist/`). Same `deno bundle` mechanics as -// `tools/browser/bundle.ts` (see that file's header) — a separate script -// per issue #22's dispatch, not a shared one, because the entry points and -// output layout differ (no `harness/browser/index.html` to sit next to). +// Builds the shell-lane bundle into tools/shell/dist/ (gitignored). // // Usage: deno run -A tools/shell/bundle.ts @@ -33,11 +29,8 @@ export async function bundle(): Promise { }); const { code } = await cmd.output(); if (code !== 0) throw new Error(`deno bundle failed with code ${code}`); - // Byte-identical .mjs copy for the node/bun lanes (host-node.mjs imports - // it): with no package.json anywhere above tools/shell/dist/, node parses - // a .js file as CommonJS and rejects the bundle's import/export syntax; - // the .mjs extension forces ESM. (The jsshells keep loading entry.js — - // one bundle, two names, so every lane runs the same bytes.) + // .mjs makes ESM explicit for Node/Bun without package.json or syntax detection. + // The shells load the byte-identical entry.js. await Deno.copyFile(out, join(dirname(out), "entry.mjs")); return; } diff --git a/tools/shell/deno-canary.ts b/tools/shell/deno-canary.ts index 53ed3d6..ef66ec1 100644 --- a/tools/shell/deno-canary.ts +++ b/tools/shell/deno-canary.ts @@ -1,9 +1,6 @@ -// Deno canary channel probe (issue #22 — substitutes for a V8/d8 lane; see -// the issue's scope-decision comment). Fetches the latest per-commit canary -// Deno build, reports its version (which embeds the V8 version — printed -// alongside the pinned lane's V8 for a diff), runs the shell capability -// preamble under it, then runs the FULL conformance suite with the canary -// binary and reports the table against the pinned lane's expected totals. +// Deno canary probe for newer V8 behavior. Reuses the cached build, or fetches +// the latest canary when uncached; reports Deno/V8 identity, capabilities and +// conformance results against the pinned lane's expected totals. // // Usage: deno run -A tools/shell/deno-canary.ts [--json ] // @@ -94,8 +91,7 @@ async function runCanary( const cmd = new Deno.Command(bin, { args, cwd, - // Env-clean per issue #22's dispatch: no ambient PLAYWRIGHT_* etc leak - // into the canary's conformance run. + // Keep ambient browser/tool settings out of the canary run. clearEnv: true, env: { PATH: Deno.env.get("PATH") ?? "" }, stdout: "piped", diff --git a/tools/shell/entry.ts b/tools/shell/entry.ts index c363aae..dec5489 100644 --- a/tools/shell/entry.ts +++ b/tools/shell/entry.ts @@ -38,11 +38,8 @@ const g = globalThis as any; type EngineName = "spidermonkey" | "jsc" | "node" | "bun" | "unknown"; function detectEngine(): EngineName { - // Order matters twice: bun defines process.versions.node too (check bun - // first), and Deno 2's node-compat layer defines globalThis.process as - // well — but this entry never runs under Deno (the Deno lane is - // harness/tests/conformance_test.ts), so process.versions.node here means - // a real node (or bun) driven via tools/shell/host-node.mjs. + // Bun also defines process.versions.node, so check Bun first. This entry runs + // Node/Bun through host-node.mjs; Deno has a separate conformance runner. if (typeof g.process?.versions?.bun === "string") return "bun"; if (typeof g.process?.versions?.node === "string") return "node"; if (typeof g.os?.file?.readFile === "function") return "spidermonkey"; @@ -69,7 +66,7 @@ function readBinary(path: string): Uint8Array { case "bun": // Installed by tools/shell/host-node.mjs (which keeps node: builtin // imports out of this browser-platform bundle). It copies out of - // node's pooled Buffer — see the preamble for why that is load-bearing. + // node's pooled Buffer so the returned bytes do not expose its whole slab. if (typeof g.__polyengineHostRead !== "function") { throw new Error( `readBinary: ${engine} detected but no __polyengineHostRead — run this ` + diff --git a/tools/shell/fetch.ts b/tools/shell/fetch.ts index 4c681d8..492846f 100644 --- a/tools/shell/fetch.ts +++ b/tools/shell/fetch.ts @@ -1,14 +1,13 @@ // Fetches engine shells into `.shell-cache/` (gitignored), caching by build // identity so re-runs don't re-download. // -// LANE IDS (issue #22 follow-up: promote pinned shells to per-push gates): -// sm-pinned — SpiderMonkey release matching the browser lane (Firefox -// 153.0), sha256-verified against tools/shell/pins.json. -// sm-nightly — SpiderMonkey mozilla-central nightly (unchanged canary). -// jsc-pinned — JSC trunk rev pinned in pins.json (318852@main), +// LANE IDS (versions and digests in tools/shell/pins.json): +// sm-pinned — SpiderMonkey release, sha256-verified. +// sm-nightly — SpiderMonkey mozilla-central nightly. +// jsc-pinned — JSC trunk rev pinned in pins.json, // sha256-verified, mirrored to a repo-owned release (see // fetchJsc's header for why webkitgtk.org can't be pinned). -// jsc-trunk — JSC trunk LAST-IS (unchanged canary; x86_64 CI only). +// jsc-trunk — JSC trunk LAST-IS (x86_64 only). // node-pinned — Node.js release pinned in pins.json (nodejs.org dist // tarball), sha256-verified; both linux arches. // bun-pinned — Bun release pinned in pins.json (oven-sh/bun GitHub @@ -17,11 +16,7 @@ // Usage: deno run -A tools/shell/fetch.ts // // -// Extraction: no system `unzip` on the dev box this was written on, and no -// suitable pure-Deno zip reader was available in the JSR registry at the -// time (checked: no `@zip/zip`). Falls back to `python3 -m zipfile` — every -// CI runner and dev box in this repo's matrix ships python3. Swapping in a -// pure-Deno unzip later is a drop-in replacement for `extractZip` below. +// Zip extraction requires python3 and preserves executable modes and symlinks. import { dirname, fromFileUrl, join, normalize } from "jsr:@std/path@1"; @@ -87,14 +82,8 @@ export function defaultShellPaths( async function extractZip(zipPath: string, destDir: string): Promise { await Deno.mkdir(destDir, { recursive: true }); - // Two things extractall() gets wrong that this loop fixes (both - // load-bearing for the JSC bundle; SpiderMonkey's zip has neither): - // * file modes are dropped — reapplied from each entry's external_attr - // (the wrapper and bin/jsc must be executable); - // * SYMLINK entries are written as tiny regular files containing the - // target path — the bundle's lib/ *.so.N names are symlinks to the - // real *.so.N.x.y files, and the dynamic linker fails on the fake - // ones with "file too short" (this lane's second CI failure). + // extractall() loses executable modes and materializes symlinks as files. + // Restore both: JSC needs executable launchers and real shared-library links. const cmd = new Deno.Command("python3", { args: [ "-c", @@ -132,8 +121,7 @@ function smArch(): string { * Checks a lane's `BUILD_IDENTITY` file against an expected identity string. * A pinned lane's identity is the pin's version/sha — when `pins.json` bumps * the version, the on-disk cache's stamp no longer matches and this returns - * `false`, triggering a refetch without any manual `.shell-cache` clearing - * (dispatch requirement: pin bumps take effect automatically). + * `false`, triggering a refetch without manual cache clearing. */ async function cacheMatches(dir: string, expectedIdentity: string): Promise { try { diff --git a/tools/shell/polyfill.ts b/tools/shell/polyfill.ts index 7445b9f..375a0f4 100644 --- a/tools/shell/polyfill.ts +++ b/tools/shell/polyfill.ts @@ -1,47 +1,13 @@ // Shell surface bootstrap — MUST be the first import in `tools/shell/entry.ts` // (see that file's header for why import order matters here). Installs -// TextEncoder/TextDecoder on `globalThis` for engines that lack them. +// encoding, SHA-256 digest and base64 helpers on globalThis when absent. // -// Verified 2026-08-09 (see issue #22): SpiderMonkey nightly (linux-aarch64 -// jsshell) and JSC (jsc, GTK 2.52 stable) both lack TextEncoder/TextDecoder -// entirely. Scope of what the runtime actually needs (grep of -// `runtime/src/cabi/strings.ts`, `runtime/src/intrinsics/transcode.ts`, -// `runtime/src/shim/translator.ts`, `runtime/src/digest/*.ts`, -// `runtime/src/cache/core.ts` — the full non-test surface): -// -// - `new TextEncoder()` then `.encode(str)` — UTF-8 encode. No `encodeInto` -// call anywhere in the runtime; not implemented here. -// - `new TextDecoder("utf-8", { fatal: true, ignoreBOM: true })` and -// `.decode(bytes)` — UTF-8 decode, throw on ill-formed input. -// - `new TextDecoder("utf-16le", { fatal: true, ignoreBOM: true })` — -// UTF-16LE decode, throw on an unpaired surrogate (definitions.py -// `load_string` treats a lone surrogate as ill-formed on decode; the -// USVString replacement only happens on the *lowering* side, via -// `String.prototype.toWellFormed`, not here — see strings.ts). -// - `new TextDecoder()` (defaults: "utf-8", fatal: false, ignoreBOM: false) -// — used once, translator.ts, to decode the shim's plan JSON. -// -// `String.prototype.toWellFormed` is present on both shells (verified), so -// USVString conversion (lone surrogate -> U+FFFD) needs no polyfilling. -// -// Web semantics matched here (WHATWG Encoding Standard): -// - UTF-8 encode: standard UTF-8 of the (already well-formed, from the -// runtime's call sites) JS string; a lone surrogate reaching `encode` -// directly (bypassing `toWellFormed`) is itself replaced with the UTF-8 -// encoding of U+FFFD, matching the spec's encoder (which never produces -// WTF-8) rather than throwing. -// - UTF-8 decode: standard UTF-8 decoder; `fatal: true` throws -// `TypeError` on any ill-formed byte sequence (overlong forms, -// unexpected continuation bytes, out-of-range code points, truncated -// sequences) instead of substituting U+FFFD. -// - UTF-16LE decode: pairs of little-endian code units become UTF-16 code -// units directly (JS strings already are UTF-16); `fatal: true` throws on -// an odd trailing byte or an unpaired surrogate. -// - `ignoreBOM` is accepted but every runtime call site passes `true` -// (never relevant) except the bare `new TextDecoder()` call, which -// therefore strips a leading U+FEFF if present (default WHATWG -// behavior) — implemented for completeness though the runtime's one use -// (JSON from the shim) never emits a BOM. +// Implements the lane's encoding subset, not the full Encoding API: UTF-8 +// encode, UTF-8/UTF-16LE decode, fatal errors and BOM handling; no encodeInto. +// Encoding replaces lone surrogates with U+FFFD. Fatal decoding rejects malformed +// input, including unpaired UTF-16 surrogates, as required by CABI load_string. +// This differs intentionally from lowering's USVString replacement. Non-fatal +// decoding substitutes U+FFFD, and ignoreBOM=false strips a leading BOM. // deno-lint-ignore no-explicit-any const g = globalThis as any; diff --git a/tools/shell/run-lane.ts b/tools/shell/run-lane.ts index 9d4b982..ca05a54 100644 --- a/tools/shell/run-lane.ts +++ b/tools/shell/run-lane.ts @@ -1,9 +1,8 @@ -// Engine-shell canary lane driver (issue #22). +// Pinned and canary engine-shell lane driver. // // Deno-lane-shaped: spawns the shell as a child process (`tools/shell/dist/ -// entry.js`, bundled by `tools/shell/bundle.ts`), CWD'd to the repo root so -// the entry's disk reads (`harness/generated/**`, the shim, the probe -// modules) resolve without any argv/config plumbing. Parses the entry's +// entry.js`, bundled by `tools/shell/bundle.ts`). The entry derives artifact +// paths from its bundle URL, not the shell's working directory. Parses the entry's // `@polyengine:`-prefixed protocol lines from stdout and classifies with the // SAME `harness/src/xfail.ts` + `Summary` + per-lane-overlay machinery the // browser lanes use (`tools/browser/classify.ts` — shared, not forked). @@ -54,7 +53,7 @@ // shell crashed mid-corpus with no results at all), regardless of // `required`. sm-nightly/jsc-trunk are `required: false` (issue #22: // "findings lanes, never gating"); sm-pinned/jsc-pinned are -// `required: true` (this track: promoted to per-push gates). +// `required: true`. import { dirname, fromFileUrl, join, normalize } from "jsr:@std/path@1"; import { classify, diffTotals, totalsOf } from "../browser/classify.ts"; @@ -148,14 +147,8 @@ type Header = any; * node/bun run `tools/shell/host-node.mjs` instead — an unbundled preamble * that installs the entry's host capabilities (binary reads, `print`) and * imports `dist/entry.mjs` (bundle.ts's byte-identical ESM-suffixed copy). - * The bun lane additionally sets `BUN_JSC_useWasmMultiMemory=1`: bun 1.3.x - * vendors WebKit's wasm multi-memory implementation but ships it - * default-off, and the CABI routinely needs >1 memory per core module — - * stock bun fails 174 corpus commands with "there can at most be one Memory - * section for now" (measured 2026-08-11; the flag-flip precedent is the - * firefox browser lane setting its own JSPI pref). Bun warns BUN_JSC_* - * options are unstable across releases; the pin freezes that risk, and a - * re-pin must re-verify the option (see the expectation's header). + * The Bun lane sets `BUN_JSC_useWasmMultiMemory=1` for multi-memory adapters. + * BUN_JSC_* options are unstable; re-verify the option when updating the pin. * * The CWD is set to the repo root but the entry does NOT rely on it: JSC * trunk bundles run through their shipped wrapper, which chdir()s into the diff --git a/tools/smoke-c0/common.ts b/tools/smoke-c0/common.ts index b80469f..e2b2153 100644 --- a/tools/smoke-c0/common.ts +++ b/tools/smoke-c0/common.ts @@ -1,13 +1,10 @@ // Consumer smoke test — shared helpers (docs/consumers.md). // -// Run legs from this directory: -// deno run --allow-read leg1_tdz.ts -// deno run --allow-read leg2_exec_model.ts -// deno run --allow-read leg3_throughput.ts -// deno run --allow-read --allow-run leg4_websocket.ts +// Run from the repo root with `just smoke-c0`, or use this directory's +// `deno task leg1` through `leg4` for focused probes. // // Prerequisite (built from source in this repo): -// cargo build -p translator-shim --release --target wasm32-unknown-unknown +// just shim // // All consumer artifacts are referenced by absolute path and are READ-ONLY; // nothing in this tree writes to the polymorph working trees. @@ -19,8 +16,7 @@ import type { WirePlan } from "../../runtime/src/plan/format.ts"; export const REPO_ROOT = new URL("../../", import.meta.url); export const POLYMORPH = Deno.env.get("POLYMORPH_ROOT") ?? "/home/lmartin/p/polymorph"; -/** experiment-mosh renamed and moved out of the polymorph tree - * (2026-08-10): it is `wosh`, a sibling OF the polymorph directory. */ +/** Wosh has a separate root from the polymorph consumer family. */ export const WOSH = Deno.env.get("WOSH_ROOT") ?? "/home/lmartin/p/wosh"; /** Absolute paths to the consumer artifacts under test (never copied). */ @@ -77,7 +73,7 @@ export interface TranslateAttempt { } /** - * One `translateRaw` + `loadEnvelope` pass. + * Time `translateRaw`, then translate again to obtain parsed artifacts. * * Rejections are captured, not thrown: contracts/plan-format.md gives the * shim three phases (validation | unsupported | internal) and this suite's job is to diff --git a/tools/smoke-c0/leg1_tdz.ts b/tools/smoke-c0/leg1_tdz.ts index a0aa047..683655a 100644 --- a/tools/smoke-c0/leg1_tdz.ts +++ b/tools/smoke-c0/leg1_tdz.ts @@ -1,15 +1,14 @@ // Leg 1 — the lann/jco#51 TDZ shape, executed. // -// deno run --allow-read leg1_tdz.ts +// deno task leg1 // -// Artifact: wosh/spikes/compose-async-tdz/composed.wasm (the consumer -// formerly named experiment-mosh) — a wac +// Artifact: wosh/spikes/compose-async-tdz/composed.wasm, a wac // composition of `tdz:plug` (async factory returning own) into // `tdz:socket` (async export awaiting it, plus an exported `handoff` // interface that names the same resource in a signature). That combination -// is exactly jco's TDZ trigger: the emitted trampoline references a resource -// class above its declaration. A runtime linker emits nothing, so the defect -// *class* cannot exist here — this leg makes that claim executable. +// reproduced lann/jco#51: an emitted trampoline referenced a resource class +// above its declaration. This leg checks polyengine's runtime linking of that +// shape; it makes no claim about current jco behavior. // // Expected values read from source, not guessed: // plug/src/lib.rs: make() -> Ok(Widget::new(WidgetRes(42))) diff --git a/tools/smoke-c0/leg2_exec_model.ts b/tools/smoke-c0/leg2_exec_model.ts index 7b28911..23fe39e 100644 --- a/tools/smoke-c0/leg2_exec_model.ts +++ b/tools/smoke-c0/leg2_exec_model.ts @@ -1,15 +1,14 @@ -// Leg 2 — polymorph-iroh's exec-model probe: the lann/jco#11 kill shot. +// Leg 2: polymorph-iroh's detached-task execution-model probe (lann/jco#11). // -// deno run --allow-read leg2_exec_model.ts +// deno task leg2 // // Artifact: polymorph-iroh/target/wasm32-wasip2/release/iroh_exec_model_guest.wasm // Guest source: polymorph-iroh/experiments/exec-model/guest/src/lib.rs // jco reference driver: polymorph-iroh/host-jco/src/run-exec.mjs // -// The probe order below is the diagnostic: jco's execution-slot queue -// serializes task lifetimes, so `start-pump()` leaves a detached task holding -// an in-flight `wait-for` and every LATER export call deadlocks before its -// first wasm slice. We drive exactly that order and assert each step. +// The historical jco issue serialized task lifetimes: a detached `start-pump()` +// task waiting on `wait-for` prevented later exports from entering. This probe +// asserts those exports can progress; it does not audit current jco behavior. // // Host glue implemented here (throwaway; the real thing is the shim): // - wasi:clocks/monotonic-clock@0.3.0 `wait-for` — an async host function @@ -45,11 +44,7 @@ function check(cond: boolean, msg: string) { console.log(` ${cond ? "PASS" : "FAIL"} ${msg}`); if (!cond) failures.push(msg); } -// Retightened after the R-fix round: probes 4a/4b originally carried -// `xfail(...)` wrappers against findings R-1 (host-pump starvation of -// pendingHostCalls) / R-2 (check-then-act poisoning via hostFailure); both are -// fixed (runtime/tests/host_pump_test.ts pins them), so every probe is now a -// hard assertion and a regression fails this leg. +// Every probe is a hard assertion; starvation and host-failure regressions fail. function note(msg: string) { console.log(` .... ${msg}`); } diff --git a/tools/smoke-c0/leg3_throughput.ts b/tools/smoke-c0/leg3_throughput.ts index c06b1e2..24f336d 100644 --- a/tools/smoke-c0/leg3_throughput.ts +++ b/tools/smoke-c0/leg3_throughput.ts @@ -1,11 +1,10 @@ // Leg 3 — translator throughput on the multi-MB consumer corpus. // -// deno run --allow-read leg3_throughput.ts +// deno task leg3 // // Times `translateRaw` cold + 3 warm per artifact, records envelope size and // plan shape, and dumps `errorDetail` verbatim for rejections (rejections are // FINDINGS for the consumer-smoke discrepancy table, not failures of this leg). -// Baseline datum: 94 KB component in ~28 ms. import { ARTIFACTS, diff --git a/tools/smoke-c0/leg4_websocket.ts b/tools/smoke-c0/leg4_websocket.ts index 2d48c37..1aeec99 100644 --- a/tools/smoke-c0/leg4_websocket.ts +++ b/tools/smoke-c0/leg4_websocket.ts @@ -1,12 +1,10 @@ // Leg 4 (best-effort, translate-only) — the polymorph-websocket // conformance suite. // -// deno run --allow-read leg4_websocket.ts +// deno task leg4 // -// Both artifacts were ALREADY BUILT in the consumer tree (see -// polymorph-websocket/conformance/driver-ct/justfile `compose-suite`), so this -// leg only reads them — nothing here builds, composes, or writes anything in -// the consumer working trees. +// Requires prebuilt consumer artifacts. This leg only reads them; it never +// builds, composes or writes into consumer working trees. // // bare: target/wasm32-wasip2/release/conformance_guest_ct.wasm // — the suite with `polymorph:websocket` still IMPORTED. Its diff --git a/tools/smoke-c0/wasi_stub.ts b/tools/smoke-c0/wasi_stub.ts index 5ae2a94..c9f92ba 100644 --- a/tools/smoke-c0/wasi_stub.ts +++ b/tools/smoke-c0/wasi_stub.ts @@ -1,21 +1,7 @@ -// Consumer smoke — throwaway host-import glue synthesized from `plan.imports`. -// -// Not a shim package. This exists so a leg can instantiate a -// real consumer component whose *binary* carries the Rust/TinyGo libc wasip2 -// baseline (wasi:cli, wasi:io, wasi:clocks, wasi:filesystem, wasi:random) -// even when its WIT world declares almost nothing — see Leg 3's import -// surfaces, where three different p2 versions (0.2.6 / 0.2.9 / 0.2.12) appear -// across the corpus. -// -// Strategy: walk `plan.imports` (`imports[].path`, contracts/plan-format.md -// schema — `{name, path, kind}`), materialize the exact nested host object the -// executor demands, and fill every leaf with a LOUD stub. Real behavior is -// injected by `overrides`, keyed *version-independently* as -// `":/"` with the `@x.y.z` stripped, because the same -// interface arrives at three versions across the corpus. -// -// Any stub that actually fires throws `StubCalled` — an unimplemented import -// must be a loud, attributable failure, never a silent zero. +// Smoke-only host glue from `plan.imports`, not a WASI provider. Consumer binaries +// may import a libc baseline beyond their WIT world. Unimplemented leaves throw +// `StubCalled`; overrides use version-independent `:/` keys +// so one implementation can serve the corpus's different interface versions. import { hostResourceType } from "../../runtime/src/exec/mod.ts"; import type { WirePlan } from "../../runtime/src/plan/format.ts"; diff --git a/tools/smoke-tls/run.ts b/tools/smoke-tls/run.ts index 04ac3b7..635cdf2 100644 --- a/tools/smoke-tls/run.ts +++ b/tools/smoke-tls/run.ts @@ -1,30 +1,12 @@ -// #18 — polymorph-tls: smoke the jco-leg surface under polyengine. +// Translate/enumerate polymorph-tls artifacts; --exec also runs composed suites. // -// deno run --allow-read --allow-hrtime run.ts # translate-only -// deno run --allow-read run.ts --exec # + execute suites -// deno run --allow-read run.ts --exec --only SUBSTRING +// just smoke-tls +// deno run --allow-read --allow-env=POLYMORPH_ROOT,WOSH_ROOT \ +// tools/smoke-tls/run.ts [--exec] [--only SUBSTRING] // -// Mirrors tools/smoke-c0 leg 4 (translate-only + import-surface enumeration) -// and then goes one step further than the issue asks: the tls conformance -// suite artifacts are fully composed (no network, no sockets — recon -// 2026-08-09), so the same ct-runner + wasi-package path that ran the -// websocket suite (the retired ports/websocket conformance runner; the -// consumer's own deltic legs carry it now) can execute them -// directly. All consumer artifacts are referenced by absolute path and are -// READ-ONLY; nothing here writes to the polymorph trees. -// -// Named residues (conformance discipline: no unnamed absorption): -// TAG-GATING (#25) — FIXED (ct-runner reads the suites' own -// `component-test:tags@0.1` inventory, ct-runner/src/tags.ts; the -// sections survive wac composition, verified on these artifacts). Each -// target below declares its missing-features and the previously -// xfailed cases schedule out as `not-applicable`, exactly like their -// harness legs; the xfail entries were pruned. -// CALLBACK-NULL-CONTEXT (#24) — FIXED (continuation-chunk attribution -// sentinels, jspi/bridge.ts); the entry below was pruned. The -// webcrypto-composed target is the only corpus that reaches the -// interleave, so this suite remains its integration pin -// (runtime/tests/jspi/chunk_attribution_test.ts is the unit pin). +// Consumer artifacts are prebuilt and read-only. Executable compositions must +// need only the WASI imports supplied here; per-target missing features select +// inapplicable cases through the suite's component-test tags. import { fmtSurface, @@ -41,7 +23,7 @@ import { wasi } from "../../wasi/src/mod.ts"; const CONF = `${POLYMORPH}/polymorph-tls/target/conformance`; -/** Everything prebuilt in the consumer tree (recon 2026-08-09). */ +/** Expected prebuilt consumer artifacts; absent targets are reported as skipped. */ const TRANSLATE_TARGETS: Array<[string, string]> = [ ["suite: plain (tls world, ed25519 only)", `${CONF}/suite-plain.wasm`], ["suite: delegated (fixture signer plugged)", `${CONF}/suite-delegated.wasm`], @@ -59,12 +41,8 @@ const TRANSLATE_TARGETS: Array<[string, string]> = [ ], ]; -/** The executable smoke matrix: [target-key, artifact, missing-features, - * xfails]. All compositions are self-contained (surfaces are pure WASI — - * phase 1), so no extra host modules are wired. `missing` mirrors what - * their harness legs pass per target (run-node.mjs); tag gating turns the - * per-target inapplicable cases into `not-applicable` rows. Any future - * xfail must name its class + issue. */ +/** [target-key, artifact, missing-features, xfails]. Only WASI is wired. + * Missing features select not-applicable cases; xfails must name a class/issue. */ const EXEC_TARGETS: Array<[string, string, string[], Record]> = [ ["polyengine-delegated", `${CONF}/suite-delegated.wasm`, [], {}], ["polyengine-delegated-webcrypto", `${CONF}/suite-delegated-webcrypto.wasm`, [], {}], diff --git a/tools/translate/README.md b/tools/translate/README.md index 7a0b99d..bda248a 100644 --- a/tools/translate/README.md +++ b/tools/translate/README.md @@ -1,7 +1,7 @@ # tools/translate — build-time translation -Translate a component **once, at build/deploy time**, so production never -ships the translator (~0.5 MB gzip of wasm). The deploy set becomes: +Translate a known component at build time so its deployment does not need the +translator. The deploy set is: ``` component.wasm # unchanged @@ -11,19 +11,30 @@ your host + @polyengine/runtime ## Translate +From the repository root: + ```sh +just shim deno run --allow-read --allow-write tools/translate/main.ts \ app.component.wasm # writes app.component.plan.json ``` -(`-o out.plan.json` to choose the destination, `--shim path` to point at a -translator build other than the repo's.) +Use `-o out.plan.json` to choose the destination, or `--shim path` for another +translator build. The default shim is +`target/wasm32-unknown-unknown/release/translator_shim.wasm`. Translation errors +are checked before writing the output. + +The `.plan.json` file is an **envelope**, not a bare plan: it includes the plan +and base64-encoded FACT adapters. Plans currently use `formatVersion: 5`; deploy +with a matching runtime, which rejects other format versions. See the +[plan contract](../../contracts/plan-format.md). ## Deploy host ```ts import { artifactsFromEnvelope, instantiate } from "@polyengine/runtime/embedder"; +const imports = {}; // Supply the component's host imports here. const [envelope, componentBytes] = await Promise.all([ fetch("/app.component.plan.json").then((r) => r.text()), fetch("/app.component.wasm").then((r) => r.arrayBuffer()), @@ -34,18 +45,28 @@ const component = await instantiate( ); ``` -Acquisition is deliberately yours (HTTP above; `Deno.readFile`/`node:fs` -work the same) — `artifactsFromEnvelope` is pure. The envelope embeds the -component's sha-256 and length, which `instantiate` verifies: a mismatched -deploy pair (stale envelope, wrong component) **fails loudly at -instantiation**, pinned by `translate_test.ts`. +Configure package resolution in the deploying application. Acquisition belongs +to the host (HTTP above, or filesystem/bundler assets); `artifactsFromEnvelope` +does no I/O. Check HTTP status before decoding in a production fetch path. +The plan records the component's SHA-256 and length. `instantiate` checks length +and, by default, the hash, rejecting a mismatched deploy pair; this is covered +by [`translate_test.ts`](translate_test.ts). + +Treat the envelope and adapters as trusted build artifacts. The component hash +binds the referenced component bytes; it does not authenticate the plan or +adapter code. See [security](../../docs/security.md). ## When to prefer runtime translation instead -Components that arrive dynamically (plugin systems) can't pre-translate: -use `instantiate({ componentBytes, translator }, …)` (contracts/embedder-api.md -§"Module wiring and instantiation") -with the translator asset, and let the runtime's artifact cache -(`@polyengine/runtime/cache`) amortize repeat visits. The full delivery -decision tree is in the design note on -[#16](https://github.com/polymorph-components/polyengine/issues/16). +For components unknown at build time, pass `{ componentBytes, translator }` to +`instantiate`. Reuse a translator from `@polyengine/translator` across calls. +Artifact caching is opt-in through `translateCached` in +`@polyengine/runtime/cache`; `instantiate` does not automatically persist +translation results. Persistent caching requires a translator with a `buildHash` +(created from shim bytes); a translator wrapped from wasm exports alone has no +binary identity for the cache key. See the +[embedder contract](../../contracts/embedder-api.md) for instantiation and +[architecture](../../docs/architecture.md) for caching. + +`just test-translate` exercises the CLI, envelope deployment, and translator +package from the repository root. diff --git a/tools/translate/main.ts b/tools/translate/main.ts index 5d6c534..7919c77 100644 --- a/tools/translate/main.ts +++ b/tools/translate/main.ts @@ -1,7 +1,6 @@ -// Build-time translation CLI (issue #16, delivery design note item 2). +// Repository build-time translation CLI. // -// Translates a component ONCE, at build/deploy time, so production never -// ships the ~0.5 MB (gzip) translator wasm — the deploy set becomes: +// Translate before deployment to omit the translator wasm from the deploy set: // // component.wasm (unchanged) // component.plan.json (this tool's output: the translation envelope @@ -17,8 +16,8 @@ // [-o ] [--shim ] // // Defaults: -o .plan.json next to the input; --shim resolves to -// the repo's built translator (consumers of the published package will get -// a default translator from @polyengine/translator once #16 packaging lands). +// the repo's built translator. Applications can instead use the packaged +// defaultTranslator() from @polyengine/translator. import { Translator } from "@polyengine/runtime/shim"; diff --git a/tools/version-guard/check.ts b/tools/version-guard/check.ts index 7bf1762..d2058a8 100644 --- a/tools/version-guard/check.ts +++ b/tools/version-guard/check.ts @@ -1,49 +1,18 @@ #!/usr/bin/env -S deno run -A -// The release version guard (three modes: `pr`, `publish`, `cut`). +// Release version guard; policy is in AGENTS.md "Versioning and publishing". +// runtime/translator/wasi/ct-runner version together; protocol is independent. // -// What it defends. The five packages publish under two rules (AGENTS.md -// §Versioning, README §Consuming): @polyengine/{runtime,translator,wasi, -// ct-runner} version in LOCKSTEP and their manifests always carry the NEXT -// release; @polyengine/protocol versions independently and publishes at the -// next cut after its manifest bumps. Breaking changes are -// declared per package by PR labels `breaking/`; no label means -// caret-compatible. Labels are MUTABLE and read live from the GitHub API -// every time — a label corrected after the merge still steers the cut, -// which is the point of reading them at cut time rather than trusting an -// event payload. +// local - working-tree lockstep, monotonicity and protocol byte identity, +// without GitHub/PR context; advisory golden-change reminder. +// pr - early version/label checks; no-op without PR_NUMBER. +// publish - compare protocol bytes with its named published version before +// release packaging, including artifact-only prereleases. +// cut - release-window labels, minor advances, goldens and release notes. // -// The three modes, and why the enforcement point is where it is: -// -// pr — early warning, inside `gha::core` on PR runs. Lockstep -// agreement, monotonicity against the last cut, label/version -// agreement in both directions, and the protocol-tear warning. -// Advisory in the sense that matters: label edits deliberately -// do NOT re-trigger CI, so a PR-time verdict can be stale by -// merge time. Cheap to be wrong here; a re-run picks up fixes. -// publish — the AUTHORITATIVE tear guard, in release.yml's publish step, -// in BOTH modes. Registry publishes happen only at explicit -// cuts (#223), so the window between a PR-time verdict and a -// publish is no longer a race — but a PR-time verdict is still -// the wrong thing to trust: it misses label edits made after -// the run, commits pushed straight to main, and any run stale -// by the time the cut happens. This check runs at the publish -// itself, reads the tree being published, and cannot be stale. -// On the prerelease path it publishes nothing and is instead -// early detection: a red means the next CUT would tear. -// cut — label/version consistency for the whole release window, plus -// the release-notes fragment, in release.yml on release=true -// only. This is where a breaking label becomes a minor bump. -// -// The tear this exists for (the concrete incident): PR #219 changed -// protocol/src without bumping protocol/deno.json, because its merge -// resolution assumed 0.2.0 was still unpublished — under the pre-#223 flow -// every green main published, and one such run had published 0.2.0 hours -// earlier. Every publish after that skipped protocol as already-published, -// so runtime@0.4.0-pre.* shipped importing exports the published -// protocol@0.2.0 did not have: an import-time failure for anyone consuming -// the pair. #221 (protocol 0.2.1) repaired it. `publish` mode is the check -// that would have made that red, loudly, at the first publish after the -// merge. +// Labels are read live because they may be corrected after merge. Label edits +// do not rerun CI, so PR checks cannot replace release-time enforcement. +// Published protocol bytes must match the tree: otherwise dependents can ship +// importing exports missing from the already-published protocol package. import { compareSemver, isMinorBumped, parseSemver } from "./semver.ts"; import { @@ -159,12 +128,8 @@ export async function latestCutVersion( export type GoldenChange = { status: "A" | "M" | "D"; path: string }; -/** Parse `git diff --name-status ... -- ` output. A rename is - * treated as an M of the old path plus an A of the new one (task authority: - * dispatch step 1) — the new content still needs the gate, but the OLD - * golden's disappearance is exactly what a plain M/D would flag, and a pure - * rename-with-no-content-change should not dodge that by virtue of the - * path move. A copy (`C...`) only introduces a new path, so it is an A. */ +/** Parse golden name-status output. Renames count as M(old) + A(new), so moving + * a golden cannot bypass the modified/deleted gate. Copies count as additions. */ export function parseGoldenNameStatus(output: string): GoldenChange[] { const changes: GoldenChange[] = []; for (const raw of output.split("\n")) { @@ -181,9 +146,7 @@ export function parseGoldenNameStatus(output: string): GoldenChange[] { } else if (code === "A" || code === "M" || code === "D") { changes.push({ status: code, path: parts[1] }); } - // Other statuses (T, U, X, B) do not occur for plain committed text - // fixtures; ignoring them fails closed only in the sense that they - // neither trigger nor excuse the gate, which matches "added is free". + // Other statuses are ignored; this check covers committed text M/D changes. } return changes; } @@ -687,25 +650,15 @@ export async function protocolVersionAtRef( return JSON.parse(atob(content.replace(/\n/g, ""))).version; } -/** The locked-golden name-status diff for the whole release window, `git - * diff --name-status .. -- `. The release - * checkout is shallow (actions/checkout@v4 default depth), so the last - * cut's tag is fetched first — mirroring fetchBase's PR-base fetch — with - * the same three-dot-unavailable fallback (two-dot local comparison; here - * there is no merge-base ambiguity to begin with, so `..` is exact rather - * than a fallback in the same sense, but the two-call shape matches the - * rest of this file's style). */ +/** Diff locked goldens between the last cut and target SHA. Fetch the tag first + * for shallow release checkouts; this compares endpoints, not a merge base. */ export async function cutGoldenChanges( fx: Effects, lastTag: string, sha: string, ): Promise { - // The full-refspec form is load-bearing: without a DESTINATION - // (`:refs/tags/…`) the fetch drops the objects into FETCH_HEAD but - // creates no local ref, so the tag NAME stays unresolvable and the diff - // below fails with "bad revision" (the v0.5.0 cut, first dispatch). The - // PR-base fetch this mirrors gets away with a bare source because a raw - // sha resolves from the object store alone; a tag name needs a ref. + // The destination ref makes the tag name resolvable locally; FETCH_HEAD alone + // supplies objects but not the ref used by the diff below. await fx.run("git", [ "fetch", "origin", diff --git a/tools/version-guard/check_test.ts b/tools/version-guard/check_test.ts index 9ace11f..3a1d9d2 100644 --- a/tools/version-guard/check_test.ts +++ b/tools/version-guard/check_test.ts @@ -3,9 +3,7 @@ // exercise the real decision logic (the fixtures below are shaped from real // jsr.io and GitHub API responses). -// Assertions are local, matching the rest of tools/ (bundle_test.ts): no -// test-only dependency enters the workspace lockfile for a guard whose -// whole point is that it cannot be knocked over by a registry. +// Local assertions avoid adding a registry dependency to the guard tests. import type { Effects, HttpResponse } from "./effects.ts"; import { sha256Hex } from "./effects.ts"; import { diff --git a/tools/version-guard/effects.ts b/tools/version-guard/effects.ts index 1716f70..41b26fb 100644 --- a/tools/version-guard/effects.ts +++ b/tools/version-guard/effects.ts @@ -1,8 +1,5 @@ -// The effects the guard needs from the outside world, as one injectable -// interface: every check below is a pure function over this, so the unit -// tests run the real decision logic against fixtures with no network, no -// `gh`, and no repository state. `realEffects()` is the only place that -// touches Deno APIs. +// Injectable network, command and filesystem effects. Tests exercise the guard's +// decision logic without network access, gh or repository state. export type HttpResponse = { status: number; body: string }; diff --git a/tools/version-guard/semver.ts b/tools/version-guard/semver.ts index ec6741c..ea497bb 100644 --- a/tools/version-guard/semver.ts +++ b/tools/version-guard/semver.ts @@ -1,9 +1,5 @@ -// A minimal semver, deliberately dependency-free: the guard runs as the very -// first step of `gha::core` and inside release.yml before `deno publish`, so -// every import it takes is a way for a registry outage to fail a release. -// Only what the guard actually decides is implemented — parse, compare, and -// the minor-level questions the versioning policy asks (AGENTS.md -// §Versioning). +// Dependency-free semver parsing/comparison for the release guard, avoiding a +// registry dependency before CI and publish checks can run. export type Semver = { major: number; diff --git a/translator/mod.ts b/translator/mod.ts index c87f497..1329604 100644 --- a/translator/mod.ts +++ b/translator/mod.ts @@ -1,17 +1,13 @@ -// @polyengine/translator — the packaged translator wasm plus its per-platform -// loader (issue #16 delivery design note, item 3). +// @polyengine/translator: the packaged translator wasm and per-platform loader. // // Why a separate package: the translator is a versioned peer of the -// runtime (plan-format coupling), so it ships inside the same release — -// but embedders that translate at BUILD time (tools/translate, -// contracts/embedder-api.md §"Module wiring and instantiation") deploy no -// translator at all, and keeping the ~1.85 MB -// asset out of @polyengine/runtime keeps their production graphs clean. +// runtime (plan-format coupling), so it ships in the same release. Embedders +// that translate at build time need not include the translator asset in +// their production module graphs. // // The asset (`translator_shim.wasm`, sibling to this module) is copied // from the cargo build by `just shim` and is gitignored — run `just shim` -// once in a fresh checkout. Publish tooling will pin the exact asset (and -// its digest) into the released package when #16's packaging lands. +// once in a fresh checkout. import { Translator } from "@polyengine/runtime/shim"; diff --git a/upstream-component-model-repo-findings.md b/upstream-component-model-repo-findings.md index 42ffc2e..3da3830 100644 --- a/upstream-component-model-repo-findings.md +++ b/upstream-component-model-repo-findings.md @@ -1,172 +1,87 @@ # Upstream findings: WebAssembly/component-model -Single source of truth for issues and PRs we file (or intend to file) against -the [WebAssembly/component-model] repository. Anything upstream-worthy -discovered during development gets an entry **here**, not a note in the design -docs — docs/architecture.md links here instead. Findings against *other* repos (wasm-tools, -wasmtime, wit-bindgen) do not belong in this file; see "Out of scope" at the -bottom. +Spec-repository findings and local dispositions. Runtime semantic exceptions +link here from [architecture §1](docs/architecture.md#1-goals). Entries are +not authorization to file publicly; no filing is claimed without a link. -All file/line references are against the submodule pin at -`third_party/component-model` — currently **`73b7ad5`**. Re-verify line -numbers before filing if the submodule has been bumped. +Current checks below use the local `third_party/component-model` pin +`7c676115e93cd7d54c1732d95c54c6a3de7c5ae0`. Historical observations name their +source snapshot separately. Spec paths are relative to `design/mvp/` unless +otherwise stated. Prefer function references over line numbers when rechecking. -Status legend: `DRAFT` (not yet filed) → `FILED #n` / `PR #n` → `RESOLVED`. +Status: `DRAFT` (unfiled finding), `FILED` / `PR` (linked upstream record), +`RESOLVED` (absent at the pin), or a named local disposition. --- ## CM-1: vestigial `$async?` immediate on `canon resource.drop` in CanonicalABI.md -**Status:** DRAFT — proposed as a one-line docs PR -**Found:** 2026-08-08, while answering "are dtors allowed to block?" +**Status:** RESOLVED at the current pin by upstream commit `dff1181` +([#698](https://github.com/WebAssembly/component-model/pull/698)). No filing needed. ### Evidence -- `design/mvp/CanonicalABI.md:4013` shows the canonical-definition template as - `(canon resource.drop $rt $async? (core func $f))`. -- The Explainer grammar has no async immediate: - `design/mvp/Explainer.md:1539` — `(canon resource.drop (core func ?))`. -- The reference implementation has no async parameter: - `canonical-abi/definitions.py:2319` — `def canon_resource_drop(rt, i)`, - hardcoding `CanonicalOptions(async_ = False)` and a sync `FuncType`. -- The prose four paragraphs below the template is explicit that drops are - synchronous: *"Because the type, lifting and lowering are all non-`async`, - the destructor may not block."* (CanonicalABI.md ~4046). - -Earlier 0.3 drafts had an async variant of `resource.drop`; the `$async?` in -the wat template is a leftover from its removal. - -### Proposed fix - -Docs PR deleting `$async?` from the template at CanonicalABI.md:4013. - -### Draft PR description - -> CanonicalABI.md's canonical-definition template for `resource.drop` still -> shows an `$async?` immediate. The Explainer grammar, the validation text, -> and `canon_resource_drop` in definitions.py all define `resource.drop` as -> unconditionally synchronous (and the surrounding prose says "the destructor -> may not block"). This looks like a leftover from the removal of async -> drops; this PR removes the stale immediate from the template. - -### Filing notes - -- Removal history: #578 removed the `async` immediate from `resource.drop` - (Explainer grammar, Binary.md opcode `0x07`, the `option` prose); - #646 then forbade async ABI options on sync-typed functions entirely — - current dtor text ("may not block; may spawn a cooperative thread that - does") dates from there. -- The `$async?` immediates on `stream.cancel-read/write` / - `future.cancel-read/write` a few sections down (~L4744-4747) are still - correct — the fix is deleting the one token at :4013 only. -- Still present at upstream `main` (re-checked 2026-08-14). A standalone - pre-tracker draft of this finding (`upstream-issue-stale-async-drop.md`) - was retired into this entry; see git history if the fuller prose is wanted. +The original `73b7ad5` template included `$async?` despite the synchronous +Explainer grammar and `canon_resource_drop` implementation. The current +`CanonicalABI.md` template is `(canon resource.drop $rt (core func $f))`, +matching both. `canon_resource_drop` uses synchronous canonical options and +a synchronous function type; destructors may not block their implicit thread. --- ## CM-2: `canon_backpressure_set` is dead code in definitions.py -**Status:** RESOLVED upstream, independently — no filing needed. Upstream -commit `1c42aeb02` ("Remove TODO from tests, remove stale backpressure.set -definition", PR [#690]) deleted the block from definitions.py; verified -absent at `main` 2026-08-14. Our runtime's annotations referencing the dead -code (`runtime/src/intrinsics/async_builtins.ts`, `intrinsics/mod.ts`, -`runtime/README.md`) came out with the submodule bump to `4142913` -(2026-08-17), which also rebased the runtime's definitions.py line -citations across the deleted block (−7). -**Found:** 2026-08-08, during the canonical-ABI reference-test port - -[#690]: https://github.com/WebAssembly/component-model/pull/690 +**Status:** RESOLVED by upstream commit `1c42aeb02` +([#690](https://github.com/WebAssembly/component-model/pull/690)); the function +is absent at the current pin. No filing needed. ### Evidence -- `canonical-abi/definitions.py:2366-2371` contains a - `### 🔀 canon backpressure.set` section defining - `canon_backpressure_set(flat_args)`. -- Neither prose document knows it: the CanonicalABI.md TOC and body document - only `canon backpressure.{inc,dec}` (CanonicalABI.md:46, and the section the - TOC points to), and the Explainer grammar defines only - `(canon backpressure.inc ...)` / `(canon backpressure.dec ...)` - (Explainer.md:1543-1544; prose at 1706). -- Because no grammar production exists, `canon_backpressure_set` is - **unreachable from any component** — dead code from the - `backpressure.set` → `backpressure.{inc,dec}` transition. -- The repo's own consistency checker fails on exactly this: - `python3 canonical-abi/diff.py` reports 4 content differences, all of them - the `canon_backpressure_set` block (definitions.py:2180-2183 in diff.py's - code-block numbering vs CanonicalABI.md jumping straight to - `canon_backpressure_inc`), and exits with - *"Error: Differences found between definitions.py and CanonicalABI.md."* - -(The formerly-proposed removal PR is exactly what upstream #690 did, -including making `diff.py` pass again; the draft text was dropped from this -entry on resolution — see git history.) +At `73b7ad5`, `definitions.py` still defined `canon_backpressure_set`, but +the grammar and CanonicalABI.md exposed only `backpressure.inc` and +`backpressure.dec`. The unused block also caused the repository's +`canonical-abi/diff.py` consistency check to fail. #690 removed it. --- ## Filing checklist (per finding) -1. Re-verify evidence against current `main` (not just our submodule pin). -2. Search existing issues/PRs for duplicates. -3. File; record the number and flip the status line here. -4. On resolution: bump the submodule, note the resolving commit here, and - remove any workaround/annotation in our code that referenced the finding. +1. Obtain operator authorization before filing in a foreign repository. +2. Recheck against upstream `main`, not only our pin, and search for duplicates. +3. If filed, record the upstream link and status here. +4. On resolution, record the resolving commit and review local workarounds. ## Out of scope (tracked elsewhere, listed so they aren't lost) -- **wasm-tools CLI 1.247 `json-from-wast` parser lag** (15/59 suite files - parse; current `wast` crate parses 59/59): version-skew, resolved by - version — `crates/testgen` links the `json-from-wast` crate at the pinned - wasm-tools release train (0.258 parses 67/67). No upstream traffic needed. -- **wasmparser 0.252 requires async function types for async lifts; wasm-tools - 1.247's validator predates the rule**: spec-tracking drift between released - versions, not a component-model repo defect. Handled by docs/architecture.md §4.1/§9 - version-pinning discipline (the translator's wasmparser is the single - validation authority). -- **wasmparser skips case-folding when comparing `[method]`/`[static]` and - interface names** (candidate bytecodealliance/wasm-tools issue; found - 2026-08-22 validating our #185, extended 2026-08-25; fix tracked on our side - as #238): Explainer.md §Name Uniqueness requires strongly-unique — lowercase - acronyms, strip the `[...]` prefix, compare — and explicitly lists - `[method]foo.BAR` vs `[method]foo.bar` as a validation error. wasmparser - implements the fold for plain labels (`KebabStr`'s case-insensitive - `Eq`/`Hash`) but `ResourceFunc` *and* `InterfaceName` derive them on the raw - string (`src/validator/names.rs`; verified identical in 0.251/0.252/0.256 - and on wasm-tools `main` as of 2026-08-25), so `[method]r.a-b` + - `[method]r.a-B` in one scope validates, as does `test:i/x-y` + `test:i/x-Y` - (wasm-tools 1.247 `validate --features component-model`, wasmtime 47 - `compile`, and our translator-shim all accept; plain-label and record-field - folded collisions are correctly rejected). Searched both upstream trackers - 2026-08-25: no existing issue covers it. Spec-side note: upstream - WebAssembly/component-model#703 (merged 2026-08-18, after our submodule pin) - rewrote strongly-unique as canonicalize-then-compare per - WebAssembly/component-model#702 (non-transitivity); the acronym fold is - retained and applies to all names — the new invalid-examples list adds - `[static]foo-BAR.FOO-bar` and `foo:bar/BAZ` — so the divergence conclusion - is unchanged under both wordings. Re-verify wording/line numbers against - post-#703 Explainer.md before filing. Consequence for us: spec-invalid - method/static and interface-name case-collisions can reach the runtime's - conventions layer — one of the triggers tracked in #185; translator-side - enforcement is #238. **Deferred 2026-08-25 (operator): parked until the - wasmtime 49 toolchain bump** — upstream is actively reworking this corner - post-#703, so re-verify the bundled wasmparser then (probes in #238); if it - folds, the bump closes #238 and this entry dies, otherwise implement - shim-side and re-raise the filing question. **Update 2026-09-04: the - wasmtime-49-dev bump landed (git rev 4675ee1, wasmparser 0.258) — the fold - is still absent per upstream `test/nyi.txt`, so #238 stays open; re-verify - probes against 0.258 before re-raising the filing question.** - -[WebAssembly/component-model]: https://github.com/WebAssembly/component-model +- **wasm-tools parser/validator version skew:** older CLI releases rejected + corpus syntax accepted by the pinned `wast`/`wasmparser` crates. The + translator's wasmparser performs implementation validation; the spec remains + the semantic authority. `crates/testgen` uses + the pinned release train rather than a separately installed CLI parser. + See [architecture §4](docs/architecture.md#4-architecture) and §9. +- **wasmparser name case-folding:** historical probes recorded in + [polyengine#238](https://github.com/polymorph-components/polyengine/issues/238) + found method/static and interface-name collisions accepted where the + Explainer's name canonicalization rejects them. The August 2026 probes + covered wasmparser 0.251/0.252/0.256; a September toolchain-bump note cited + upstream `test/nyi.txt` for 0.258, not a fresh reproduction. Neither upstream + status nor the current translator behavior is established by those snapshots. + Re-run the issue's probes before proposing enforcement or an upstream filing. + The current `harness/src/xfail.ts` separately retains `validation/kebab.json` + exclusions under `name-rules-nyi`, linked to + [polyengine#248](https://github.com/polymorph-components/polyengine/issues/248). ## CM-3: `cancel_copy` returns a stale COMPLETED where wasmtime reports CANCELLED -**Status:** DRAFT — candidate upstream issue/PR against `definitions.py` -**Found:** 2026-08-08, implementing the stream copy protocol (M2 phase 2c review) +**Status:** DRAFT, unfiled. This is the approved corpus exception in +[architecture §1](docs/architecture.md#1-goals), reversible if upstream +adjudicates otherwise. The current pin still returns the pending payload +unchanged in `cancel_copy`. +**Found:** 2026-08-08, implementing the stream copy protocol. ### Evidence -`definitions.py` `cancel_copy` (line 2652): +`definitions.py` `cancel_copy`: ```python e.state = CopyState.CANCELLING_COPY @@ -182,8 +97,8 @@ is returned verbatim. For a stream write that was partially satisfied by a rendezvous, that event is `COMPLETED | (count << 4)` (armed by `on_copy` in `stream_copy`), so a subsequent `stream.cancel-write` reports COMPLETED. -wasmtime instead supersedes it -(`wasmtime-47.0.3 runtime/component/concurrent/futures_and_streams.rs:4004`): +The recorded wasmtime source comparison is against **47.0.3**, +`runtime/component/concurrent/futures_and_streams.rs:4004`: ```rust match (code, event) { @@ -194,22 +109,23 @@ match (code, event) { } ``` -i.e. an undelivered **stream** `Completed(count)` becomes `Cancelled(count)`; +An undelivered **stream** `Completed(count)` becomes `Cancelled(count)`; `Dropped` is unchanged, and a **future** `Completed` is unchanged. The official suite asserts wasmtime's answer, not the reference's: -`test/async/big-interleaving-test.wast:1520-1531` writes 8, reads 4, then -cancels the write and expects `0x42` (`CANCELLED | 4<<4`). Under the -reference's rule the answer is `0x40`. The neighbouring test at :1504 does not -disagree — it `poll`s the event first, so the cancel finds nothing pending and -takes the `shared.cancel()` path to CANCELLED either way, which is why only -the no-poll variant exposes the difference. +`test/async/big-interleaving-test.wast` writes 8, reads 4, then cancels the +write without polling and expects `0x42` (`CANCELLED | 4<<4`). Under the +reference's rule the answer is `0x40`. This no-poll write cancellation is +still present at the current pin. The neighboring test first polls the +write completion, then starts and cancels another read; it does not exercise +the disputed pending write event. The evidence is at lines 1520-1531 +(no-poll write cancellation) and 1504-1518 (poll/read cancellation). ### Why wasmtime looks right -The guest never observed the completion. Reporting COMPLETED would tell it the -write finished when in fact it was cancelled after copying 4 of 8 elements, and -the count alone cannot distinguish the two. +The original rationale favored CANCELLED for an unobserved partial write. +That intuition is not semantic proof; the approved exception rests on the +corpus/source evidence above, subject to upstream adjudication. ### Suggested change @@ -220,121 +136,74 @@ In `cancel_copy`, when the pending event is a stream `COMPLETED`, deliver ## CM-4: `sync-streams.wast:145` overfits wasmtime's scheduler — entry-status timing is not normative -**Status:** ADJUDICATED (operator, 2026-08-10) — upstream **test defect**, -not a reference-semantics issue. **Filing kit DROPPED as stale** (verified -only against 73b7ad5; the submodule pin has since advanced to 2f13265 -(CM#705), which rewrote the targeted assertion — `upstream-issue- -sync-streams-schedule-overfit.md` and `upstream-sync-streams-schedule- -agnostic.patch` were removed from the repo root; re-derive the patch against -the current pin before filing). Filing -itself remains tracked by -[polyengine#15](https://github.com/polymorph-components/polyengine/issues/15). -Archived evidence tree (mechanism docs, both experiment patches, trace, -verify script): `4f3351f:exams/wasmtime-exclusivity/`. -**Found:** 2026-08-08 (JSPI flip, M2 exit). **Mechanism corrected:** -2026-08-10 (#44 — the 08-09 analysis wrongly attributed a -release-at-resolution gate to wasmtime). **Runtime migrated:** hold gate + -drain-to-quiescence entry decision, PR #45. - -- Gate semantics — held for the whole core invocation, mid-frame parks - included — are agreed by `definitions.py`, wasmtime, and the - CanonicalABI.md prose alike; no semantics were ever in conflict. What - differs is a **scheduler policy**: *when* an async-lowered call's - STARTING/STARTED status is decided. The reference decides eagerly at - the call instant (STARTING in the wast scenario, under every schedule - it can produce); wasmtime defers until work queued ahead of the call - has drained (STARTED, deterministic under FIFO). Timing is not - normative: two conforming policies over identical gate semantics give - two different ABI-visible answers, so the hard STARTED assertion at - `sync-streams.wast:145` pins wasmtime's policy, not semantics — the - corpus was co-developed on wasmtime as its runner. -- Proposed upstream fix: make the assertion schedule-agnostic (accept - STARTING|STARTED; on STARTING, wait for the SUBTASK event, then assert - as today). Secondary, structural: run the wast corpus against the - reference in spec-repo CI — today `run_tests.py` is the only CI step, - so reference↔corpus contradictions have no detector by construction. -- Same class as NOTE-1 below (tests assuming a particular scheduler), - sharper instance: engine-policy overfit — the reference itself fails - the assertion deterministically. `definitions.py`, CanonicalABI.md, and - `test_callback_interleaving` all need **no change**; both previously - sketched amendments (release-at-resolution; deferred-entry - normativization) are withdrawn. -- polyengine disposition: hold-lifetime gate = spec conformance; - drain-to-quiescence entry decision (`Store.hasRunnableWork`, sole - consumer `createAsyncStartCall`, pinned by - `runtime/tests/entry_deferral_test.ts`) = deliberate **non-normative - scheduler policy** — satisfies the suite as written and is order-robust - under `POLYENGINE_SCHED_SEED` shuffles, unlike wasmtime's FIFO-dependent - formulation. Legal under any upstream adjudication of the test; no - flip-back trigger. +**Status:** ADJUDICATED locally as a schedule-dependent test assertion, not +a reference-semantics defect (2026-08-10). The original patch targeted +`73b7ad5`; upstream #705 subsequently rewrote the assertion. No current +upstream defect or filing is established by that patch. The local follow-up +record is [polyengine#15](https://github.com/polymorph-components/polyengine/issues/15). + +Historical evidence (mechanism notes, experiment patches, trace and verify +script) is preserved at `4f3351f:exams/wasmtime-exclusivity/`. + +- The instance-entry gate lasts for the whole core invocation, including + mid-frame parks. The reference, CanonicalABI.md and surveyed wasmtime + implementation agreed; the initial release-at-resolution interpretation + was wrong and was withdrawn. +- Entry-status timing is scheduler policy. The reference decided eagerly + and returned STARTING in the recorded scenario; wasmtime drained queued + work first and returned STARTED. The old test's hard STARTED assertion + distinguished conforming schedules, not gate semantics. +- Polyengine holds the gate for the invocation and defers entry while + runnable work remains (`Store.hasRunnableWork`, `createAsyncStartCall`, + `runtime/tests/entry_deferral_test.ts`). The latter is a non-normative + scheduler choice, not the CM-3 semantic exception. + +Any renewed upstream proposal must reproduce the issue against the current +test. The historical proposal was to accept STARTING or STARTED, waiting for +the SUBTASK event when necessary; neither the gate rule nor +`test_callback_interleaving` needed changing. --- ## NOTE-1: several official async tests assume the deterministic profile -**Status:** NOTE (documentation candidate, not a defect) -**Found:** 2026-08-08 (`async-calls-sync.wast` run-cb, M2 seeded-scheduling -investigation) +**Status:** Historical NOTE, not a current defect claim. -`async-calls-sync.wast`'s guest asserts each subtask's returned value equals -its index — an order pinned only by `DETERMINISTIC_PROFILE` -(definitions.py:1373): when backpressure clears, all waiters become ready at -once and the reference's `Store.tick` picks with `random.choice`. A host -exploring the spec's allowed nondeterminism beyond the deterministic profile -fails the guest's own assertion. Worth an upstream doc note on `test/async` -(tests assume the deterministic profile) or making the guests -order-tolerant. Hosts adding seeded-schedule testing should profile-scope -pins for such fixtures (we did). +The August 2026 seeded-scheduling investigation found +`async-calls-sync.wast` assertions tying each subtask's value to its index. +That order depends on `DETERMINISTIC_PROFILE`: when backpressure clears, +multiple waiters become ready and the reference's `Store.tick` can choose +among them. Recheck individual fixtures before extending this observation +to the current corpus. Seeded-scheduling exclusions must identify the +schedule-dependent assertion rather than treating a host order as normative. --- ## CM-5: `SharedFutureImpl.drop`'s pending-buffer assert looks internally inconsistent -**Status:** DRAFT — candidate upstream issue against `definitions.py` -**Found:** 2026-08-10, adversarial conformance review of the stream/future -territory (polyengine#84/#98) +**Status:** RESOLVED at the current pin by upstream commit `4acb0de` +([#708](https://github.com/WebAssembly/component-model/pull/708)). No filing needed. +**Found:** 2026-08-10, stream/future conformance review (polyengine#84/#98). ### Evidence -`definitions.py:1150` (`SharedFutureImpl.drop`): - -```python -if self.pending_buffer: - assert(isinstance(self.pending_buffer, WritableBuffer)) -``` +The original `SharedFutureImpl.drop` asserted that `pending_buffer` was a +`WritableBuffer` (a parked reader). A writable future cannot drop before +delivery, and a busy readable end cannot drop itself, so the reachable +pending side is instead a writer whose reader dropped first. -The assert says: if anything is parked on the shared future when an end -drops, the parked side is a *reader* (a future read parks a WritableBuffer -— the buffer the value will be written into). But by the surrounding rules -that state is unreachable from the drop paths: - -- a **writable** end may not drop before delivering its value - (`WritableFutureEnd.drop`, definitions.py:1183-1184 traps unless - `state == DONE`) — so a drop can never find the *reader* still parked via - this path with the value undelivered; -- a **readable** end that parked its read is the pending side itself; when - the reader end drops, `CopyEnd.drop` (definitions.py:1098-1101) traps on - a busy end before reaching the shared drop; -- and `definitions.py:2614` independently asserts a readable future end can - never observe DROPPED. - -The only guest-reachable pending side at shared-drop time is therefore a -*writer* (pending `ReadableBuffer`, reader dropped first — legal), which is -exactly what the assert rejects. Either the assert is inverted, or it -documents an invariant whose enforcing traps make the guarded branch dead; -in both readings it does not describe reachable states. +Both current `definitions.py` and CanonicalABI.md assert +`isinstance(self.pending_buffer, ReadableBuffer)`, matching that writer-side +buffer. This is the correction proposed by the original finding. ### polyengine disposition -polyengine's port omits the assert (`runtime/src/task/streams.ts`, -`SharedFutureImpl.drop`) — its teardown extension (#66/#84) *deliberately* -creates the writer-died-unwritten state for trap-poisoned instances and -resolves it with a reader-side trap, which the assert would spuriously kill. -No behavioral divergence on spec-reachable states. +`runtime/src/task/streams.ts` `SharedFutureImpl.drop` omits the assertion. +Instance-poisoning teardown also permits an unwritten writer to die and +traps its reader, an extension outside guest-reachable reference drop paths. +That teardown distinction remains; the upstream pending-buffer assertion +is no longer a defect. ### Suggested upstream fix -Either delete the assert (the neighboring traps already enforce the real -invariants) or flip it to assert the reachable shape -(`isinstance(self.pending_buffer, ReadableBuffer)`) with a comment naming -the reader-dropped-first case. +None outstanding for this finding; #708 corrected the assertion. diff --git a/upstream-consumer-findings.md b/upstream-consumer-findings.md index 3e8e15c..716ae6a 100644 --- a/upstream-consumer-findings.md +++ b/upstream-consumer-findings.md @@ -1,183 +1,104 @@ # Upstream consumer-repo findings -Single source for issues/PRs to file against the **polymorph consumer -repositories** (docs/consumers.md) — and the upstream toolchains their -components are built with — discovered while running their artifacts under -polyengine. Mirrors the conventions of -`upstream-component-model-repo-findings.md`: entries carry status -(`DRAFT` → `FILED #n` → `RESOLVED`), evidence, and proposed fixes. All -filing is the operator's (foreign repos). +Historical integration findings from consumer artifacts run under polyengine. +These are **snapshots, not a current audit of foreign repositories**. Paths, +versions and measurements below identify the observed artifact or source; +unless explicitly stated, they have not been rechecked in the current +consumer tree. `DRAFT` means unfiled, not confirmed still reproducible. + +Current integration guidance lives in [docs/consumers.md](docs/consumers.md). +Reproduce against current sources and obtain operator authorization before +filing publicly. Existing links below are evidence, not new filing claims. --- ## IROH-1 — endpoint holds a `RefCell` borrow across a post-resolution `block_on` (RESOLVED-BY-HOST via polyengine#43; see Disposition) -**Repo:** polymorph-iroh. **Where:** `endpoint/src/endpoint_impl.rs:13` -(claim: "the `RefCell` borrows never cross an await") vs the actual path: - -``` -State::drain() # under shared.borrow_mut() - -> noq/rustls handshake - -> Signer::sign # core/src/crypto/sign.rs:104 - -> wit_bindgen::block_on(polymorph:webcrypto/signature#signing-key.sign) -``` - -`block_on` on an async import blocks the thread mid-frame (sync -`waitable-set.wait` under the callback ABI); whether other same-instance -tasks may run during that window is the load-bearing question — see the -sharpened semantics below. Every other endpoint task parks in `wait_until` -(`endpoint_impl.rs:939`) whose first act is `shared.borrow_mut()` → -`RefCell already borrowed` → `unreachable` trap. - -**Evidence:** found by `exams/iroh-endpoint/` (polyengine C3 exam). -Instrumenting the host's `SigningKey.sign` shows the trap always lands -inside the TLS CertificateVerify signature window; relay-auth signs -(at bind, no poller parked yet) never trip it. Measured under -polyengine: ~90% of runs with `accept` parked across the -handshake. The 5 ms bounded-polling cadence (their jco workaround) -re-arms `wait_until` on the same timescale as the signing window, making -the collision near-certain on any host that interleaves there. - -**The precise semantics (corrected 2026-08-10; see -`upstream-component-model-repo-findings.md` CM-4 and -[polyengine#43](https://github.com/polymorph-components/polyengine/issues/43); exam kit archived -at `4f3351f:exams/wasmtime-exclusivity/`):** the -collision window was **polyengine-specific**, not spec-pinned. - -- *Before* `task.return`, a callback task's instance-entry gate holds - across mid-frame blocks on every implementation surveyed (polyengine, - wasmtime, and `definitions.py` alike) — borrows held across a - pre-resolution `block_on` are safe. -- *After* `task.return`: **wasmtime keeps holding the gate** for the rest - of the invocation (`do_not_enter` spans each core invocation; source + - trace verified), and gates event delivery to other same-instance tasks - the same way (`GuestCall::is_ready`, concurrent.rs:765). Under wasmtime - the poller *cannot* be resumed inside the pump's parked signing window - — the collision is **unreachable by semantics**, not by timing. - `definitions.py` agrees on the gate lifetime. **polyengine was the - outlier** until #43 landed: its since-removed release-at-resolution - rule (the 2026-08-09 CM-4 working assumption) admitted same-instance - tasks during the post-resolution parked span — that admitted window is - where this trap lived. The official suite (`sync-streams.wast`) pins - neither gate rule; its STARTED assertion is schedule-dependent (CM-4, - adjudicated 2026-08-10: an upstream test defect overfitting wasmtime's - deferred-entry scheduler policy). - -The endpoint's pump does its `block_on(sign)` **after** `bind` resolved, -inside the window polyengine's since-removed rule admitted, with the -`RefCell` borrow live. - -**Why the wasmtime leg is green — a semantics guarantee, not timing -luck (corrected 2026-08-10):** the previous revision of this entry -predicted latency injection would reproduce the trap on wasmtime; the -corrected model predicts the opposite — under wasmtime the poller's -timer event sits gated in `pending` until the pump's invocation exits, -at any signing latency. (Falsifiable both ways: add ~1 ms to the -wasmtime host's `sign`; the corrected model says it stays green.) Under -pre-#43 polyengine the same window was open by our own rule, and the 5 ms -poll cadence landed in it ~90% of the time with a `crypto.subtle` signer. - -**Disposition (2026-08-10, updated after polyengine#43 landed):** polyengine now -implements wasmtime's hold + deferred-entry model -([polyengine#43](https://github.com/polymorph-components/polyengine/issues/43)) — the admitted -window this trap lived in **no longer exists on any surveyed host**, by -semantics (pinned by `runtime/tests/entry_deferral_test.ts`). Empirical: -`just iroh-exam` scenarios 1/2/4/5 pass post-#43, including the -IROH-1-shaped legs (accept parked across a handshake, scenarios 2 and 4); -the exam's retry workaround for scenarios 2–4 is expected redundant and -can be retired after a few more green runs. (Scenario 3 fails on this -machine with a WebRTC backend-resolution error — differentially confirmed -pre-existing on pristine main, unrelated to #43.) **This entry is -RESOLVED-BY-HOST; no consumer filing needed.** The guest-side hygiene -below remains advisable independent of host (borrows across any -`block_on` are fragile under future spec evolution and under hosts -exploring allowed nondeterminism). - -**Proposed fix (guest-side, now optional hardening):** scope the borrow -inside `drain`'s inner steps, or move signing out of the borrowed region -(take what `sign` needs, release, sign, re-borrow). - -**Workaround in-tree:** the exam retries scenarios 2–4 (observed 8/20 -attempts trip it); residual all-attempts-fail probability < 1%. +**Status:** RESOLVED-BY-HOST; no consumer filing needed. +**Snapshot:** August 2026 polymorph-iroh endpoint exam, before polyengine#43. + +The observed path held `shared.borrow_mut()` through `State::drain()`, a +noq/rustls handshake, `Signer::sign`, and `wit_bindgen::block_on` of the +webcrypto signing import. With another endpoint task parked in `wait_until`, +pre-fix polyengine could resume that task during signing and trap on its +second mutable borrow. + +**Corrected interpretation:** this was a polyengine gate-lifetime bug, not +a consumer violation. The instance-entry gate spans the entire core +invocation, including post-`task.return` blocking. The reference and +surveyed wasmtime source agreed; polyengine's removed release-at-resolution +rule admitted the conflicting task. Deferred-entry status timing is a +separate scheduler choice; see +[CM-4](upstream-component-model-repo-findings.md#cm-4-sync-streamswast145-overfits-wasmtimes-scheduler--entry-status-timing-is-not-normative). + +**Disposition:** [polyengine#43](https://github.com/polymorph-components/polyengine/issues/43) +corrected the host rule, pinned by `runtime/tests/entry_deferral_test.ts`. +The historical exam passed the affected handshake scenarios after that fix. +Its old retry rates and proposed wasmtime latency reproduction do not describe +current behavior. Mechanism evidence is archived at +`4f3351f:exams/wasmtime-exclusivity/`; no current consumer workaround is asserted. --- ## WEBCRYPTO-PORT-1 — resource classes must be published under the DEFINING interface (RESOLVED in-tree; upstream doc note optional) -Not a consumer bug — recorded for the eventual upstreaming of -`ports/webcrypto`: `signing-key-options` is defined by -`polymorph:webcrypto/signature` (webcrypto.wit:604,613) and only `use`d -by `ed25519-sign`; a component linking both resolves the resource type -against the definer. Fixed in `ports/webcrypto/src/signature.ts` (the -class is published under both). General rule for all ports: every -resource class goes under its defining interface; `use`rs may re-export. +**Status:** Historical port correction, not a consumer bug. + +The former in-tree `ports/webcrypto` published `signing-key-options` under +`ed25519-sign`, which only `use`d the resource from +`polymorph:webcrypto/signature`. Components linking both needed the class +under its defining interface. The port was corrected to publish it there +as well. The general constraint remains: `use` imports may re-export a +resource, but do not replace its defining interface's binding. The old +`ports/` path is historical, not a current source location. --- ## POLYMORPH-TEST-HARNESS-1 — freshCases re-pick is a linear name() scan (quadratic in suite size) -`runCases`' freshCases branch re-enumerates and scans front-to-back, -calling `name()` per entry until the match (js/viewer/harness.mjs:181-189). -For an n-case suite that is Σi ≈ n²/2 `name()` round-trips per run — -~182M for polymorph-webcrypto's 19k-case shared suite — and every -`name()` is a host-boundary crossing on any runner. polyengine's ct-runner -mirrored the scan verbatim and now fronts it with a same-index-first -fast path (census index as a hint, full scan as the fallback, drift -semantics unchanged — `ct-runner/src/run-suite.ts` `findByName`); the -same fix transplants to harness.mjs directly. Filing upstream is the -operator's call. +**Status:** DRAFT, historical consumer-harness observation. + +The observed `js/viewer/harness.mjs` `runCases` implementation re-enumerated +fresh cases and scanned from the front, calling `name()` until it found the +requested case. Across an n-case suite this makes about n²/2 host-boundary +calls. No current consumer implementation is asserted here. + +Polyengine's `ct-runner/src/run-suite.ts` `findByName` tries the census index +first, verifies the name, then falls back to the scan if the index drifted. +That optimization preserves name-based selection and is a candidate for the +consumer harness if the original scan remains. --- ## IROH-2 — post-#71 redundant tier-(c) overrides: delete in favor of the wasi-shims parking kernel (DRAFT) -**Repo:** polymorph-iroh. **Where:** -`experiments/iroh-relay-ws/host/sockets.ts` (main, post their #40/#41 — -the polyengine-leg rewrite): its `Pollable`, `poll` and `monotonic-clock` -sections (~200 lines) re-implement the tier-(c) parking that -[polyengine#71](https://github.com/polymorph-components/polyengine/pull/71) shipped in -the engine's own WASI package (`@deltic/wasi-shims` when #71 landed; the -package is `@polyengine/wasi` today — this project renamed from deltic to -polyengine, and iroh still pins the `@deltic` line) — written before the -kernel existed, against -the same jco-shim ancestor, so the designs converged. - -**Why deleting wins (beyond dedup):** their `block()`/`poll()` are -`async` functions — EVERY call returns a Promise, so every call suspends -and pays the engine's continuation hop (polyengine jspi pin (j)) even when -the pollable is already ready; the kernel's implementations take sync -fast paths. Their duck-typed `PollableLike` seam (foreign pollables -without `waitPromise`) exists only because two `Pollable` classes -coexist; the kernel's publicly-constructible -`new Pollable(ready, wait)` is the interop seam that dissolves it — the -sockets provider mints kernel pollables and the whole foreign-pollable -distinction disappears. - -**Proposed fix:** on their next engine pin bump (past #71), drop the -`wasi:io/poll@0.2` and `wasi:clocks/monotonic-clock@0.2` entries from -`syntheticNetImports()` (wasiShims() now provides parking versions) and -replace the local `Pollable` with the kernel's. Their sockets surface is -unaffected (it constructs pollables; the constructor shape is -identical). Filing upstream is the operator's call. - -**Related:** [polyengine#74](https://github.com/polymorph-components/polyengine/issues/74) -tracks adopting their sockets surface + routing hooks into wasi-shims, -sequenced after this convergence so they swap once. +**Status:** DRAFT, snapshot after polyengine#71 and polymorph-iroh#40/#41. + +The observed `experiments/iroh-relay-ws/host/sockets.ts` duplicated pollables, +polling and monotonic-clock parking already supplied by the engine's WASI +package. Its async `block()`/`poll()` always returned Promises, including for +ready pollables; the engine kernel has synchronous ready paths. The observed +consumer still used the old `@deltic/wasi-shims` package name. Its current +pin and overrides have not been checked. + +**Candidate change:** after verifying the consumer's package API, use its +WASI kernel's `Pollable` and remove the duplicated poll/clock import entries. +The historical constructor shapes matched; do not assume they still do. +Related integration work was recorded in +[polyengine#74](https://github.com/polymorph-components/polyengine/issues/74). --- ## CGO-1 — `cabi_realloc` can call clock imports under GC pacing: traps (and poisons) under any conforming CM host (DRAFT) -**Repo:** componentize-go (upstream Go component toolchain; exact repo per -operator). **Where:** the generated `cabi_realloc` / Go runtime allocation -path. +**Status:** DRAFT, historical componentize-go artifact finding; no current +toolchain reproduction or wasmtime poisoning result is claimed. -**Evidence:** [polyengine#145](https://github.com/polymorph-components/polyengine/issues/145) — -consistently captured across ~10 reproductions against two -separately-composed wosh builds (downstream report: -[wosh#71](https://github.com/lann/wosh/pull/71)): +**Evidence:** [polyengine#145](https://github.com/polymorph-components/polyengine/issues/145) +and [wosh#71](https://github.com/lann/wosh/pull/71) recorded two composed wosh +builds trapping during cross-instance `list` lowering: -``` +```text Trap: cannot leave component instance 1 (may_leave violation) at clock_time_get (wasm) at runtime.clock_time_get (Go runtime) @@ -185,70 +106,43 @@ Trap: cannot leave component instance 1 (may_leave violation) at time.now ``` -from inside `cabi_realloc`, during the copy window of a cross-instance -`list` lowering. Under allocation pressure the Go runtime decides an -allocation crosses a GC-pacing threshold and reads the wall clock — from -inside the callee's realloc. - -**Spec basis (why this is theirs, not a host bug):** the reference sets -`may_leave = False` around the entire realloc call -(`definitions.py:670-683` `LiftLowerContext.reallocate`) and `canon_lower` -traps when it is false (`definitions.py:2244`); CanonicalABI.md's -"realloc must be called reentrantly…" paragraph makes the guard -load-bearing — it is what licenses compiling realloc calls as plain -synchronous calls instead of the specced fresh-thread semantics. Net -constraint: **`cabi_realloc` must not (transitively) call imports.** A -Go runtime that reserves the right to read the clock in any allocation -violates it on every conforming host: polyengine instance-poisons -(verified), wasmtime store-poisons (presumed from shared semantics; -unverified on wasmtime's host-lowering path — its native gates simply -never generated the same allocation pressure). - -**Proposed fix (at the source, fixes every host):** make the generated -`cabi_realloc` unable to trigger GC pacing — a pre-reserved arena sized -per copy, a GC hold across the realloc frame, or a runtime knob -deferring pacing clock reads while inside the CABI entry. Any of these -also removes the load-dependent flakiness (polyengine#145: the trap needs an -allocation to cross a GC threshold *inside* the window, so it only shows -under flood). - -**Related:** polyengine#145 (asks 1–3; ask 1 — refusals naming the poison -cause — implemented host-side), -[polyengine#147](https://github.com/polymorph-components/polyengine/issues/147) (polyengine's own -host-entry lowering runs realloc outside the window — the lenient gap -that made #145 reproduce only under composition). Filing upstream is the -operator's call. +The recorded call originated in `cabi_realloc`: allocation pressure caused +the Go runtime's GC pacing to read the clock within the realloc frame. + +**Spec basis:** `LiftLowerContext.reallocate` in the pinned `definitions.py` +sets `may_leave = False` around realloc, and `canon_lower` checks that guard. +Thus realloc cannot transitively leave the component through a lowered +import. The observed clock call violated that constraint. Polyengine +poisoning was reproduced; the old assertion of wasmtime store-poisoning was +an inference, not a measured result. + +**Candidate fix:** prevent GC-pacing clock imports during canonical realloc, +for example by deferring those reads or allocating from pre-reserved space. +The choice belongs to the toolchain and requires a current reproduction. +The related historical host-entry guard gap was tracked separately in +[polyengine#147](https://github.com/polymorph-components/polyengine/issues/147); +it is not evidence of a current runtime gap. --- ## POLYVISOR-1 — docs spike vendors a full deltic-0.1.0 engine bundle (DRAFT) -**Repo:** polyvisor. **Where:** `docs/spike-todomvc/app.js` — a checked-in -minified bundle containing an entire deltic-era engine copy (protocol -brands, canonical classes, plan validation, CABI; `RUNTIME_VERSION` -"0.1.0", `deltic.*/1` brand keys, `DELTIC_SCHED_SEED`). - -**Why it matters:** this is the one raw `deltic.witError/1` spelling left -anywhere in the consumer family (audited 2026-08-22 during the brand-key -renames; every live consumer imports the canonical classes via -`@deltic/runtime/embedder` or `@polyengine/runtime/embedder`, none -hand-roll brands or import the protocol package directly). A vendored -bundle is exactly the multi-copy scenario the deltic→polyengine rename -leaves undiagnosed (docs/consumers.md §"Pins and the scope rename"): its -brand namespace is disjoint from both the current `@deltic@0.2.x` line -and every `@polyengine` line, so if the spike is ever revived next to a -current engine copy, values simply go unrecognized — no error names the -mismatch. - -**Proposed fix:** regenerate (or delete) the bundle when polyvisor takes -its migration turn; until then it is inert documentation. Filing is the -operator's call. +**Status:** DRAFT, snapshot from the 2026-08-22 consumer brand-key audit. + +The observed `polyvisor/docs/spike-todomvc/app.js` contained a vendored +deltic-0.1.0 engine, including the old protocol brands. If revived alongside +a newer engine, its disjoint brand namespace could leave cross-copy values +unrecognized without a version-mismatch diagnostic. The bundle was inert +documentation in that audit; neither its continued presence nor its status +as the only old-brand occurrence is asserted now. + +**Candidate fix:** regenerate or delete the bundle if the spike is revived. --- ## Out of scope here, tracked where they belong - Spec/reference findings: `upstream-component-model-repo-findings.md`. -- `ports/webrtc` foreign-entry npm resolution (import-map requirement): - documented in `ports/webrtc/README-import-map note` and the exam's - deno.json; a Deno resolution mechanic, not a consumer defect. +- The former `ports/webrtc` foreign-entry npm resolution problem was a Deno + import-map requirement, not a consumer defect. Its old port/exam paths + are historical and are not current integration instructions. diff --git a/wasi/src/filesystem_node.ts b/wasi/src/filesystem_node.ts index 534fbd4..740ab42 100644 --- a/wasi/src/filesystem_node.ts +++ b/wasi/src/filesystem_node.ts @@ -26,16 +26,14 @@ // every path-taking op realpaths the parent directory (and, when // symlink-follow is set, chases the final component's link chain) and // refuses — `not-permitted` — anything that resolves outside the -// preopen's realpath root. Symlinks the guest creates itself therefore -// cannot be used to read or write outside the preopen, and neither can -// symlinks that were already in the preopened tree (issue #177). +// preopen's realpath root. This checks both guest-created and pre-existing +// symlinks (issue #177), subject to the race below. // // Residual risk is cross-process TOCTOU: node has no openat2 / // RESOLVE_BENEATH analogue, so between the realpath check and the OS -// call another PROCESS could swap a component for a symlink. The guest -// itself cannot interleave — every backend op is synchronous on the -// guest's own thread — so this is only reachable when something else -// with write access to the preopened tree races us. +// call another process or thread with write access to the tree could swap a +// path component for a symlink. Synchronous operations prevent interleaving +// on this JS thread, not concurrent filesystem mutation elsewhere. // // PLATFORM TRAPS the containment code deliberately works around — both // observed on Deno's node compat, both silent, both only when the diff --git a/wasi/src/mod.ts b/wasi/src/mod.ts index 1186fe3..d771bcd 100644 --- a/wasi/src/mod.ts +++ b/wasi/src/mod.ts @@ -1,17 +1,8 @@ -// `@polyengine/wasi` — the WASI providers for polyengine hosts, and the -// executable check that the embedder conventions -// (`@polyengine/protocol` — this package is protocol-only, per -// embedder-api.md §"The host-ABI surface and its version") -// serve WASI (docs/architecture.md §2 keeps implementations out of -// the RUNTIME — this package is where they live). Scope: p2 -// baseline + p3 clocks + à la carte sockets on BOTH tracks (the -// poll-shaped `@0.2` surface std::net links, and `@0.3` UDP + TCP -// client/listener; one node-builtins backend serving Deno and Node; -// `@polyengine/wasi/sockets`, issue #4, server-JS hosts only). Sockets is -// deliberately not merged here: this root module stays host-agnostic -// web-platform code, and `wasi()` merges only AMBIENT, side-effect- -// benign capabilities (time, entropy, stdio capture, an empty -// filesystem). Anything granting network egress or host storage is +// `@polyengine/wasi`: WASI providers using only `@polyengine/protocol` for +// the host ABI (contracts/embedder-api.md §"The host-ABI surface and its version"). +// This root module stays host-agnostic web-platform code. `wasi()` merges +// time, entropy, stdio capture, and an empty filesystem. +// Anything granting network egress or host storage is // opt-in regardless of how portable it is — sockets, the fetch-backed // http fragment, the host-stdio cli impl (`./cli-stdio` — real // stdin/stdout/terminal access; the unqualified `./cli` stays the @@ -51,8 +42,7 @@ // // `wasi(options)` returns one flat imports-record fragment, keyed by // compatibility-**track** keys per contracts/embedder-api.md §"Version -// canonicalization" (`@0.2`, `@0.3`) — this package is the flagship -// track-key-registration consumer: one `@0.2` provider serves every p2 +// canonicalization" (`@0.2`, `@0.3`): one `@0.2` provider serves compatible p2 // leaf regardless of whether the guest's binary says `0.2.6`, `0.2.9` or // `0.2.12`, and one `@0.3` union provider serves both // divergent `monotonic-clock@0.3.0` drafts the corpus actually links