Repository navigation
feat(ai-components): generated UI, with A2UI and OpenUI renderers - #3388
Merged
Merged
Conversation
…sign-in AIConversation lays an agent conversation out by turn. parseMessageParts reads a reply's ai_reasoning and ai_tool_call attachments, shown by AIReasoning and AIToolCall, with AIToolApproval for a tool that asks first and AIToolAuthorization for a connector sign-in, run in a popup by createPopupAuthorizer and finished by completeConnectorSignIn. Also AISuggestions, AIReplyActions, AIMessageComposer.ChatOptions, default attachment previews and a className merge on the composer. Mirrors StreamChatAI on iOS and stream-chat-android-ai-compose.
- Sanitise attachment links and images, keeping blob and image data URLs. - Keep waiting for a sign-in while a provider page severs the popup. - Retry loading older messages after a failure or an empty page. - Measure turns only when they change, and on scroll once a frame. - Drop the entry memo, so a new rating shows with unchanged messages. - Write the thinking clock only from effects. - Build ChatOptions on the SDK's Dropdown, and sizes on prettifyFileSize. - Keep one preview URL per picked image.
The counterpart of StreamChatAI's generative UI. A reply's ai_ui field shows in place of its text when the conversation's AIUIHost has a renderer for its format and version; otherwise the text shows. The components know no format: renderers plug one in. - AIUIDocument, AIUIAction and aiUIActionJSON: the router's ai_ui and ui_action, read and written as the Swift SDK does. - AIUIHost: renderers (first match wins), one engine per renderer kept while it lives, actions sent in order and checked against the router's limits first, web and email links only by default, client tools, per-reply state and capabilities. - App components described once in portable JSON shared with iOS and Android, with the agent's arguments checked and decoded before the view gets them. - AIConversation takes ui: the thinking indicator gives way to a reply whose UI shows, and reply actions wait until it is written. - A live reply's UI and its parts fade in, 0.1 s apart and at most 0.4 s later; replies from history show at once.
A new stream-chat-react/a2ui entry: createA2UIRenderer, an AIUIHost renderer for A2UI 1.0 and the counterpart of StreamChatA2UI, built on @a2ui/web_core and @a2ui/react 0.15.0 as optional, pinned peers. - Each reply has its own processor, fed each message as it completes; messages the agent rewrites start its surfaces again. Every spelling of 1.0 is read, and a surface naming no catalog gets the app's. - The app's components join the catalog from their portable JSON as on iOS (place_card is PlaceCard), under getstream.io:app-catalog and the basic catalog, with bound values worked out before they render. - Actions reach host.send with Swift's label and payload; openUrl goes through host.open from a tap only; Text renders with AIMarkdown. - Errors go to onError and the rest still shows; a batch one bad component spoils is fed again a component at a time. - The A2UI packages load in the browser when a reply first shows A2UI, so server rendering shows a placeholder. The entry is ESM only. - Stream's AI tokens theme the basic catalog's --a2ui-* variables.
A reply's UI being written faded in a frame after it mounted, and a page that paints no frames, such as one in the background or in some web views, never got one, so the UI stayed hidden. It now shows at most 50 ms later either way, and still fades in where frames are painted.
- A link in A2UI's Text, which the agent writes, opens through
host.open, as openUrl does, rather than wherever it points.
- A Modal opens as a native dialog: focus stays in it while it is
open, Escape and the backdrop close it, focus goes back to what
opened it, and it shows above a transformed layout. A trigger other
than a Button opens it from the keyboard.
- require('stream-chat-react/a2ui') says it is an ES module only, as
the A2UI packages it runs are, rather than failing in one of them.
Button, Text, Card, TextField, ChoicePicker and Icon render with Stream's components and theme; the rest of the basic catalog stays A2UI's own elements, themed. A Button's words keep the agent's case. A live reply's parts come in as the agent writes them, through a new useAIUIEntrance hook that puts the entrance on a part's own element. A reply from history now shows whole, its app components included. Icon draws Stream's icon for about half of A2UI's names; the renderer's `icons` and `iconFont` options cover the rest, and an icon none of them draws shows nothing rather than its name. A Modal whose trigger is a Button no longer wraps it in a second button: it looks the trigger up by the id the agent wrote, and follows it when it arrives later.
An agent's answer had no message styling of its own: lists fell back to the browser's 40px indent, headings, links and quotes were unstyled, paragraphs sat 8px apart at the chat's 20px line height, and a 12px inset pushed the text off the edge of the steps above it. The answer is now set in the SCR scale, without a bubble: 16/24 body, headings a step up with more room above than below, 12px between blocks, lists, links, quotes, rules, inline code and tables styled on SCR tokens, and the text starts at the steps' edge. The steps, reasoning, cards and typing indicator move from 15/13px to the scale's 14/12px. `ai-scale()` reads the SCR spacing, type and radius tokens with literal fallbacks, as `ai-token()` does for colours. Code text, punctuation and comments follow the SCR text colours, so they read in dark mode.
StreamingMessage typed two letters every 30ms, about 67 a second, however fast the answer arrived, so a 1,500-character reply delivered in 3 s took 22 s to finish on screen. By default it now sets its pace as text arrives: enough letters a frame to write what is behind within 400ms, and at least one, so a long chunk appears almost at once and a trickle types as it comes. Text there from the start still shows at once. letterIntervalMs and renderingLetterCount still give a fixed pace, and the core useMessageTextStreaming hook is unchanged.
DefaultMessageData gains a reply's ai_ui (AIUIDocumentJSON) and a person's ai_ui_action (AIUIActionJSON), so an app reads them typed through the custom data it extends, as it does the SDK's other default fields.
stream-chat-react/openui is the renderer of OpenUI for AIUIHost, the
counterpart of StreamChatOpenUI, built on @openuidev/react-lang and
@openuidev/react-ui (optional, exact-pinned peers). It loads them when a
reply first shows OpenUI.
- The app's components join OpenUI's chat library in every content slot.
- Actions go to host.send in StreamChatOpenUI's shape:
continue_conversation with { context, params, form, values }, secret
fields left out; links open through host.open.
- Each reply's form state is kept by the host.
- A finished program's errors go to onError once.
- openUIInstructions writes the agent's instructions with OpenUI's own
prompt generator.
- OpenUI's development inspector stays off the app's page unless
devtools: true.
- ai-components.css themes OpenUI with Stream's tokens.
Also adds a2ui and openui to typesVersions, adds AIUIHost.toolNames, and
dedupes @floating-ui to one copy.
a2uiInstructions(config) writes what an agent is told to write A2UI with, from the same configuration as StreamChatA2UI's prompt.json (components, catalogId, fence, keep, preamble, notes, rules, examples). It uses the A2UI core's own catalog, and the app's components come out as iOS writes them. It rejects with an A2UIInstructionsError when an example would not render. a2uiProblems(text, config) lists what is wrong with A2UI messages, as StreamChatA2UI's A2UIProgram does: unreadable or out-of-order messages, unknown components, dangling references, a missing root, orphans and cycles. It also checks each component against its schema as the renderer does. Both run on the A2UI core alone, loaded when called, so they work on a server or in a build. The app catalog helpers move to catalogDocument.ts for that, and an app component's required parameters are listed sorted, as on iOS.
A reply's UI that throws while it renders, a renderer whose packages can't load, a finished A2UI reply that made no surface (now from history too) and a finished OpenUI program that draws nothing each show the reply's text instead of an empty or stuck UI. The host gains refresh() and onRenderError, and A2UI and OpenUI share one runtime loader. A createSurface whose inline components hold a bad one shows the rest. The A2UI linter flags a surface created again without being deleted. Errors about the renderer rather than a reply reach onError without a message id. Shared helpers replace their copies, and the OpenUI test uses the repo's ResizeObserver fake.
The basic components drawn as Stream's apply what the agent says about a part with its accessibility (label, description, live, hidden), as A2UI's own parts and StreamChatA2UI do: a field's name goes on its input, and a Card or Text the agent names is a group. The A2UI and OpenUI renderers' capabilities carry the model's instructions with the app's components, as on iOS. Writing them loads the format's packages, so host.capabilities() is now async; without them, when they can't load, the rest still goes to the backend.
Juno is in release-v14 as #3376, squashed, with the composer fixes made after it was last merged here. The merge takes them, and leaves this branch as release-v14 with generated UI on top.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
This was referenced Nov 4, 2026
Contributor
|
Size Change: +59.7 kB (+4.76%) 🔍 Total Size: 1.31 MB 📦 View Changed
ℹ️ View Unchanged
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release-v14 #3388 +/- ##
===============================================
+ Coverage 86.84% 87.21% +0.36%
===============================================
Files 526 587 +61
Lines 16699 20082 +3383
Branches 5251 6324 +1073
===============================================
+ Hits 14503 17514 +3011
- Misses 2196 2568 +372 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
MartinCupela
approved these changes
Nov 4, 2026
The A2UI and OpenUI packages are optional peers at ~ ranges rather than exact versions, so an app with a newer patch installs without a peer conflict. Minors stay out: the 0.x packages change their API between them, and the renderers rely on parts of both beyond their documented API. The dev dependencies stay exact, so CI tests one version. The README says so, and that ai-components is in 14.15.0.
OpenUI's step numbers set z-index 2, and nothing contained it, so they showed over an app's layer above the conversation, such as Juno's call screen. A reply's generated UI is now its own stacking context. It is no containing block, so a fixed overlay inside it still covers the page, and OpenUI's popovers and A2UI's dialog render outside it anyway.
A2UI's borderless variant has no border or background so that it reads as a link, and iOS draws it in the accent colour. Ours was Stream's secondary ghost button, grey words that looked like a heading until hovered. It is now the primary ghost button.
- An app's component that throws leaves only itself out of a reply's UI, is tried again when its arguments change, and is reported once the reply is written; a reply whose UI failed while written is tried again once it is written. - Renderers tell the app what went wrong after rendering, the host forgets a reply that left only after the conversation renders without it, and the runtime loader no longer changes state while rendering. - host.send resolves to whether the action was sent. - A middle click opens a link of a reply's UI; the menu button doesn't. - The ai-components entry exports what the README documents, and the a2ui entry no longer exports its stream helpers. DefaultMessageData is empty again; apps type ai_ui themselves. - zod ^4.2.0, which OpenUI's app components need; zustand and @openuidev/react-headless are no longer peers of this package. - The build no longer ships the unreachable CJS A2UI renderer.
… does A reply's generated UI looks the same in either format, and as on iOS: the reply on a raised card, a callout tinted in its variant's colour behind its icon, chips as pills with the chosen ones in the accent colour, the main button a pill as wide as the reply, and follow-ups in the accent colour with a hairline between them, under no heading. A2UI buttons carry their variant as a class, and its chips are neutral buttons with a class for the chosen ones, so choosing one changes only their colour. The marks are CSS, with no images a page's content policy could block.
…-ups A2UI's Button works out from its surface whether it is a borderless button in a Column, and whether one is before it, and says so with its own classes, rather than the styles reaching for @a2ui/react's host elements. A borderless button elsewhere, such as in a Row, reads as a link. OpenUI's chosen chips drop their tick, so choosing one moves nothing, and an app's component decodes its arguments only when they change.
github-actions Bot
pushed a commit
that referenced
this pull request
Nov 4, 2026
## [14.16.0](v14.15.0...v14.16.0) (2026-11-04) ### Features * **ai-components:** generated UI, with A2UI and OpenUI renderers ([#3388](#3388)) ([d0a17b3](d0a17b3)), closes [GetStream/Vision-Agents#1516](GetStream/Vision-Agents#1516) [GetStream/stream-chat-swift#4370](GetStream/stream-chat-swift#4370) [GetStream/stream-chat-swift#4373](GetStream/stream-chat-swift#4373) [GetStream/stream-chat-swift#4374](GetStream/stream-chat-swift#4374) [GetStream/Vision-Agents#1712](GetStream/Vision-Agents#1712)
|
🎉 This PR is included in version 14.16.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
oliverlaz
added a commit
to GetStream/Vision-Agents
that referenced
this pull request
Nov 4, 2026
## Why
A reply can be UI the agent writes, and what the person does in it comes
back with their next message as `ui_action` on `POST
/v1/agents/sessions/{id}/responses` (#1516). Go has `client.UIAction`
and Swift `Responses.create(_:uiAction:)`; JavaScript had only the
generated type, so a web app could not send one. Juno on the web and
`stream-chat-react`'s generated UI (GetStream/stream-chat-react#3388)
need it.
## How it works
```
AIUIHost.send(action)
└─ session.responses.create(action.label, { uiAction: aiUIActionJSON(action) })
└─ POST /v1/agents/sessions/{id}/responses
{ text: label, request_id, ui_action: { v, message_id, format, name, payload } }
└─ router: the person's message carries it as custom.ai_ui_action,
and the model is told it after their words
```
| Change | Result | Test |
| --- | --- | --- |
| `CreateResponseOptions.uiAction`, exported `UIAction` (the generated
`ui_action` type) | sent as `ui_action` |
`sdks/js/tests/resource.test.ts`: the request carries `ui_action` |
| An empty `uiAction` | not sent | same file |
| A text-only question with an action | keeps its `request_id`, which
the router needs to keep the action on the message | same file |
## Validation
- `npm test` in `sdks/js`: 420 pass. `npm run types -- --check`: in step
with the spec.
- `go build ./...` and `go test ./internal/session/` pass on the branch
as merged with `accelerate`.
---------
Co-authored-by: Martin Mitrevski <martinmitrevski.oh@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
oliverlaz
added a commit
to GetStream/Vision-Agents
that referenced
this pull request
Nov 4, 2026
> Stacked on the JavaScript SDK's `uiAction` (#1711). Until it merges, this diff shows it too; this PR's own change is `examples/voice_agents/react_assistant`. ## Why `swift_assistant` plans a trip as interactive UI, in A2UI or OpenUI (#1516). Juno on the web (#1645) showed only text. With `stream-chat-react`'s generated UI (GetStream/stream-chat-react#3388), it shows the same replies: the same backend, the same `UIComponents.json`, the same actions back to the agent. ## How it works ``` reply's ai_ui ─> AIConversation ui={AIUIHost} ├─ createA2UIRenderer() A2UI 1.0 ├─ createOpenUIRenderer() OpenUI └─ tripComponents place_card, trip_map, calendar_check, from swift_assistant's UIComponents.json tap, form ─> AIUIAction ─> ChatScreen.send(label, uiAction) └─ responses.create(label, { uiAction }) ``` | Change | Where | | --- | --- | | A reply-format menu in the top bar: text, A2UI or OpenUI, the backend's `/reply-format` | `src/ReplyFormatMenu.tsx`, `src/App.tsx` | | The host with both renderers and the trip components; an action the screen can't send yet, while the agent answers, says so over the composer rather than doing nothing | `src/ChatView.tsx` | | The trip components: a Wikipedia place card, a Leaflet map, a calendar check | `src/tripComponents.tsx` | | A reply with UI is not empty; `ai_ui` and `ai_ui_action` typed on Stream messages; the action passed on with the question | `src/agentChat.ts`, `src/chatScreen.ts` | | OpenUI's styles, the a2ui and openui packages | `src/main.tsx`, `package.json` | The AI components and the generated UI come from `stream-chat-react` 14.16.0, released with GetStream/stream-chat-react#3388, which the example installs from npm. The branch also has a test, `TestAHandedOnCommandTellsTheModelWhatThePersonDidInTheUI` (`acceleration/internal/session/command_test.go`): a dispatch worker's session must tell the model what the person did, not only "Update the plan". The fix it was written for came with #1516, which it holds to. ## Validation - `npm test` in the example: 35 pass, including the trip components. `npm run build` (types, then the bundle) passes against `stream-chat-react` 14.16.0 from npm. - `go test ./internal/session/` passes on the branch as merged with `accelerate`. - Checked by hand against a local router and the `swift_assistant` backend: in A2UI and OpenUI, "Plan a weekend in Ohrid" streams in as a card with a map, places and the calendar; tapping a place and submitting the form send their actions, which reach the person's message as `custom.ai_ui_action` and get a new reply; reloading shows history at once; switching to text shows text. --------- Co-authored-by: Martin Mitrevski <martinmitrevski.oh@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Goal
An agent can answer with UI rather than text: a plan with a map, a form to change it, cards the person taps. The agents router carries a reply's UI to Stream Chat as
ai_uiand takes back what the person did in it asai_ui_action(GetStream/Vision-Agents#1516). StreamChatAI shows it on iOS (GetStream/stream-chat-swift#4370, GetStream/stream-chat-swift#4373, GetStream/stream-chat-swift#4374). This bringsstream-chat-react/ai-componentsto the same place, with A2UI and OpenUI renderers built on their official packages rather than on ports of them.🛠 Implementation details
ai-components, no new dependencies)AIUIHost,AIUIRenderer/AIUIEngine,AIUIDocument,AIUIAction,AIUIView,AIUIEntrance, app components from portable JSON (defineAIUIComponent), client toolsAIUIHost,AIUIRenderer,AIUIComponentAIConversationViewstream-chat-react/a2ui)a2uiInstructionsanda2uiProblemsStreamChatA2UIstream-chat-react/openui)openUIInstructionsStreamChatOpenUIWhere it goes further than iOS:
onRenderErrorsays why once the reply is writtenonRenderError/onErrorsay why; a reply that failed while written is tried again once it is writtenupdateComponentsorcreateSurface"v1.0"as the message versiononErrorsays why, andhost.sendresolves tofalsehost.opencallAgentFunctionThe A2UI and OpenUI entries are ESM only, client only (they load in the browser when a reply first shows the format, so server rendering shows a placeholder) and take their packages as optional peers, in
~ranges of the versions they are tested with since the 0.x packages change their API between minor versions, so apps that never show generated UI install and load nothing new. The README's Generated UI section covers the API.Tested with 3,373 passing tests, including the Ohrid trip plan rendered in both formats, StrictMode, server rendering, and a regression test for each failure case above that fails without its fix. Checked end to end with the Juno web example (GetStream/Vision-Agents#1712) against a local router: A2UI and OpenUI replies render, forms and cards send their actions, and history shows again on reload.
🎨 UI Changes
Generated UI shows in AI replies only when the app gives
AIConversationauihost; nothing changes otherwise. It is drawn the same in A2UI and OpenUI, as StreamChatOpenUI draws it on iOS: the reply on a raised card, callouts tinted in their variant's colour behind their icon, chips as pills with the chosen ones in the accent colour, the main button a pill as wide as the reply, and follow-ups in the accent colour with a hairline between them. Its marks are CSS, with no images a page's content policy could block.