feat(stream): experimental Framegen WebGPU frame interpolation - #718
Open
MONZikWasTaken wants to merge 3 commits into
Open
Conversation
Add optional client-side neural frame interpolation for the embedded WebRTC stream path using the Framegen WGSL runtime (npm framegen). - New frameInterpolation settings (factor 2-4x, quality 360/480/720) - WebGPU pipeline overlay (web client only; disabled for native streamer) - Settings UI under Video with license notice for non-commercial weights - postinstall copies weights into public/; CDN fallback if missing - Unit tests for settings normalization Off by default. Adds display latency and GPU load when enabled.
Contributor
|
Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
|
@capyai review pr |
![capy-ai[bot]](https://avatars.githubusercontent.com/in/1915919?s=60&v=4){kind=small}
- Update bun.lock so CI `bun install --frozen-lockfile` succeeds - Abort in-flight frame steps after dispose/settings generation bumps - Clear sticky rejected weights cache and stop loops on hard runtime failure
Address remaining race and failure-path issues: - generation gate after every await; drop stale present queues - rebuild runtime when factor changes (not only resolution) - clear sticky rejected weights cache; set initFailed and stop loops - safer texture readback cleanup; device-lost invalidation
Author
Follow-up fixes pushedAddressed the Capy medium findings and CI lockfile issue:
Local verification
Note for maintainersCI shows |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an optional experimental WebGPU neural frame interpolation overlay (Framegen) for the embedded Chromium WebRTC streaming path, wiring it through shared settings, main-process settings migration, and renderer UI/pipeline code. This extends the existing “video shader pipeline” approach with a second overlay pipeline that can synthesize intermediate frames and present them on a canvas above the decoded <video>.
Changes:
- Introduces shared
frameInterpolationsettings (defaults + normalization + tests) and migrates persisted settings in main. - Adds a renderer-side
FrameInterpolationPipelineusing Framegen (framegennpm) and exposes controls in Settings → Video; integrates pipeline activation intoStreamView. - Adds postinstall weight-copy script + third-party notices, plus dependency and TypeScript/WebGPU typing updates.
Reviewed changes
Copilot reviewed 21 out of 25 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| opennow-stable/tsconfig.json | Adds framegen type shim path mapping and enables @webgpu/types. |
| opennow-stable/third_party/framegen/WEIGHTS_LICENSE.md | Adds a local notice for non-commercial model weights licensing. |
| opennow-stable/third_party/framegen/README.md | Documents Framegen usage and licensing split (runtime vs weights). |
| opennow-stable/src/shared/gfn/settings.ts | Adds frameInterpolation to Settings and default settings creation. |
| opennow-stable/src/shared/gfn/settings.test.ts | Ensures nested frameInterpolation defaults are fresh/mutable per instance. |
| opennow-stable/src/shared/gfn/index.ts | Re-exports new frameInterpolation module from the shared barrel. |
| opennow-stable/src/shared/gfn/frameInterpolation.ts | Defines types/defaults and normalization helpers for persisted/user-provided settings. |
| opennow-stable/src/shared/gfn/frameInterpolation.test.ts | Adds unit tests for normalization and activation helper. |
| opennow-stable/src/renderer/src/vite-env.d.ts | Adds WebGPU ambient types to the renderer environment. |
| opennow-stable/src/renderer/src/types/framegen.d.ts | Provides a local declare module "framegen" typing surface. |
| opennow-stable/src/renderer/src/platforms/gfn/index.ts | Exports the new renderer pipeline from the platform barrel. |
| opennow-stable/src/renderer/src/platforms/gfn/frameInterpolationPipeline.ts | Implements the WebGPU/Framegen runtime pipeline and presentation loop. |
| opennow-stable/src/renderer/src/components/StreamView.tsx | Activates/deactivates the pipeline based on settings + streaming mode. |
| opennow-stable/src/renderer/src/components/settings/stream/StreamVideoSection.tsx | Adds the new interpolation controls into the Video settings section. |
| opennow-stable/src/renderer/src/components/settings/stream/FrameInterpolationControls.tsx | Implements UI controls for enable/factor/quality and weights notice. |
| opennow-stable/src/renderer/src/App.tsx | Plumbs settings.frameInterpolation into StreamView. |
| opennow-stable/src/renderer/public/.gitkeep | Adds a placeholder file for the renderer public/ directory. |
| opennow-stable/src/main/settings.ts | Normalizes/migrates persisted frameInterpolation settings. |
| opennow-stable/scripts/copy-framegen-weights.mjs | Copies Framegen weights into public/ at install time and writes a notice file. |
| opennow-stable/package.json | Adds framegen dependency + runs the weight-copy script on postinstall. |
| opennow-stable/package-lock.json | Locks framegen and @webgpu/types additions. |
| opennow-stable/electron.vite.config.ts | Adds framegen to optimizeDeps.include for the renderer build. |
| opennow-stable/bun.lock | Updates Bun lockfile for new deps (even though npm is primary). |
| opennow-stable/.gitignore | Ignores copied weights directory under renderer public assets. |
| locales/en.json | Adds English UI strings for the new Frame Interpolation settings controls. |
Files not reviewed (1)
- opennow-stable/package-lock.json: Generated file
Suppressed comments (1)
opennow-stable/src/renderer/src/components/settings/stream/FrameInterpolationControls.tsx:102
- Same issue as above:
htmlForpoints at anidon a<button>chip, which isn’t a labelable form control. RemovinghtmlFor+ the dynamicidwill avoid a misleading association for assistive tech.
<label className="settings-label" htmlFor="settings-stream-frame-interpolation-quality">
{t("settings.frameInterpolation.quality")}
</label>
<span className="settings-value-badge">{fi.quality}p</span>
</div>
<div className="settings-chip-row">
{QUALITY_OPTIONS.map((quality) => (
<button
key={quality}
type="button"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+42
to
+44
| function align16(value: number): number { | ||
| return Math.max(16, Math.floor(value / 16) * 16); | ||
| } |
Comment on lines
+66
to
+80
| <label className="settings-label" htmlFor="settings-stream-frame-interpolation-factor"> | ||
| {t("settings.frameInterpolation.factor")} | ||
| </label> | ||
| <span className="settings-value-badge">{fi.factor}×</span> | ||
| </div> | ||
| <div className="settings-chip-row"> | ||
| {FACTOR_OPTIONS.map((factor) => ( | ||
| <button | ||
| key={factor} | ||
| type="button" | ||
| id={ | ||
| factor === fi.factor | ||
| ? "settings-stream-frame-interpolation-factor" | ||
| : undefined | ||
| } |
| }, | ||
| "frameInterpolation": { | ||
| "title": "Frame Interpolation", | ||
| "hint": "Experimental neural frame interpolation (Framegen WebGPU runtime) that synthesizes extra frames between decoded stream frames for smoother motion. Adds display latency and GPU load. Web client mode only; requires WebGPU + shader-f16.", |
Comment on lines
+24
to
+39
| mkdirSync(noticeDir, { recursive: true }); | ||
| writeFileSync( | ||
| join(noticeDir, "WEIGHTS_LICENSE.md"), | ||
| `# Framegen model weights | ||
|
|
||
| The files under \`src/renderer/public/framegen-weights/\` are the Framegen | ||
| v7-small neural frame-interpolation weights, redistributed from the | ||
| [\`framegen\`](https://www.npmjs.com/package/framegen) npm package. | ||
|
|
||
| **Non-commercial research and personal use only.** See the upstream notice: | ||
| https://github.com/MONZikWasTaken/Framegen/blob/main/WEIGHTS_LICENSE.md | ||
|
|
||
| Framegen **runtime code** (JavaScript/WGSL) is MIT-licensed separately. | ||
| `, | ||
| "utf8", | ||
| ); |
| @@ -0,0 +1 @@ | |||
| | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional experimental client-side neural frame interpolation for the embedded Chromium WebRTC stream path, powered by the Framegen hand-written WGSL/WebGPU runtime (
framegennpm).This is intentionally scoped like the existing video-filter pipeline:
third_party/framegen/User-facing behavior
Implementation notes
frameInterpolationonSettingswith normalize helpers + unit testsFrameInterpolationPipelinenext toVideoShaderPipelinepostinstallintosrc/renderer/public/framegen-weights/(gitignored); runtime falls back to pinned jsDelivr if missingTrade-offs (documented in UI)
Test plan
npm --prefix opennow-stable run typechecknpm --prefix opennow-stable test(includes new frameInterpolation normalize tests)