Add Dynamic Parameter Rendering

[leszko]

Add support for defining parts of the frontend in plugins.

Demo

demo_dynamic.mp4

Sample plugin

It's simplest to understand how it works by looking at a sample plugin: https://github.com/leszko/scope-plugin-ui

1. Look at schema.py, it introduces custom UI fields, e.g.

    config_slider: int = Field(
        default=50,
        ge=0,
        le=100,
        description="Integer 0–100 in Settings",
        json_schema_extra=ui_field_config(
            order=3, is_load_param=True, label="Slider"
        ),
    )

2. Install the plugin with DAYDREAM_SCOPE_PREVIEW=1 uv run daydream-scope install git+https://github.com/leszko/scope-plugin-ui
3. Look the fields rendered in UI

What and how to define

Any field in schema.py that includes json_schema_extra will be rendered in the UI.
You can define the following keys inside json_schema_extra to control how the field is displayed:

• order (int) — Controls the rendering order in the UI (lower numbers appear first).
• category (str) — Determines where the field is displayed. Possible values:
  • input
  • configuration (default)
• component (str) — The name of a custom or complex UI component to use for this field.
• label (str) — The label shown in the UI. If not provided, the field’s description is used.
• modes (list[str]) — Specifies which modes the field is visible in.
  Defaults to ["text", "video"].

Resources:

• Original RFC: Notion
• Discord discussions: thread

Non-Goals

• Complex components can be improved, but I'll leave it for now, better to send a separate PR for that; the intention of this PR is to have support for primitive fields for plugins
• Complex components and their usage in SettingsPanel, I also leave it as a separate PR
• UI Schema JSON returned from the Server API endpoint (for now let's just use the same schema endpoint)

[yondonfu]

This looks pretty good! I tested install of the sample plugin and confirmed that the schema driven fields show up in InputAndControls + Settings panel and that updating the fields in the former trigger parameter update sends across the WebRTC data channel during runtime.

There seems to be a stray box by default when I open up the app and look at the default LongLive settings.

[yondonfu]

Does this belong here? The distinction between runtime vs. load params seems like a data level distinction as opposed to a UI level distinction?

[leszko]

We need to decide if the UI elements are disabled when the stream is active. E.g. you cannot modify height/width when the stream is ongoing.

Some other options:

• Rename to enabled_when_stream_active or something similar to clarify the intention
• Do not allow load params at all to define by plugins. So, all plugin-defined UI elements are always active

[yondonfu]

Right that's why the UI needs to know if it is a load param. I'm wondering if it makes sense to just make that distinction outside of json_extra_schema since this is just being used for UI logic and whether something is a load vs runtime param affects whether it should be passed in the pipeline init or the call which is not just a UI concern. The immediate thought is having some way to mark a field as runtime vs load eg separate load/runtime config. I don't have a strong pref here so just making a note. If you think this can be considered separately no problem.

[leszko]

Let’s keep things as they are for now.

I agree that the backend could eventually use is_load_param to decide whether params should be passed to __init__() or __call__(). However, at this stage, I think the simplest and safest approach is to pass all params to both.

The main reason I don’t want to introduce that separation yet is that some params will likely apply to both cases, which requires more careful design. For now, filtering this in the UI feels like the best and least complex solution.

[leszko]

This looks pretty good! I tested install of the sample plugin and confirmed that the schema driven fields show up in InputAndControls + Settings panel and that updating the fields in the former trigger parameter update sends across the WebRTC data channel during runtime.

There seems to be a stray box by default when I open up the app and look at the default LongLive settings.

Ahh, sorry, my bad I haven't removed this part form the PR after testing. Removed now.

[yondonfu]

Doesn't need to happen immediately, but for my own understanding will we be able to slim down BasePipelineConfig too? It seems to have a lot of stuff.

[yondonfu]

Will we be able to in a separate PR remove the app-defined sections if this is here?

[leszko]

Yeah, let's maybe do it in separate refactoring PR, try to convert every part of the Input & Settings section into Complex Components. For now, it's ok, these code should be applied to each pipeline (though it may be hidden for some), so it's mainly about refactoring it into Complex Components.

[leszko]

Doesn't need to happen immediately, but for my own understanding will we be able to slim down BasePipelineConfig too? It seems to have a lot of stuff.

Yes, agree. The same actually with some parts of the frontend, like StreamPage.tsx, schemaSettings.tsx, and SettingsPanel.tsx. I have a feeling there is too much happening there. But let's keep it as a separate PR. This is only have a lot of changes, I don't want to introduce additional refactor here.