Real-time Generation
Continuous Visual Synthesis as You Steer
Streaming text generation produces tokens left-to-right. Real-time generation produces a complete artifact that updates as the user manipulates inputs — drag a slider, the image updates within milliseconds. Krea Realtime Canvas, Leonardo Realtime, Stable Diffusion Live.
Framing
The problem
Streaming generation produces tokens left-to-right; visual iteration wants the whole artifact updating with every input twiddle, not waiting for a turn to finish.
The pattern
Render a complete output that re-synthesizes continuously as the user manipulates inputs — millisecond-grained, not turn-grained.
Why chat breaks here
Chat is turn-based; there is no native surface for "render this as I move" — every parameter change requires a new generation cycle.
Risks
Cost and energy of continuous regeneration; visual flicker can feel chaotic without smoothing or commit-to-result handoff.
Avoid when
The output is text or the user benefits more from a deliberate one-shot than from continuous re-synthesis.
Use when
Visual iteration benefits from continuous synthesis as inputs change, not turn-based regeneration.
DOPE evaluation
- Directability
- Manipulate any input — drag, sketch, drop reference — and watch the artifact respond live
- Observability
- The full output is visible at all times — no waiting for a turn to complete
- Predictability
- Each input change produces a coherent, deterministic delta to the artifact
- Explainability
- The mapping from input to output is visible in motion, so users learn the controls by using them
In the wild
- Krea AI · Realtime Canvas (Krea) — Drag a slider or sketch on the canvas; the generated image re-synthesizes ~30fps with no submit button. The clearest production example of generation-as-direct-manipulation — built on SDXL Turbo behind the scenes.
- Leonardo · Realtime Canvas (Leonardo.Ai) — Paint shapes with a brush, the image regenerates to match your strokes live. Style prompt stays fixed while the structure follows the canvas — a different cut of the same pattern.
- Figma · Make (real-time interface generation) (Figma) — Interface components regenerate as you steer through the design surface. Not yet pixel-fluid like Krea, but the same input-as-render-trigger contract applied to UI building blocks.
- fal · Realtime SDXL Turbo (fal.ai) — The infrastructure layer — sub-second generation served at scale. Krea, Leonardo, and many internal tools sit on top. Pattern relies on this latency envelope; without it, real-time collapses back into turn-based.
FAQ
When should I use the Real-time Generation pattern?
Visual iteration benefits from continuous synthesis as inputs change, not turn-based regeneration.
When should I avoid the Real-time Generation pattern?
The output is text or the user benefits more from a deliberate one-shot than from continuous re-synthesis.
What problem does Real-time Generation solve?
Streaming generation produces tokens left-to-right; visual iteration wants the whole artifact updating with every input twiddle, not waiting for a turn to finish.
Why is chat the wrong fit for this?
Chat is turn-based; there is no native surface for "render this as I move" — every parameter change requires a new generation cycle.
Related patterns
- Often paired with: Region Lock — Real-time canvas + region lock means continuous synthesis only where you want it; the locked rest stays still.
- Often paired with: Multi-Modal Input — Continuous synthesis benefits most when many input channels — sketch, reference, text, voice — feed the same live canvas.
- Alternative to: Parallel Alternatives — Many candidates side-by-side vs one artifact that reshapes as you steer. Different shapes for the same exploration goal.