Streaming generation produces tokens left-to-right; visual iteration wants the whole artifact updating with every input twiddle, not waiting for a turn to finish.
← Back to patterns
Direct Manipulation: A Step Beyond Programming Languages
Visual & Generative
EmergingReal-time Generation
Continuous Visual Synthesis as You Steer
Streaming text generation produces tokens left-to-right. Real-time generation produces a complete artifact that updates as the user manipulates inputs — drag a slider, the image updates within milliseconds. Krea Realtime Canvas, Leonardo Realtime, Stable Diffusion Live.
Metaphor
Synthesizer
Interaction
Live, Continuous, Bidirectional
Use case
Image Generation, Video Sketching, Live Visual Iteration
Failure mode
Submit-and-Wait
Control mechanisms
Canvas · Sliders · Reference Drops · Brushstrokes · Live Preview
Reference examples
Krea AI Realtime Canvas · Leonardo Realtime Canvas · Figma Make (real-time interface generation) · fal Realtime SDXL Turbo
Interactive Example
The live side-by-side is the comparison.
This prototype already contains the contrast between chat and the pattern response. It stands as the primary comparison surface for this page.
Tokens arrive in a line. Artifacts respond in a field.
Streaming generation paints left-to-right, one token at a time, and every parameter change means waiting for a fresh turn to finish before you see what it did. For visual iteration this is the wrong grain entirely: you want to nudge one input and watch the whole image answer, but the turn-based loop forces you to commit blind and wait between each twiddle.
Real-time generation re-synthesizes the complete output continuously as you drag, sketch, or drop a reference — millisecond-grained, not turn-grained. Because the mapping from input to output is visible in motion, the user learns the controls by moving them, not by reading a manual. Each change is a coherent delta, so steering feels like operating an instrument rather than placing an order.
When the artifact answers every move within a frame, the control panel disappears and only the work remains.
DOPE Lens
How this pattern improves AI interaction quality.
Observability
The full output is visible at all times — no waiting for a turn to complete
Directability
Manipulate any input — drag, sketch, drop reference — and watch the artifact respond live
Predictability
Each input change produces a coherent, deterministic delta to the artifact
Explainability
The mapping from input to output is visible in motion, so users learn the controls by using them
Fit
Where it earns its place.
Use it
when
when
Visual iteration benefits from continuous synthesis as inputs change, not turn-based regeneration.
Avoid
when
when
The output is text or the user benefits more from a deliberate one-shot than from continuous re-synthesis.
In the wild
Where this pattern already ships.
The pattern is shipping but still maturing across products. Entries below show how different teams have approached it; expect variation as the space settles.
Krea AI · Realtime Canvas
Krea
Drag a slider or sketch on the canvas; the generated image re-synthesizes ~30fps with no submit button. The clearest production example of generation-as-direct-manipulation — built on SDXL Turbo behind the scenes.
Visit Krea ↗
Leonardo · Realtime Canvas
Leonardo.Ai
Paint shapes with a brush, the image regenerates to match your strokes live. Style prompt stays fixed while the structure follows the canvas — a different cut of the same pattern.
Visit Leonardo.Ai ↗
Figma · Make (real-time interface generation)
Figma
Interface components regenerate as you steer through the design surface. Not yet pixel-fluid like Krea, but the same input-as-render-trigger contract applied to UI building blocks.
Visit Figma ↗
fal · Realtime SDXL Turbo
fal.ai
The infrastructure layer — sub-second generation served at scale. Krea, Leonardo, and many internal tools sit on top. Pattern relies on this latency envelope; without it, real-time collapses back into turn-based.
Visit fal.ai ↗
Research Frame
Why this pattern exists, and where it breaks.
Render a complete output that re-synthesizes continuously as the user manipulates inputs — millisecond-grained, not turn-grained.
Chat is turn-based; there is no native surface for "render this as I move" — every parameter change requires a new generation cycle.
Cost and energy of continuous regeneration; visual flicker can feel chaotic without smoothing or commit-to-result handoff.
Common questions
What people ask about this pattern.
When should I use the Real-time Generation pattern?
+
Visual iteration benefits from continuous synthesis as inputs change, not turn-based regeneration.
When should I avoid the Real-time Generation pattern?
+
The output is text or the user benefits more from a deliberate one-shot than from continuous re-synthesis.
What problem does Real-time Generation solve?
+
Streaming generation produces tokens left-to-right; visual iteration wants the whole artifact updating with every input twiddle, not waiting for a turn to finish.
Why is chat the wrong fit for this?
+
Chat is turn-based; there is no native surface for "render this as I move" — every parameter change requires a new generation cycle.
Grounded in Research
Not just opinion.
The papers, guidelines, and usability studies this pattern rests on.
Ben Shneiderman
IEEE Computer · Vol. 16, Iss. 8 · 1983
The original case for visible, continuous, reversible direct manipulation over command-line abstraction. The patterns that move from prose-prompt to controllable surface are this argument applied to LLM I/O.
Related Patterns
Curated connections, not just neighbors.
Hand-picked relationships — what this pattern pairs with in practice, what it replaces, what it builds on, and what builds on it.
Often paired with
Region Lock
With Region Lock, continuous synthesis runs only where you want it; the locked parts stay still.
Often paired with
Multi-Modal Input
Feed Multi-Modal Input's sketch, references and text into the live canvas, and it re-renders with every weight change.
Alternative to
Parallel Alternatives
Real-time Generation reshapes one artifact as you steer; Parallel Alternatives shows three candidates side by side.