Multi-Modal Input
Combine Sketch, References, and Text
Instead of forcing users to describe visual layouts purely through text, let them combine modalities — sketch a rough layout, point to reference examples, and add text annotations. The AI interprets each signal and shows how it weighs them.
Framing
The problem
Users often know something visually before they can phrase it.
The pattern
Let text, sketch, and references combine as weighted inputs.
Why chat breaks here
A single text channel forces premature translation of visual intent.
Risks
Multiple input modes can complicate attribution and onboarding.
Avoid when
The task is purely textual and extra modalities add noise.
Use when
Users have visual or audio context that text alone forces them to translate prematurely.
DOPE evaluation
- Directability
- Each modality is a separate input revised independently — redraw the sketch, swap a reference, set a reference's influence to subtle or strong — without disturbing the others
- Observability
- Sketch, references, and text each carry a visible weight readout, so how much each channel steers the result is displayed, not inferred from the output
- Predictability
- Weights behave like a mixing desk: raise the sketch channel and the layout tracks the drawing more, lower it and the references take over — the preview moves with the fader
- Explainability
- The preview updates against the weight mix in real time, so which input drove which aspect of the layout is demonstrated live, not asserted afterwards
In the wild
- tldraw Make Real (tldraw) — Sketch a UI + add sticky-note annotations + click "Make Real" — HTML/CSS comes back. Cleanest sketch+text→working-output example.
- Google Stitch (Google) — Galileo AI successor (Mar 2026 update) with AI-native infinite canvas, voice canvas, and the ability to feed sketches/screenshots/text together to generate up to 5 interconnected screens.
- Leonardo AI · Omni Models multi-reference (Leonardo.Ai) — Up to six reference images combined via plain-language relational instructions ("this character holds this product, in the colour scheme of this image") — multiple visual inputs plus text merged into one generation.
FAQ
When should I use the Multi-Modal Input pattern?
Users have visual or audio context that text alone forces them to translate prematurely.
When should I avoid the Multi-Modal Input pattern?
The task is purely textual and extra modalities add noise.
What problem does Multi-Modal Input solve?
Users often know something visually before they can phrase it.
Why is chat the wrong fit for this?
A single text channel forces premature translation of visual intent.
Related patterns
- Often paired with: Visual Intent Expression — Mood board feeds the visual channel; sketch and text fill the others.
- Often paired with: Disambiguation Branch — Multiple modalities can surface several readings before the run starts.
- Alternative to: Inline Prompt Controls — Three weighted channels vs structured fields parsed from a single prompt.