Annotated Capture
Point at it — visually — instead of describing it
Describing a visual element with words is slow and ambiguous — "the third button from the left, next to the search bar". Annotated Capture lets the user draw on a screenshot directly: circle the element, arrow it, label it with the change you want. The annotations themselves become the prompt. Distinct from Multi-Modal Input (which composes a new artefact from sketch + text + reference) and Region Lock (which freezes parts of generated output): this pattern is markup of a captured surface as a steering signal. Made practical by vision-language models that can resolve per-pixel pointing — a 2024-era capability shift that turned hand-drawn arrows into a precise input channel. By 2026 the pattern ships natively, not as a workaround: Cursor's Design Mode puts click, multi-select, and draw-to-select annotation directly in the canvas, and Figma Make reads on-canvas annotations as build instructions.
Framing
The problem
Describing a visual element with words is slow and ambiguous — "the third button next to the search bar" leaves the AI guessing, and every clarification round multiplies tokens without adding precision.
The pattern
Let the user mark up a captured screen directly with arrows, circles, and labels. The annotations become the prompt: deictic markup ("this one, this big") instead of descriptive prose ("the one between X and Y").
Why chat breaks here
Chat is a single text channel; visual pointing has to be translated into prose at the user side and back into spatial reasoning at the model side, which loses precision at both steps. Vision-language models can now resolve markup directly, and first-class annotation canvases — Cursor's Design Mode, Figma Make — ship it natively rather than as a pasted-screenshot workaround.
Risks
Over-annotated captures get noisy and the AI can fixate on the markup instead of the underlying element. If the system silently reads only some markup types (e.g. circles but not arrows), the user has no signal which were used. Annotation tooling that drifts from native conventions (Apple Pencil, browser screenshot apps) creates a second thing to learn.
Avoid when
The visual is simple enough to describe in one sentence, the underlying element is text where Select & Transform is already a fit, or the user has no easy capture path (mobile-only flows without screenshot APIs).
Use when
Visual pointing translated into prose costs tokens, time, and precision — now that vision-language models can resolve markup directly, the translation layer is unnecessary.
DOPE evaluation
- Directability
- Add, move, edit, or delete annotations directly on the canvas before submitting
- Observability
- Every annotation is visible on the canvas — the user sees exactly what the AI is being told to look at
- Predictability
- Markup is unambiguous: a circle around an element means that element, not "something similar"
- Explainability
- Each annotation carries its tool kind and an optional label that names the change requested
In the wild
- Google AI Studio · Stream Realtime (Google) — Live screen-sharing session with Gemini where the cursor itself becomes a deictic gesture — the model resolves "this thing here" without prose. Closest production-grade approximation of pointing-as-prompt today.
- Apple Intelligence · Image Wand (Apple) — Draw a rough shape around something on screen and the wand resolves the bounded region as an input. Ships in iOS/macOS as part of Apple Intelligence — closest mass-market expression of the pattern.
- Figma Make · Canvas Annotations (Figma) — In-canvas comments and arrows that the AI-driven build flow reads alongside the design — annotations on a frame become instructions, not just feedback.
- Cursor · Design Mode (Cursor) — First-class annotation canvas (Cursor 3.0, April 2026; extended in 3.7): click, multi-select, or draw-to-select UI elements directly and add a note or voice narration — the agent edits from the pointing itself, replacing the old pasted-PNG workaround.
FAQ
When should I use the Annotated Capture pattern?
Visual pointing translated into prose costs tokens, time, and precision — now that vision-language models can resolve markup directly, the translation layer is unnecessary.
When should I avoid the Annotated Capture pattern?
The visual is simple enough to describe in one sentence, the underlying element is text where Select & Transform is already a fit, or the user has no easy capture path (mobile-only flows without screenshot APIs).
What problem does Annotated Capture solve?
Describing a visual element with words is slow and ambiguous — "the third button next to the search bar" leaves the AI guessing, and every clarification round multiplies tokens without adding precision.
Why is chat the wrong fit for this?
Chat is a single text channel; visual pointing has to be translated into prose at the user side and back into spatial reasoning at the model side, which loses precision at both steps. Vision-language models can now resolve markup directly, and first-class annotation canvases — Cursor's Design Mode, Figma Make — ship it natively rather than as a pasted-screenshot workaround.
Related patterns
- Alternative to: Multi-Modal Input — Compose a new artefact from sketch + text + reference vs annotate a captured surface for steering. Different relationships to the image.
- Often paired with: Region Lock — Lock regions of generated output + annotate regions of captured input. Two halves of the spatial-reference pattern.
- Alternative to: Select & Transform — Point at text by selection vs point at pixels by markup. Same pointing instinct, different substrate.