Pointer as Prompt
The cursor becomes the deictic anchor, voice becomes the verb
The cursor stops being a passive pointer and becomes the deictic anchor of a prompt — "this thing, here, right now". Voice carries the verb. The AI captures the visual and semantic context the cursor is resting on, names what it resolved, and acts in place. Distinct from Annotated Capture (which requires drawn markup that persists) and Contextual Overlay (which surfaces a UI at the cursor): here the cursor itself is the input channel. Demonstrated production-grade by Google DeepMind's AI Pointer for Chrome and "Magic Pointer" on Googlebook (May 2026).
Framing
The problem
AI tools live in their own windows — the user has to drag context into the AI rather than the AI meeting context where it already is, and describing what is "this" in prose multiplies tokens without adding precision.
The pattern
Make the cursor itself the deictic anchor of the prompt and voice the verb. The AI captures the visual + semantic context under the pointer, names what it resolved, and acts in place. Cross-app by default — the surface is the OS pointer, not the AI window.
Why chat breaks here
Chat is a destination users have to travel to with hand-written descriptions of what they're looking at. Pointer-as-Prompt inverts the geometry: AI travels to the user's cursor, with no app-switch and no prose translation in between.
Risks
Voice + cursor is a high-bandwidth input and a high-stakes one — misresolution of "this" can act on the wrong target. Without a visible captured-element confirmation, the pattern degrades into "blind action at cursor". Persistent always-listening surfaces also raise privacy expectations the user must consent to.
Avoid when
The action is sensitive enough that explicit selection + confirm is safer than gestural pointing, or the user's flow is keyboard-driven where reaching for cursor + voice is friction.
Use when
AI lives in its own window today — the user has to translate context into prose to get the AI to see it. Pointer-as-Prompt inverts the geometry: AI meets the cursor where it already is.
DOPE evaluation
- Directability
- Pointing and speaking fold the entire input loop into one gesture — no app switch, no copy-paste, no descriptive prose
- Observability
- Visible highlight around the resolved region — the user sees what context the AI captured before acting
- Predictability
- The same gesture (cursor on X + "do Y") produces the same in-place action across apps
- Explainability
- The captured context is named back to the user ("captured: link, status 404, in paragraph 3") before the action runs
In the wild
- Google · Gemini in Chrome (Select from screen to prompt) (Google DeepMind) — Cursor + Gemini in Chrome: hover, speak "fix this" or "show me directions" — the model resolves what is under the pointer (text, image, button) and acts in place. The canonical example of cursor-as-deictic-anchor + voice-as-verb at OS-adjacent scale.
- Google · Magic Pointer (Googlebook) (Google) — Same pattern on the Pixel laptop class — system-level pointer that captures context across native apps, not just the browser. Closest production approximation of "the OS cursor is the AI input channel" (announced May 2026, hardware ships fall 2026).
- Apple Intelligence · Image Wand (Apple) — Closest neighbour at the OS level — draw a rough shape around something on screen, the wand resolves the bounded region. Less continuous than a hover-tracking cursor, but the same "point and let the system resolve" geometry.
- Anthropic · Computer Use (Anthropic) — The agent-side analogue: the model resolves coordinates and clicks at them. Pointer-as-Prompt is the user-side counterpart — both treat the cursor as the resolution surface between intent and pixels.
FAQ
When should I use the Pointer as Prompt pattern?
AI lives in its own window today — the user has to translate context into prose to get the AI to see it. Pointer-as-Prompt inverts the geometry: AI meets the cursor where it already is.
When should I avoid the Pointer as Prompt pattern?
The action is sensitive enough that explicit selection + confirm is safer than gestural pointing, or the user's flow is keyboard-driven where reaching for cursor + voice is friction.
What problem does Pointer as Prompt solve?
AI tools live in their own windows — the user has to drag context into the AI rather than the AI meeting context where it already is, and describing what is "this" in prose multiplies tokens without adding precision.
Why is chat the wrong fit for this?
Chat is a destination users have to travel to with hand-written descriptions of what they're looking at. Pointer-as-Prompt inverts the geometry: AI travels to the user's cursor, with no app-switch and no prose translation in between.
Related patterns
- Alternative to: Annotated Capture — Draw persistent markup on a screenshot vs ephemeral hover + voice. Both fold pointing into the input — different ergonomics for different stakes.
- Often paired with: Contextual Overlay — The overlay is the surface that appears at the cursor; the cursor is the input channel that triggers it. They live on the same point.
- Extends: Select & Transform — Select & Transform is text-only and menu-driven; Pointer as Prompt extends the move to any content with voice as the verb.