When people do knowledge work, they constantly interpret, compare, and question what they see — but almost none of that reasoning reaches the AI.
Contexty capture those in-situ thoughts in place, then surfaces the AI's context as a canvas that users can inspect and correct.
In a within-subjects study (N=12) Contexty improved task awareness, thought structuring, sense of authorship, and perceived control. Answers grounded in snippet-based context were preferred in 78.1% of blind comparisons.
AI sees your activity, not your reasoning.
Much of what shapes a complex task happens in passing: why a finding feels significant, how two options are being weighed, which criterion just shifted. These judgments arise mid-task and disappear just as quickly.
Behavioral traces and screen observations can show an AI what you did and where you spent time, but not why you did it or how you understood it. Making that reasoning explicit often means pausing to write a prompt, and what you write can end up scattered across chat turns or folded into a system-generated summary. As a result, the AI may work from a view of your task that gradually drifts from your own.
Contexty treats those in-situ thoughts as core material for the AI's context and places them in a canvas you can inspect and correct.
One shortcut, from wherever you already are.
A global shortcut captures whatever is on screen and asks a single question: what were you thinking just then? A short memo — an interpretation, a doubt, a criterion — is attached to the captured snippet and sent straight into the AI's context. No app switch, no prompt to compose.
Passive screen observation and chat feed the same context alongside it.
How snippet memoing supports cognitive externalization
Ten daily LLM users completed a real-world sensemaking task using a probe system that enabled snippet memoing.
Expressing user-framed perspectives and forward-looking thoughts
Memos captured how people were making sense of something, not just what they were looking at, and let them note what they wanted to do next or how they hoped the AI would use the snippet.
Lower barrier to externalizing thoughts in situ
Because there was no need to pause and compose a full prompt or switch applications, people annotated continuously as they worked.
Captured snippets as cognitive scaffolds
Looking back at what they had seen and thought helped people keep a coherent view of the task, their own reasoning, and what they had already shared with the AI.
What it couldn't do
As snippets accumulated, people lost track of what they had captured and how the AI had interpreted it, and leaving the management of all that material entirely to them did not work.
Ground AI context in users' cognitive processes.
Treat in-situ thoughts as first-class inputs, rather than relying on system-inferred signals alone.
Make AI context inspectable and correctable by users.
Externalize AI's context so users can identify mismatches and fix them — keeping authorship and control with them.
Context you can look at — and correct.
Every capture — from computer use observation, snippet memoing, or the chat history — becomes an item on a canvas, and the canvas is what the AI actually reads. When you regroup items, correct misinterpreted labels, or remove something irrelevant, you are directly changing the basis for the next answer.
What one item holds
- ATitleA generated title that makes the canvas scannable at a glance.
- BProvenanceThe app, URL, and timestamp, read from the OS at capture time.
- CCaptured contentThe snippet itself, preserved verbatim.
- DTagsTags proposed by the system; users can add or remove them.
- EAI's understanding What the system infers the user was doing; users can correct this when it is inaccurate.
- FYour memoThe in-situ thought, pinned directly to the card.
New captures are scored against existing items and automatically grouped with related context.
Later captures can change the interpretation of earlier ones; affected items get an “Updated” badge.
Users can drag, regroup, and edit items, or issue high-level commands such as “group these by project,” and the system restructures the canvas accordingly.
Seeing the context changed how people worked.
We conducted a within-subjects study with twelve participants. The baseline kept snippet memoing, observation and grounded chat, butremoved the canvas, so isolates what seeing your context adds.
picked the snippet-grounded answer
For every query, the system silently generated two answers — one on the full context, one with all snippet-derived context stripped out — and participants chose one without knowing which was which. In this blind A/B comparison, snippet-grounded answers were preferred in 0.0% of cases.
Making AI context visible and editable helps users maintain task awareness, organize evolving thoughts, and stay in control of AI-assisted work, whether they actively reorganize the canvas or only glance at it.
BibTeX
@inproceedings{kim2026contexty,
author = {Kim, Yoonsu and Park, Chanbin and Son, Kihoon and Yang, Saelyne and Kim, Juho},
title = {Contexty: Capturing and Organizing In-situ Thoughts for Context-Aware AI Support},
year = {2026},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
booktitle = {Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology},
keywords = {Human-AI Collaboration, Context Sharing, User-Inspectable AI Context, Context-Aware AI Agents},
series = {UIST '26}
}
