Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Abstract
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
Community
What if your agent could remember your screen - without a model ever writing the memory?
We introduce activity frames: a deterministic, zero-model compiler that turns locally-captured screen activity into memory an agent can trust - and recurring tasks into scripts it can replay.
- A day of raw capture becomes a context block 86x smaller, compiled in 68 ms, byte-identical every run
- An agent reading it answers questions about the day at 98.4%, vs 66-80% for an LLM summary of the same capture
- Recurring routines replay at zero model tokens on a guard-matched hit - model fully out of the loop
- First measurements of two agent-cost parameters read from passive, pre-delegation human activity rather than agent rollouts: routine overhead R = 60-343x (a modeled upper bound) and delegable recurrence ~8%
Everything is local-first and open source (MIT): schema, compiler, MCP server, and the full evaluation harness.
Happy to answer anything - especially skeptical questions about the modeled numerator or the single-user corpus.
What if your agent could remember your screen - without a model ever writing the memory?
We introduce activity frames: a deterministic, zero-model compiler that turns locally-captured screen activity into memory an agent can trust - and recurring tasks into scripts it can replay.
- A day of raw capture becomes a context block 86x smaller, compiled in 68 ms, byte-identical every run
- An agent reading it answers questions about the day at 98.4%, vs 66-80% for an LLM summary of the same capture
- Recurring routines replay at zero model tokens on a guard-matched hit - model fully out of the loop
- First measurements of two agent-cost parameters read from passive, pre-delegation human activity rather than agent rollouts: routine overhead R = 60-343x (a modeled upper bound) and delegable recurrence ~8%
Everything is local-first and open source (MIT): schema, compiler, MCP server, and the full evaluation harness.
code: https://github.com/nossa-y/activity-frames
project: https://usenocta.app
Happy to answer anything - especially skeptical questions about the modeled numerator or the single-user corpus.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- PreAct: Computer-Using Agents that Get Faster on Repeated Tasks (2026)
- User as Code: Executable Memory for Personalized Agents (2026)
- Auto: The AGI Compiler (2026)
- Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings (2026)
- PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agents (2026)
- AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents (2026)
- Why Git Is the Memory Solution for the Agentic Development Lifecycle (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
The part that got me is the zero-model pipeline. No LLM in the loop, byte-identical output, mechanically auditable — that's the opposite of how most of us build agent memory, which is usually "dump everything into a vector store and hope retrieval figures it out." I've watched my own agents burn frontier inference re-deriving routines the user already did, so the diagnosis lands. But I want to interrogate the framing: it records what the user did, not what the user said. That's a real trade, not a free win. Screen activity tells you the keystrokes and the clicks, but a lot of intent lives in the reasoning that never touches the screen. Compiling actions into replayable frames is great for "do it again exactly," less obviously great for "do it differently next time." The deterministic angle is the honest part — no model means no drift, no hallucinated episode boundaries, and you can cache and diff the whole thing. I'd want to see how the typed frames hold up when the same routine happens across different apps or slightly different UI states, because that's where deterministic segmentation usually starts to fray. Still, shipping memory that's auditable instead of vibes is a direction I'd steal from.
yeah it's a real trade. replay is only for "same task, new values" - steps replay, slots change, and if it needs doing differently the model just does it. intent was left out of the measured tier on purpose. code can't see why you did something, so that stuff goes in tier 2 as tagged inference, because I didn't want guesses sitting next to measurements looking identical.
drift is where I expect it to break too honestly, it's why every step carries an expected element/role/app guard and a miss hands back to the model rather than clicking the wrong thing. coverage came out at 0.415 median so the savings in the paper are bounded by that number. one thing that surprised me in the live run: element refs came back different between two executions of the same plan, name grounding still resolved them. on the wrong page it grounded nothing and did zero actions, which is the behavior I wanted. cross-app I just don't claim - same routine in another app is a different routine to the compiler, at least today.
Get this paper in your agent:
hf papers read 2608.05784 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper