How Agents Work: A Deep Dive Into the Context Window
The LLM is the brain. The harness is the body. The context window is the live whiteboard the brain re-reads before every move.
LLM / Harness
To understand an agent, split the brain from the body:
The top row is the model. Its job is to read text and write text. The second row is the harness. Its job is to turn the tool calls the model writes into real actions.
GPT-5.5 or Opus 4.8 does not read files, run tests, edit code, open a browser, or access GitHub, Slack, or a database by itself. Codex, Claude Code, OpenClaw, Hermes, and Cursor are the systems that put tool manuals in front of the model, receive the calls it writes, execute those calls, and write the results back.
Before the model can decide the next move, the harness hands it one bundle of material.
That bundle is the context window.
The Context Window Is the Whiteboard in Front of the Model
The context window is the whiteboard the harness hands to the LLM before each model call.
The current task, system rules, developer instructions, tool manuals, previous dialogue, file contents, search results, command output, patch history, test results, screenshots, and reasoning traces all live on that board.
Anything that wants to shape the model's next judgment has to enter the window first: code search results, file reads, patch results, test output, screenshots, and browser feedback all travel through the same door.
Every time the LLM thinks about the next move, it re-reads that whiteboard. It has no other memory. It has the file in front of it.
Agent Loop: Think, Act, Read the Result
We analyzed 191 real local Codex session logs. Every turn follows the same heartbeat:
task_started ← you hit enter user_message ← your instruction enters the window reasoning ← the model thinks function_call ← it writes the command it wants to run function_call_output ← the harness executes it and writes back the result reasoning ← the model thinks again function_call ← it acts again ... (repeat N times) agent_message ← it reports back to you token_count ← the harness records the bill task_complete ← the turn ends
Notice the word function_call. The model did not run a command. It wrote down the command it wanted to run. The harness pressed the execution key. The result came back as function_call_output, got written onto the whiteboard, and the model read it before choosing the next move.
That is the whole difference between an agent and a normal chatbot. A single chat response is a closed-book exam: answer once, with no way to test it. The agent loop is an open-book lab: think one step, do one step, read the result, think again.
Across the 190K events we counted, function_call appeared 28,341 times and reasoning appeared 16,387 times. Each time, the model read the whiteboard from the top.
The Agent's Five Organs: Seeing, Editing, Feedback, Memory, Reach
The agent's body has five organs: eyes to perceive, hands to change the world, nerves to bring results back, a hippocampus to handle memory and compression, and tentacles to reach external systems.
These organs are tools exposed by the harness. The model writes a call in text, the harness executes the action, and the result goes back into the context window. On the next turn, the model reads a thicker whiteboard.
When we classify the tool calls from real sessions, they land cleanly in these five organs.
Organ 1: Eyes (seeing the world)
Every time the agent looks, what it sees gets written into the window. The eyes do not blink: what was seen does not disappear by default.
| item | role | measured |
|---|---|---|
| exec_command → cat/sed/ls | Read files | Read-class calls inside 23,630 exec calls |
| exec_command → rg/grep/find | Search code | Search output: 510K tokens |
| view_image | Inspect screenshots | 208 calls |
| web_search_call | Search the web | 532 calls |
| input_image (pasted by the user) | Receive an image you showed it | ~4,000 tokens per image |
Organ 2: Hands (changing the world)
Every time the hands move, the window gets another record of what changed. Edit the same file ten times, and the window now carries ten versions of that file trail.
| item | role | measured |
|---|---|---|
| apply_patch | The main code-writing path | 4,602 calls |
| exec_command → git/npm | Commit, install, build, test | — |
| write_stdin | Feed input into a running process | 2,234 calls |
| js | Run JavaScript | 1,067 calls |
Organ 3: Nerves (the feedback loop)
Eyes and hands make moves. The nerves make the loop work, because every move has an echo.
| item | role | measured |
|---|---|---|
| function_call_output | stdout, stderr, exit code | One output for each of 23,630 exec calls |
| Test / build output | The judge for whether the change worked | The heavy part of exec output |
| agent_message ↔ your reply | The human loop: it reports, you correct | 15,049 agent messages |
| token_count | The harness ledger | Once per turn |
Organ 4: Hippocampus (memory and compression)
When the desk fills up, compaction sweeps the surface into an encrypted compressed bundle so work can continue. The problem is that the model can no longer inspect that bundle clearly.
| item | role | measured |
|---|---|---|
| context window itself | Short-term memory; the working desk | ~272K ceiling |
| update_plan | A task note pinned to the whiteboard | 313 calls |
| compaction | Memory compression, stored as encrypted_content | 4 times / session |
| EchoMem tools | Long-term memory across sessions | 765 calls |
Organ 5: Tentacles (extension through MCP)
A tool is, at the model boundary, a JSON Schema instruction. To give an agent a new capability, you do not retrain the model; you put a new tool manual into the window.
| item | role | measured |
|---|---|---|
| MCP document tools | Read and write Google Docs, etc. | 7 tools, including _get_document_text |
| MCP Slack tools | Read Slack | 5 tools |
| MCP GitHub tools | Search repos and issues | 3 tools |
| spawn_agent / wait_agent / close_agent | Spin up sub-agents | multi-agent |
| tool_search_call | Search the tool directory first | 233 calls |
One Real Turn: How Your Instruction Becomes the Next Move
Now replay one real turn. Erik session, turn 74. You said:
For mobile, let's just do a simple vertical scroll experience
Frame 1: you speak. One user message enters the window.
Frame 2: look at what was already in the window. This turn's input was 228,649 tokens. Of that, 219,520 tokens were cached history replayed into the call. Only 9,129 tokens were new.
In other words, your sentence was floating on an ocean of history. New material was only 4% of the window; the other 96% was history: file reads, command output, patch records, encrypted reasoning traces, and four compaction bundles from the previous 73 turns.
Frame 3: the loop starts. The brain thinks. The eyes search rg "mobile". The eyes read related components. The hands patch the code with apply_patch. The nerves return each result to the whiteboard. The brain thinks again, the hands edit again, and finally the human loop reports back to you and waits for your next message.
Frame 4: every move leaves a trace. The search results, file reads, patch attempts, and every returned echo from this turn all stay in the window and become part of turn 75's 226,660-token history.
The brain moved. The organs moved. And every move made the next heartbeat's file thicker.
The Cost: Every Move Makes the Window Thicker
Put the five acts together and the shape becomes obvious: every move made by the five organs leaves a trace, and by default those traces are not cleared.
In one session we attributed token by token, 63.1% of the window was waste: old screenshots the eyes had seen accounted for 17.2%, old command output from the hands accounted for 24.3%, stale file reads accounted for 15.1%, and truly live file reading accounted for only 6.4%.
Your agent is not failing because it is dumb. It is bending under the weight of its own history.
Use This X-Ray Check in Your Next Session
Next time you open an agent session, keep these in view:
- Every screenshot you paste is roughly 4,000 tokens, and it stays in the window.
- Before every answer, your agent re-reads the whole window.
- Compaction is not cleaning. It is putting the mess into a compressed bag the model cannot inspect clearly.
- If you want the agent to remember across sessions, you need external memory: EchoMem or another memory tool.
There is no magic. There is a brain that keeps re-reading the full file, five organs that faithfully record what they do, and a desk that gets more crowded every turn. Once you see those three things, you can see the agent.
Data source: the context-golden-standard project's turn-by-turn teardown of real Codex sessions. Event counts come from the JSONL archive at ~/.codex/sessions. Per-payload token counts come from the response.create request bodies in the SQLite log at ~/.codex/logs_2.sqlite. Window composition comes from retained-window reconstruction and the SESSION_WASTE_DASHBOARD dataset.
Next: how dirty is your context window?
Once you understand how an agent works, the next step is to open the crowded desk and ask which tokens are still doing work, and which ones are just sediment.