Context Engineering: Cost, Latency and Security in AI Agents

An agent asked to fix a failing test may make twenty model calls before it answers. The first call carries a system prompt, a few thousand tokens of tool definitions and the user’s request. By the tenth call it is also carrying three file reads, two test runs, a stack trace, a search result and the model’s own earlier reasoning. None of that was written as a prompt, yet all of it shapes the next decision. Context engineering is the work of deciding what goes into each of those calls. As AI applications become stateful, tool-using systems, the wording of the prompt becomes one part of a larger problem: managing context for quality, cost, latency and security all at once.

That last part is the argument of this piece. Most explanations of context engineering treat it as a quality discipline, about getting the right information to the model. It is that, but the same decisions also set the input bill, the time to first token and the set of instructions an attacker can reach. A choice made for one of those reasons changes the other three.

Key Takeaways

• Context engineering decides the exact tokens in each model call. Prompt wording is one part of it, and in agents often a small one.
• Agent loops are input-heavy. In the worked example, input was over 90% of the uncached cost, and good cache layout cut the run by about 78%.
• Prompt order is a cost decision. A volatile value at the top of the prompt can make caching more expensive than not caching.
• Long context mainly lengthens prefill and time to first token. The size of the effect depends on hardware, serving and cache hits.
• Everything in context can steer the model. Treat retrieved text, tool outputs and third-party tool descriptions as untrusted, and enforce permissions outside the model.
• Every context rule should have a test: task success, tokens, cache-read share, latency and injection resistance.

---

Quick Navigation

---

The shift from prompt engineering to context engineering

The term is newer than the practice. Anthropic’s Applied AI team, in a September 2025 engineering post titled Effective context engineering for AI agents, described it as “the natural progression of prompt engineering”. It defined context engineering as “the set of strategies for curating and maintaining the optimal set of tokens” during inference. That is a useful working definition, but it is a practitioner’s framing, not a standard. Retrieval-augmented generation, conversation summarisation and tool design all existed before anyone grouped them under one name.

So “replaced” overstates it. Prompt design did not go away; it became one input among several. A well-written system prompt still matters, but in an agent it may be only a few percent of the tokens in a late-stage call. The rest arrives from places the prompt author did not write and often never sees.

Table: scroll sideways to view all columns when needed.

Discipline	Main question	Typical artefacts
Prompt engineering	How should the instructions be worded?	System prompt, few-shot examples, output format
Retrieval (RAG)	Which external documents are relevant to this query?	Index, retriever, reranker, chunks
Memory	What from earlier sessions or turns should persist?	Notes, summaries, user profiles, vector stores
Orchestration	Which call happens next, and with which tools?	Agent loop, planner, sub-agents, tool router
Context engineering	What exact set of tokens enters this call, in what order, and with what trust level?	Context assembler, budgets, compaction rules, cache layout

Context engineering sits across the other four. Retrieval decides what might be relevant. Memory decides what might be kept. Context engineering decides what is actually sent, and that decision is made again for every call.

The context budget: what actually enters a model call?

Three numbers are often confused. The context window is the maximum the model will accept. The context budget is what an application chooses to spend, given cost and latency targets. The task requirement is what the call actually needs to succeed. In a healthy system the third is well below the second, and the second well below the first. UniverseBlend’s LLM context window comparison makes the same point from the other side: a model accepting a million tokens “says nothing about how well it uses them”.

A single call in an agent loop typically assembles:

• System instructions: stable, written by the developer, high trust.
• Tool definitions: stable within a session. They are often larger than people expect, because every tool brings a name, a description and a JSON schema.
• Conversation history: grows every turn.
• Retrieved material: documents, search results and file contents, chosen per query.
• Tool outputs: API responses, logs and test results, frequently verbose and rarely edited.
• Summaries and notes: compressed forms of earlier work, written by the model itself.

These components differ in how often they change, how much they cost, and how far they can be trusted. That is why context assembly should be task-specific. A call that decides which file to open needs the file list, not the contents of last week’s design document. A call that writes the final answer may need the reverse.

Anthropic’s post recommends several patterns for keeping the budget under control:

• Compaction: summarising a conversation as it approaches the limit and starting a fresh window.
• Tool-result clearing: dropping raw outputs once they have been used.
• Structured note-taking outside the window.
• Sub-agents: each works in a clean context and returns a condensed summary.
• “Just-in-time” retrieval: the agent keeps lightweight identifiers, such as file paths or queries, and loads content only when needed.

Each of these is a decision about what survives into the next call.

The cost of context engineering

Agent loops are input-heavy in a way chat is not, because each call re-sends the accumulated transcript. A worked example makes the size of this visible.

Hypothetical workload. Rates are Anthropic’s published Claude Opus 5.5 prices, checked on 22 October 2026: $4 per million input tokens, $5 per million for 5-minute cache writes, $0.20 per million for cache reads, and $20 per million output tokens.

The agent makes 20 calls. Each carries a 12,000-token stable prefix (system prompt and tool definitions). The transcript grows by 1,500 tokens per call (tool output plus the model’s reply). Each call generates 500 output tokens. Across the run, that is 525,000 input tokens and 10,000 output tokens, and the last call alone sends 40,500 input tokens.

Table: scroll sideways to view all columns when needed.

Strategy	What is billed	Total
No caching	525,000 input × $4 + 10,000 output × $20	$2.30
Cache the stable prefix only	12,000 written once, 228,000 read, 285,000 uncached history, output	$1.45
Cache the growing prefix each turn	40,500 written, 484,500 read, output	$0.50
Prefix broken on every call (e.g. a timestamp at the top)	525,000 written at $5, output	$2.83

Three things stand out. Input is $2.10 of the $2.30 uncached total, so here context, not generation, is the bill. Caching the whole growing prefix cuts the run cost by about 78%. And a single volatile value placed early in the prompt makes caching cost more than not caching, because every call pays the write premium and none of them get a read.

That last row is the context-engineering lesson. Anthropic’s prompt-caching documentation says cache hits “require 100 percent identical prompt segments” and that prefixes are built in the order tools, system, then messages. OpenAI’s prompt-caching guide tells developers to keep stable content first and dynamic content last, and to move timestamps and user-specific values later. On both platforms, cache economics depend on where things sit in the prompt, not just on how much there is.

Caching also has limits worth stating. Anthropic’s default cache lives five minutes (refreshed on use), so a tool call that runs longer than that lets the prefix expire. Opus 5.5 will not cache prompts under 512 tokens, and OpenAI’s minimum on GPT-5.6 and later is 1,024. On Anthropic’s API, changing a tool definition invalidates the entire cache. The worked example assumes none of these interruptions happen, and a real run will do worse.

The latency tax of context engineering

Inference has two phases. In prefill, the model processes every input token, largely in parallel, and builds the key-value (KV) cache that generation will read from. In decode, it produces output one token at a time. The Splitwise paper from Microsoft researchers calls these “a compute-intensive prompt computation” and “a memory-intensive token generation”. The DistServe authors tie time to first token (TTFT) to prefill, and time per output token to decode.

Long context therefore lands mainly on TTFT. A 40,500-token prompt has far more prefill work than a 12,000-token one, and that work happens before the user sees anything. The relationship is not a fixed per-token cost, though. It depends on the attention implementation, the hardware, how the provider batches requests, queueing at the moment of the call and, above all, cache hits. Both major providers say caching reduces time spent processing input. Anthropic says users “generally see improved time-to-first-token for long documents”. Neither publishes a universal figure, and this article does not invent one. UniverseBlend’s LLM inference latency measurements show how far TTFT varies by provider even before prompt length is considered.

For agents, the model’s prefill time is only part of the wait. End-to-end latency adds network round trips, retrieval queries, tool execution and orchestration logic, multiplied by the number of steps. Trimming context helps the inference share of that. It does nothing for a slow database query or a test suite that takes four minutes. A latency budget for an agent should be measured per component before anyone decides to cut context to fix it.

Context engineering is also a security boundary

Everything in a model’s context can influence its next action. That is how the model is designed to work, and it is also the security problem. Greshake and colleagues’ 2023 paper on indirect prompt injection put it plainly: “LLM-Integrated Applications blur the line between data and instructions”. An attacker who cannot reach the prompt can still plant text in a web page, a document or a code comment that the application is likely to retrieve.

OWASP’s LLM01:2025 Prompt Injection entry separates direct injection from indirect injection, “when an LLM accepts input from external sources, such as websites or files”. It also notes that RAG and fine-tuning “do not fully mitigate” the risk. Agents widen the set of sources: retrieved documents, tool outputs that talk back, earlier conversation turns, and tool descriptions themselves. The Model Context Protocol specification says “descriptions of tool behavior such as annotations should be considered untrusted, unless obtained from a trusted server”. It also states that “MCP itself cannot enforce these security principles at the protocol level.” UniverseBlend’s eight classes of prompt injection maps how each of these entry points behaves.

An illustrative scenario, not a reported incident: a support agent can read tickets and issue refunds through a tool. A customer submits a ticket whose body contains text formatted like a system note telling the assistant to refund the order in full and close the ticket. The ticket enters the context as retrieved material. Whether the model follows it depends on the model, the prompt and the formatting. Context engineering can lower the odds: mark the ticket body as untrusted data, keep it in a delimited block, and avoid putting it near the instructions. None of that is a guarantee. OpenAI’s research on an instruction hierarchy reports that training models to prioritise privileged instructions substantially improves robustness. That lowers the risk; it doesn’t remove it.

The control that actually holds sits outside the context. The refund tool should check, on the server, that the caller is authorised to refund that order, that the amount is under a limit, and that refunds above the limit need human approval. OWASP’s mitigations say the same: least privilege, functions handled in code rather than by the model, and “human-in-the-loop controls for privileged operations”. The distinction to keep is between what the model sees and what the application lets it do. Context filtering reduces exposure. Authorisation, input validation on tool arguments, sandboxing and approval gates decide the damage.

A context engineering architecture that can be tested

Treating context as an engineered component means every assembly rule should have a measurement attached. A practical architecture has five parts:

• An assembler that builds each call from typed components (instructions, tools, history, retrieved items, tool outputs), each tagged with its source and trust level.
• A budget per call type, set in tokens. Compaction or retrieval triggers when the budget is reached, not when the model’s limit is.
• A cache layout: stable components first, in a fixed order, with volatile values pushed to the end.
• A retrieval policy that prefers identifiers and on-demand loading over pre-stuffing.
• An enforcement layer for tool calls, independent of anything in the context.

Table: scroll sideways to view all columns when needed.

Decision	What to measure	How
What stays in the active context	Task success rate vs tokens per call	Run the same eval set with and without each component
What is retrieved on demand	Retrieval precision and recall; task success	Labelled queries with known relevant documents
What is summarised	Accuracy of facts carried across compaction	Probe questions about pre-compaction details
What is removed	Success rate and tokens after clearing tool outputs	Ablation on long-running tasks
What is cached	Cache-read share of input tokens; cost per task	Provider usage fields per request
What is treated as untrusted	Injection success rate	Seeded malicious documents and tool outputs in a red-team suite
What needs independent enforcement	Unauthorised actions attempted vs executed	Server-side logs of denied tool calls

The cache row is the easiest to start with, because both major providers report cached tokens in the response usage data. The security row is the one most often skipped. An agent that has never been tested against a poisoned document hasn’t shown that its context handling is safe.

When more context is the wrong answer

Long windows invite a simple strategy: include everything and let the model sort it out. The evidence says that is not free. The Lost in the Middle study (Liu et al., TACL) found that performance was “often highest when relevant information occurs at the beginning or end of the input context” and dropped when the relevant facts sat in the middle. Chroma’s Context Rot report, testing 18 models in July 2025, found that performance “grows increasingly unreliable as input length grows” and that “even a single distractor reduces performance”. The authors are clear that their evaluation is “not exhaustive of real-world use cases” and that they do not explain the mechanism. These are measured tendencies on specific tasks, not a law that every extra token makes every model worse.

In practice, more context goes wrong in a few recognisable ways:

• Stale information (an earlier file version, a superseded plan) competes with current facts.
• Duplicated evidence inflates the apparent weight of one source.
• Conflicting sources force the model to arbitrate without being told which source is authoritative.
• Summaries can be wrong. A compaction that drops a constraint will mislead every later call, and nothing in the context will flag it.
• Unedited tool output buries a single error line in kilobytes of log.

A larger window still earns its place for some work: reasoning across a whole contract, comparing two long documents, or a codebase question whose answer is spread across many files. The case for it is strongest when the task genuinely needs the whole input and weakest when the window is used as a substitute for retrieval. Selective retrieval and a smaller context usually win on cost, latency and attack surface together. The burden of proof belongs on the larger context, and an eval on your own tasks is where that proof should come from.

References and further reading

1. Effective context engineering for AI agents. Anthropic Applied AI team (Rajasekaran, Dixon, Ryan, Hadfield et al.), 29 Sep 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents. Vendor engineering guidance: the definition, compaction, note-taking, sub-agents and just-in-time retrieval.
2. Lost in the Middle: How Language Models Use Long Contexts. Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang. TACL; arXiv v1 6 Jul 2023. https://arxiv.org/abs/2307.03172. Research: position effects in long contexts.
3. Context Rot: How Increasing Input Tokens Impacts LLM Performance. Hong, Troynikov, Huber (Chroma), 14 Jul 2025. https://www.trychroma.com/research/context-rot. Vendor research report: reliability declines with input length; distractor effects; stated limitations.
4. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake, Abdelnabi, Mishra, Endres, Holz, Fritz. arXiv, Feb 2023 (v2 May 2023). https://arxiv.org/abs/2302.12173. Research: indirect prompt injection.
5. LLM01:2025 Prompt Injection. OWASP Gen AI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/. Industry guidance: direct vs indirect injection; RAG and fine-tuning don’t fully mitigate it; least privilege; human approval.
6. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. Wallace, Xiao, Leike, Weng, Heidecke, Beutel (OpenAI), arXiv, 19 Apr 2024. https://arxiv.org/abs/2404.13208. Research: training-based robustness to injection.
7. Model Context Protocol Specification (schema 2026-07-28), Security and Trust & Safety section. https://modelcontextprotocol.io/specification/latest. Official spec: untrusted tool descriptions; the protocol cannot enforce its security principles.
8. Prompt caching. Anthropic (Claude API docs). https://platform.claude.com/docs/en/build-with-claude/prompt-caching. Official docs: exact-match requirement; tools → system → messages order; 5-minute default TTL; 512-token minimum; TTFT improvement.
9. Pricing. Anthropic (Claude API docs). https://platform.claude.com/docs/en/about-claude/pricing. Official docs: Opus 5.5 rates used in the worked example.
10. Prompt caching. OpenAI (API docs). https://developers.openai.com/api/docs/guides/prompt-caching. Official docs: static-first ordering; full-prefix match; 1,024-token minimum; “up to 95%” discount.
11. Splitwise: Efficient generative LLM inference using phase splitting. Patel, Choukse, Zhang, Shah, Goiri, Maleki, Bianchini (Microsoft), arXiv, 30 Nov 2023. https://arxiv.org/abs/2311.18677. Research: prefill is compute-intensive and decode memory-intensive.
12. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving. Zhong et al., arXiv, 18 Jan 2024. https://arxiv.org/abs/2401.09670. Research: TTFT tied to prefill; TPOT tied to decode.

Frequently Asked Questions

What is context engineering in AI?

Context engineering is the practice of deciding exactly what information enters each model call: instructions, history, retrieved documents, tool definitions, tool outputs and summaries, along with their order and trust level. It matters most in multi-step agents, where the context is rebuilt for every call.

How is context engineering different from prompt engineering?

Prompt engineering focuses on how instructions are written. Context engineering includes that, but also manages everything else the model sees, much of which the developer never writes directly, such as search results and tool outputs.

Is RAG a form of context engineering?

RAG is one technique within it. Retrieval decides which documents might be relevant. Context engineering also decides how many of them to include, where to place them, whether to summarise them and how far to trust them.

How does context engineering reduce LLM costs?

It reduces costs by sending fewer unnecessary tokens and by ordering prompts so that stable content can be served from cache at a fraction of the input price. Savings depend on the provider’s cache rules, minimum lengths, cache lifetimes and write charges. Caching is not automatically cheaper.

How does context engineering improve AI agent security?

It limits and labels untrusted content, which reduces the chance that injected instructions steer the model. It does not prevent prompt injection on its own. Tool permissions, argument validation, sandboxing and human approval have to be enforced outside the model.

What tools or techniques are used for context engineering?

Common techniques include retrieval with reranking, compaction or summarisation, clearing tool results, structured notes stored outside the window, sub-agents with isolated contexts, cache-aware prompt ordering, and per-call token budgets. The tooling varies; the techniques are what carry over.

---

Keep reading

Here are the latest posts from the blog.