Engineering

Prompt Engineering vs Context Engineering: What Works in Production

Updated 6 min read

Originally published on Medium

Prompt engineering is about how you phrase instructions. Context engineering is about controlling everything the model sees when it answers: instructions, retrieved facts, tool results, memory and chat history. In production, most failures come from the wrong or too much context, not from poor wording, so fix the context pipeline first.

Diagram showing system prompt, retrieved data and history flowing into a model context window

Key takeaways

  • A model only knows what is in its context window at the moment it answers.
  • Long system prompts and ever-growing chat history cause two failures: context confusion and context poisoning.
  • Production systems rebuild the context before every call instead of appending to it.
  • Compacting old history, using structured inputs and validating outputs make behaviour more predictable.
  • A weekend demo needs a good prompt. A product needs a context pipeline.

Prompt engineering vs context engineering

Prompt engineeringContext engineering
Question it asksHow should I phrase this?What does the model need to know right now, and what should it forget?
ScopeThe instruction textInstructions, retrieved data, tool outputs, memory and history
Fails whenWording is ambiguousContext is missing, stale, noisy or contradictory
Typical fixRewrite the promptChange what is retrieved, kept, summarised or removed

Prompting still matters. Clear instructions are part of the context. But once an application serves real users, the bigger variable is what surrounds those instructions. Anthropic's engineering team describes the same shift in effective context engineering for AI agents.

Why prompts fail at scale

Andrej Karpathy, a co-founder of OpenAI, has compared a language model to a CPU and its context window to RAM: the temporary working memory of the machine. A huge system prompt fills that memory with static text before the user types a word. As the conversation continues, user messages, tool outputs, database snippets and error traces pile in. If your system sends this growing log back on every turn, two problems appear:

  • Context confusion. The model starts weighing the long history above the instructions at the top. It forgets its boundaries because the rules are buried under tens of thousands of tokens.
  • Context poisoning. A bad data point, error message or formatting glitch enters the conversation and is treated as fact. The model repeats the mistake and compounds it.

This is the pattern behind many "it worked in testing" stories. A prompt tuned in a clean test workspace meets real, messy conversations and breaks. Adding capital letters and more rules rarely helps, because the problem is not the wording.

Technique 1: Rebuild the context for every call

Amateur builds load a giant text file on turn one and hope the model remembers it. Production systems keep state outside the model and assemble a focused context just in time.

[Base instructions]
        |
        v
[State router]
  |-- user asks about code    -> inject code context, drop billing logs
  `-- user asks about billing -> inject billing schema, drop code context

If the user asks for a financial report, the system removes unrelated support logs and injects the data structure needed for the calculation. A small, deterministic router decides what goes in. You are not asking the model to ignore noise. You remove the noise before it sees it. This links closely to how agent memory is designed.

Technique 2: Compact history and isolate memory

Passing raw history back wastes tokens and lowers accuracy. A common pattern shrinks it in three tiers:

  1. Sliding window. Keep only the last three to five turns word for word, so the conversation feels natural.
  2. Summarisation loop. Have a fast, cheap model condense older turns into a short state summary.
  3. Scratchpad. Maintain a small structured note of goals, user preferences and verified facts. Read this on each turn instead of the whole log.

Compaction is lossy, so test it. Check that decisions, constraints and names survive the summary.

Technique 3: Prefer structure to prose

Free-form paragraphs of background text invite misreadings. Convert runtime information into clean JSON or Markdown objects before it reaches the model, and validate outputs with a schema library such as Pydantic or Instructor.

{
  "current_state": {
    "user_authenticated": true,
    "active_project_id": "proj_9982"
  },
  "grounded_facts": [
    { "source": "database_row_1", "content": "Verified balance is $142.00" }
  ],
  "allowed_tools": ["fetch_invoice", "terminate_session"]
}

Structured input shifts the model from interpreting a messy conversation to executing a defined task. It also makes tool calls easier to validate.

Technique 4: Retrieve instead of stuffing

Do not paste your whole knowledge base into the prompt. Retrieve only the few passages relevant to the current question, show their source, and let the model answer from them. This is the idea behind retrieval-augmented generation, and it is also the best defence against hallucination.

A context audit checklist

  1. Print the full context sent on a failing call. Most bugs are visible there.
  2. Remove anything the model does not need for this step.
  3. Move rules that must never be broken into code, not text.
  4. Check for stale or contradictory facts in retrieved data.
  5. Validate every output against a schema, with a retry or a human handoff.
  6. Replay saved conversations after each change. See how to test AI agents.

The shift

Prompt engineering produces a great demo and a good social post. Context engineering lets an application handle thousands of real conversations without drifting. Stop asking the model to try harder. Look at the architecture of the data around it, and engineer that.

Related guides

Frequently asked questions

What is context engineering?

The practice of designing and managing everything a model sees when it generates an answer: instructions, retrieved information, tool results, memory and history.

Is prompt engineering dead?

No. Clear instructions still matter. They are just one part of a wider context pipeline.

What is context poisoning?

When an error or false statement enters the context and the model treats it as true, repeating or amplifying the mistake in later turns.

Do larger context windows fix this?

Not fully. More space lets you include more, but irrelevant or conflicting content can still distract the model and raises cost.

Where do I start?

Log the exact context sent on each call. Then remove noise, structure what remains and test with saved real conversations.

Sources and further reading

Primary documentation and research behind this guide.

Keep reading