Context
A model knows only its training and the text in front of it. That text is the context.
You know that the model sees only what is in its context. You can write a request that gives it what it needs.
You can explain what the model sees for one answer, why the window has a size, and how to give good context in a request.
You can describe the context window as the model's only working input, its cost and limits, and how Codex fills it on demand.
Layer two is context. Context is everything the model can see when it answers.
The second layer is context. It is the text the model sees for one answer.
Layer two: the input sequence. Everything the model uses beyond its weights has to be in it.
What the model can see
The model knows two things. It knows what it learned in training. It also knows the text in front of it right now. That text is the context.
The context holds the rules from the product, your message, the earlier messages in the thread and any files Codex has read. It also holds the results of tools.
If a fact is not in the context, the model does not see it. It may guess instead.
Context is everything the model can see for one answer. It includes the product's own instructions, your message, earlier turns in the thread, the files Codex has read and the results of tool calls.
The model does not remember last week's chat, and it cannot see a file it has not read. If a fact is not in the context and not in its training, the model has to guess.
This is why a request that names the file and the goal gets a better answer than a vague one.
The context is the token sequence passed to the model for a single generation. It is assembled by the harness from product instructions, your message, earlier turns, file content that the model has read and tool results. The weights supply general knowledge up to the knowledge cutoff, and nothing else.
Self-attention lets every token in the window condition on every other token. This is why a fact in the context can change an answer, and why nothing outside the window can.
The model keeps no state between calls. A harness that appears to remember a thread sends the stored thread back in as input each time.
The window has a size
The context window is the most text the model can see at once. It is counted in tokens.
A long thread fills the window. Then the software must shorten the older parts. It may write a summary of them, or drop them.
So start a new thread for a new task. A fresh thread gives the model a clear page.
The context window is the maximum size of the context, counted in tokens. A long thread, or a large file, uses up part of that space.
When a thread outgrows the window, the harness summarises or cuts the older parts. This is compaction. Details from early in a long thread can be lost.
Start a new thread for each new task. Put the key facts in your request again instead of relying on an old thread.
The window is a hard limit on input length, and the cost of attention grows with it. A long context also costs more time and allowance on every call, because the whole window is processed again.
Position matters as well as size. "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023) reports that models use information at the start and end of a long input better than information in the middle.
Harnesses manage the limit with compaction, which replaces older turns with a summary, and with retrieval, which adds only the passages a task needs. Codex reads files on demand through tools. It does not load a whole folder into the window.
Good context gives good answers
You can help the model by putting the right facts in your request. Name the file. Say who will read the answer. Say what shape you want. Say what limits apply.
The example below does all four.
Good context has four parts. Name the file or source. Name the reader. State the format. Set the limits, such as length and what to leave out.
A request with these four parts is easier to answer and easier to check. The example below shows them for a summary of an RFQ, which is a request for quotation.
Specific context narrows the space of plausible outputs. A named source tells the harness which file to read. A named reader and format fix the register and the structure. A limit such as "no facts outside the file" gives you a rule to verify against.
Context you do not supply is filled from training data, which is how a fluent but unsupported claim enters an answer. The example below closes that gap for a summary of an RFQ, a request for quotation.
| Source | Example | Added by |
|---|---|---|
| Product instructions | Rules that tell the model how to behave in Codex | The product |
| Your message | Summarise the RFQ for a finance manager | You |
| Earlier turns | Your last request and the answer to it | The harness |
| Files read | The text of procurement/rfq-laptops.md | A tool call |
| Tool results | The list of files from a folder listing | A tool call |
Summarise procurement/rfq-laptops.md for a finance manager. Use five bullet points. Include the closing date and the weights. Use no facts from outside the file. Do not change any files.
This request names the file, the reader, the format and the limits.
Check the result against the file. The closing date and the weights should match the RFQ exactly.
The last line is a verifiable constraint. Compare each bullet with the file, and treat any bullet you cannot find there as a model addition.
Key terms
- Context
Everything the model can see when it writes one answer.
Everything the model can see for one answer.
The input token sequence for one generation: instructions, history, file content and tool results.
- Context window
The most text the model can see at one time.
The maximum size of the context.
The maximum length, in tokens, of the input the model accepts for one call.
- Prompt
The message you send.
The message you send.
The message you send. In the harness it joins the rest of the context as one part of the input.
- Compaction
Making older parts of a thread shorter so that the thread fits the window.
Summarising older parts of a thread to fit the window.
Replacing older turns with a summary to stay inside the window. Detail can be lost in the summary.
Check yourself
A fact is not in the context and not in the model's training. What happens?
The model does not have the fact. It may guess, so you should check the answer.
The model has to guess. Put the fact in the request, or point to the file that holds it.
The model fills the gap from learned patterns, so the answer can be fluent and wrong. Supply the source in the context and verify against it.
Why start a new thread for a new task?
A long thread fills the window. Older parts get shortened or dropped.
A long thread can outgrow the window, and compaction can lose early detail. A new thread starts with a clean context.
A long context costs more and can lose early detail to compaction or to weak use of the middle of the window. A new thread resets the input.
Which four things does good context name?
The file, the reader, the format and the limits.
The file or source, the reader, the format and the limits.
The source, the reader, the format and the limits. Together they narrow the output and give you something to verify.
OtterBlock is not affiliated with OpenAI. Codex and ChatGPT are OpenAI products.