What is a context window? How does a chatbot "remember" a conversation, and what do you do when the conversation gets too long?
Suggested answer
The context window is all the text the model can read in a single call: the system prompt, the tool definitions, every message you send along, and even the reply it is writing. Its size is measured in tokens, and each model has its own limit.
The first thing the interviewer wants to hear is that the model's API is stateless. It remembers nothing between two calls. A chatbot "remembers" because your code keeps the array of messages and sends all of it again on every turn. If you forget to send it, the customer's second message reaches the model like the first sentence of a stranger.
The consequence is that the longer the history, the more each call costs. The Nth call resends the N−1 earlier turns, so the total number of tokens in the whole conversation grows much faster than the number of messages. With models that have a large context window, you usually run into cost and latency before you hit the limit.
When a conversation gets long, there are three common approaches, and you can combine them:
- Sliding window: send only the last few turns. You must cut at a turn boundary, because a tool result (
tool_result) has to come right after the matching tool call (tool_use). Cutting by message count easily splits a pair in two and the API rejects it. - Structured memory: your code pulls out the important facts (order ID, customer name) from tool results and puts them in the system prompt. Less information, but never wrong.
- Summarizing: ask the model to summarize the old part into a short paragraph. It remembers more, but may leave out or misstate details, so don't use it for numbers that must be exact.
Bonus points: say that you would measure the real input tokens (the usage field in the response) before optimizing, keep the full history in a database and only trim the copy that is sent, and that you know the difference between memory within one conversation and long-term memory across sessions.