Tokens and Context Windows Explained: Why AI Forgets and What It Costs

What is a token?#

A token is the smallest unit of text a language model works with. Depending on the model, a token can be a whole word, part of a word, a single character or even a byte. Before a model reads your prompt, the text is split into tokens, and the model writes its reply one token at a time.

Anthropic's glossary explains that the number of characters a token represents depends on the model and the language. That is why the same paragraph can count as a different number of tokens in two tools, and why no simple rule converts words to tokens exactly. Providers offer token counting tools for exact figures.

Why parts of words?#

Splitting text into pieces involves a trade-off. Anthropic's glossary notes that larger tokens make processing more efficient, while smaller tokens let a model handle uncommon or never-before-seen words. A 2015 research paper on arXiv showed one influential approach: encoding rare and unknown words as sequences of smaller "subword units", including a method based on byte pair encoding. A rare product name or a Singlish expression may therefore be broken into several pieces rather than treated as one.

What tokens mean for you#

You rarely see tokens in a chat tool, but they shape three things:

• How much fits. Context windows are measured in tokens, not pages.
• Speed. Longer prompts and replies mean more tokens to process.
• Cost. Providers can measure and bill usage in tokens, as explained below.

What is a context window?#

The context window is all the text a language model can refer to when it writes a response, including the response itself. Anthropic describes it as a working memory, separate from the large body of training data the model learned from. If something is not in the context window, the model cannot use it in that answer.

Context windows vary by model. Anthropic's documentation, for example, lists context windows of up to one million tokens depending on the model. A larger window lets you include longer material, but the same page notes that more context is not automatically better.

What counts towards the window?#

Everything in the request counts, not just the message you typed. Anthropic's documentation lists the following.

This is why a chat that started small can become very large: each turn carries all the previous turns with it.

Why does AI seem to forget?#

AI tools seem to forget because every message and reply adds tokens to the context window, and long conversations crowd out or blur earlier details. Some chat tools quietly drop the oldest parts. Even when everything fits, accuracy and recall can degrade as the token count grows.

There are three separate causes:

1. The window is full. Anthropic notes that chat interfaces can manage the context window on a rolling "first in, first out" basis, so the oldest messages fall out. When a request sent directly to the model is too long, the system can return a "prompt is too long" error instead.
2. Too much to keep track of. Anthropic describes a drop in accuracy and recall as the token count grows, which it calls context rot. Curating what goes into the window matters as much as its size.
3. Lost in the middle. A 2023 research paper found that language models often performed best when the relevant information was at the beginning or end of the input, and significantly worse when it sat in the middle of a long context, even for models designed for long inputs.

Forgetting and hallucination often appear together. When the detail the model needs has been pushed out or buried, it may fill the gap with something plausible. Our guide to AI hallucination covers how to spot and reduce that.

What do tokens cost?#

Tokens affect cost because providers such as Anthropic measure and bill API usage in tokens: the tokens you send in and the tokens the model writes back. Long chat histories, large pasted documents and long replies all add up. Exact charges vary, so check your provider's current terms.

Anthropic's documentation describes input and output tokens being billed, and offers a token counting tool so developers can count tokens before sending a request and manage rate limits and costs. Its prompt engineering guide also notes that cost and speed can sometimes be improved more easily by choosing a different model than by rewriting a prompt.

For everyday users on a fixed subscription, tokens still matter. Anthropic's glossary lists the complexity of the prompt and the generated response among the factors that affect response time, and the context window limits how much the tool can hold in mind. Small language models (SLMs), covered in our guide on when a smaller model is better, are one way organisations manage cost for simple, high-volume tasks.

Habits that save tokens and improve answers#

• Paste only the sections of a document that matter for the question.
• Start a new chat for a new task instead of continuing an old one.
• Ask for the length you need: "five bullet points" rather than "tell me about".
• Ask for a short summary of decisions so far, then carry it into a fresh chat.

What might this look like in a Singapore workplace?#

Imagine an administration team at a small precision engineering firm in Woodlands that uses one long chat to review a lengthy supplier terms document. After many exchanges, the tool starts ignoring the formatting rules set at the start and misses a clause from the middle of the document. Tokens and context windows explain both problems.

This is an illustrative scenario. A better approach might be:

• Split the job. Review the document section by section, in separate chats, instead of all at once.
• Restate the rules. Put the key instructions, such as "British English, a table with clause number and summary", at the start of each new chat.
• Put the question near the material. Paste the relevant section, then ask the question straight after it.
• Ask for quotes. Anthropic's guidance on reducing hallucinations suggests asking for word-for-word quotes first when working with long documents, then basing the answer on them.
• Check the result. Someone compares the summary table against the document before it is used. Supplier terms may be confidential, so only an approved tool should be used.

What are the common mistakes?#

The common mistakes are running every task in one endless chat, pasting whole documents when only a section matters, assuming a bigger window means better answers, and burying key instructions in the middle. Each wastes tokens and makes it more likely that the model misses something important.

• One chat for everything. Old, unrelated material keeps using space and attention.
• Pasting everything. Large inputs cost more and make the important part harder to find.
• Trusting a big window. More room does not guarantee better recall, as the research above shows.
• Instructions in the middle. Put key instructions and the question at the start or end.
• Ignoring signs of drift. If the tool starts breaking your rules, start a fresh chat with a summary.
• Guessing token counts. Use the provider's counting tools if exact numbers matter.

What to learn next#

Once you understand tokens and context windows, you can write prompts that fit the model's working memory, avoid paying for wasted tokens and spot when a long chat is going off track. The guides below build on this, from the basics of language models to practical prompting habits for everyday office work.

• Revisit the foundations in what an LLM is.
• Browse more guides for everyday users in AI for users.
• See where this fits in the AI learning pathway.
• To manage context in an app you build yourself, the GenAI Builder Sprint is a 2-day classroom course where adults plan, build, test and launch a simple AI app. No previous AI or programming experience is needed, although formal entry requirements apply.

Sources#

1. Glossary, Anthropic (Claude Platform Docs), accessed 8 November 2026
2. Neural Machine Translation of Rare Words with Subword Units, arXiv, accessed 8 November 2026
3. Context windows, Anthropic (Claude Platform Docs), accessed 8 November 2026
4. Lost in the Middle: How Language Models Use Long Contexts, arXiv, accessed 8 November 2026
5. Token counting, Anthropic (Claude Platform Docs), accessed 8 November 2026
6. Prompt engineering overview, Anthropic (Claude Platform Docs), accessed 8 November 2026
7. Reduce hallucinations, Anthropic (Claude Platform Docs), accessed 8 November 2026

Frequently asked questions#

What is a token in AI?+

A token is the smallest unit of text a language model works with. Depending on the model, a token can be a whole word, part of a word, a single character or even a byte. How many characters a token represents depends on the model and the language.

What is a context window?+

The context window is all the text a language model can refer to when it writes a response, including the response itself. Anthropic describes it as the model's working memory, which is separate from the large body of data the model was trained on.

Why does an AI chatbot forget what I said earlier?+

Every message and reply in a conversation adds tokens to the context window. Some chat interfaces drop the oldest parts on a first in, first out basis as the window fills. Even within the limit, accuracy and recall can degrade as the token count grows.

Is a bigger context window always better?+

No. A larger window lets you include longer material, but Anthropic notes that more context is not automatically better, and research has found that models often miss information placed in the middle of long inputs. What you put in matters as much as how much fits.

What counts towards the context window?+

Everything in the request: instructions such as the system prompt, every message in the conversation, pasted documents, images, tool results and tool definitions, plus the output the model generates in reply.

How do tokens affect cost?+

Providers such as Anthropic measure and bill API usage in tokens, counting both input and output. Anthropic's token counting tool helps developers manage rate limits and costs. Longer prompts, long chat histories and long replies all use more tokens.

How can I stop an AI tool losing track in a long task?+

Start a fresh chat for each new task, restate key instructions, paste only the parts of a document that matter, put the most important material at the start or end, and ask for a short summary you can carry into a new conversation.