Why Your AI Chatbot Forgets: Context Windows Decoded

Ask an AI chatbot for help with a long task, and somewhere around message thirty it starts contradicting itself, re-asking for details you already gave, or quietly dropping a rule you set at the start. It is not broken, and it is not getting lazy. You have run into the single most misunderstood limit in modern AI: the context window. Understanding it is the difference between fighting your chatbot and getting clean answers on the first try.

This is a practical guide to what the context window actually is, why ChatGPT, Gemini and Claude all forget, and the simple habits that keep them sharp.

What a context window really is

Think of a large language model as a brilliant assistant with no notepad and a strict short-term memory. Everything it can "see" at one moment — your instructions, your earlier messages, its own replies, and any file you upload — has to fit inside one fixed space. That space is the context window.

It is measured not in words but in tokens. A token is a chunk of text: in English, one token is roughly four characters, or about three-quarters of a word. "Unbelievable" might be three tokens; "the" is one. When you hear a model has a 128,000-token window, that is roughly 90,000–100,000 English words — a short book.

The key point: the window is a hard ceiling on the whole conversation, not just your latest message. Every exchange you have adds to the pile.

Why your chatbot suddenly "forgets"

Once a conversation fills the window, something has to give. The model does not warn you — it simply trims the oldest text to make room for new input. The instruction you carefully wrote in your first message is often the first casualty.

There is a second, subtler problem. Researchers have repeatedly found that models read a long input unevenly: they pay most attention to the beginning and the end, and skim the middle. This is nicknamed the "lost in the middle" effect. So even when your crucial detail technically still fits inside the window, if it is buried in the middle of a giant chat, the model may behave as though it never saw it.

That explains the classic frustrations:

• You set a rule ("reply only in bullet points") early, and it fades after a while.
• You paste a long document and the AI answers using only the first and last pages.
• A long back-and-forth slowly drifts off-topic and the replies get vaguer.

Bigger windows are not a magic fix

The headline numbers have exploded. Some Gemini models advertise windows of one million tokens or more, and several Claude and GPT-class models offer 200,000 tokens and up. That sounds like the problem is solved.

It is not, for two reasons.

First, capacity is not the same as attention. A model can technically ingest a million tokens and still lose the thread of what matters inside them — the lost-in-the-middle effect gets worse, not better, as inputs balloon. Second, every token you feed in costs money and time. On paid plans and APIs, you are billed per token, so stuffing entire PDFs into the prompt for a one-line question is slow and expensive. A tight, relevant prompt frequently beats a bloated one.

The India angle: your language eats the window faster

Here is a quirk that catches Indian users off guard. Tokenizers — the part that chops text into tokens — were largely tuned on English. They handle English very efficiently. Indian-language scripts like Devanagari, Tamil or Bengali are far less efficient: the same sentence in Hindi or Tamil can use two to four times more tokens than its English translation.

The practical consequence is real. A Hindi chat, a Marathi document or a Kannada transcript fills the context window much faster than the equivalent English text, so the model starts forgetting earlier. It can also make regional-language use cost more on metered plans. If you are working with long Indian-language material and the AI keeps losing track, this hidden token tax is often why. One workaround: keep your instructions in English even when the content is in an Indian language.

Context window vs "memory": two different things

Many chatbots now advertise a memory feature, and people assume it makes the window bigger. It does not. They are separate systems doing separate jobs.

• Context window — short-term, single-conversation working space. Wiped clean when you start a new chat.
• Memory — a small, persistent store of facts the AI keeps across chats (your name, that you prefer concise answers, your job). It saves only a handful of flagged items, not whole transcripts.

So memory might remember that you are a teacher in Pune, but it will not remember the 40-message debugging session from last Tuesday. For that, the conversation has to still be inside the window — or you have to remind it.

Practical habits that keep answers sharp

You cannot change the size of the window, but you can work with it instead of against it. A few reliable tactics:

1. Start a fresh chat for a new topic. Don't let one mega-thread sprawl across unrelated tasks. A clean window means full attention on what matters now.
2. Front-load and repeat the rules. Put your most important instruction at the very top, and if a long chat starts drifting, restate it near the end — both high-attention zones.
3. Paste only what's relevant. Instead of an entire 80-page report, paste the two sections the question is actually about. Less noise, better answers, lower cost.
4. Summarise to reset. When a long chat gets sluggish, ask the model to summarise the key decisions so far, copy that summary into a new chat, and continue. You compress the useful bits and drop the dead weight.
5. Use memory deliberately. For preferences you want to persist (tone, format, your role), tell the model to remember them explicitly rather than hoping it will infer them.
6. Keep instructions in English for Indian-language tasks to save tokens, while letting the content stay in your language.

What comes next

The frontier is moving from raw window size toward smarter use of context. Expect more models that automatically retrieve only the relevant slice of a long document (a technique often called retrieval-augmented generation), better long-term memory that genuinely carries facts between sessions, and tokenizers tuned for Indian languages so the token tax shrinks.

Until then, the mental model is simple and powerful: your AI assistant is sharp but forgetful, reading a page that can only hold so much. Feed it the right things, in the right order, and keep the page uncluttered — and it will stop forgetting just when you need it most.