Context Window

What is a Context Window?

An AI’s context window is its short-term memory; the limited amount of text (both your input and its output) that it can consider at one time when generating a response.

The context window, also called a context limit, is one of the most important constraints of a large language model (LLM). It defines the total amount of information the AI can “keep in mind” during a single conversation. This includes your initial prompt, any follow-up questions, and all of the AI’s own replies. If a conversation exceeds this limit, the AI will begin to “forget” the information from the beginning of the chat.

This “memory” is measured in units called tokens, which are roughly equivalent to pieces of words (e.g., the word “Canadian” might be two tokens: “Canad” and “ian”). Different AI models have different context window sizes, ranging from a few thousand tokens (a few pages of text) to over a million. Understanding this limitation is the key to working effectively with large documents or complex, multi-step tasks.

Think of it this way…

Think of an AI’s context window as a single sticky note. Everything you tell it, and everything it says back to you, has to fit on that one note. At first, there’s lots of space. But as the conversation continues, the note fills up. Once it’s full, to write something new, the AI has to erase the oldest information. This is why if you ask a question on Friday about a detail you discussed on Monday in the same chat, the AI might have no memory of it—it’s been erased from the sticky note.

Why It Matters for You

For a non-profit leader or a community manager, understanding the context window solves one of the most common AI frustrations: “Why did the AI ignore my instructions?” Often, it’s because the conversation has become too long and the key details have “fallen off” its sticky note. This is especially critical when you’re trying to summarize a long document, like a grant application or a transcript from a board meeting. If you paste in 20 pages of text, the AI will likely only focus on the last few pages, leading to an incomplete or inaccurate summary. By knowing this limit exists, you can learn to work within it, for example by summarizing long documents in smaller chunks, to get far more reliable results.

Example: The “Weak vs. Strong” Approach

Here’s how to handle a long interview transcript while respecting the context window.

Weak Approach (Exceeding the Limit):

Prompt: “Here is the full 60-minute interview transcript with our new member [paste 10,000 words of text]. Please write a 500-word blog post about them.”

Result:

The AI generates a blog post that only uses information from the last 15 minutes of the interview, completely missing the founder’s crucial origin story from the beginning.

(The AI didn’t ignore you; it literally forgot the beginning of the transcript.)

Strong Approach (Working Within the Limit):

Prompt: “I’ve reviewed an interview transcript. Please act as a storyteller and write a 500-word blog post using the following key information:

– Founder’s Story: Jane Doe started the business in her garage after seeing a need for local, sustainable products. – Key Quote: “We wanted to build something that our community could be proud of.” – Main Product: Hand-crafted candles made from ethically-sourced soy wax. – Call to Action: Announce their grand opening at 123 Main Street this Saturday.”

Result:

A well-structured, compelling blog post that includes all the most important details, because you pre-summarized the key points to fit within the AI’s “short-term memory.”

(You did the strategic thinking, and let the AI do the writing.)

Key Takeaways

  • The context window is the AI’s active, short-term memory.
  • It includes both your input (prompts) and the AI’s output (responses).
  • Information is forgotten once the conversation exceeds the context window limit.
  • This is the primary reason AI might “ignore” instructions in a long conversation.
  • To work with long documents, summarize them first or process them in smaller sections.

Go Deeper

Categories