Two claims about long context are both wrong, and the truth sits between them.
One camp says the giant context window solved memory… just pour everything into the prompt and let the model sort it out. The other says long context is a marketing number, that models can’t really read past the first few pages. Neither holds up.
Here’s what the benchmarks actually show. Ask a frontier model to find one fact buried in a huge document… a single needle in the haystack… and it does that remarkably well. Right? That part is real. But ask it to find several facts scattered across that same haystack and relate them to each other, and the floor gives way. In one multi-needle test, a top model asked to pull ten facts out of about 25,000 tokens returned four. Add a reasoning step on top of retrieval and it gets worse, not better. The RULER benchmark found the “effective” context is often only half to two-thirds of the advertised number, because multi-hop and aggregation tasks break long before the window fills.
So cognitive load didn’t disappear when the windows got big. We moved it. This is the pattern I keep seeing in agentic work… we rediscover a classic engineering constraint wearing new clothes. You can’t design around a limit you’re pretending isn’t there.
Monday morning: open one agent you’ve built and look at what you’re actually stuffing into the prompt. A big context window is not permission to fill it. Cut what goes in down to what genuinely bears on the question, and you’ll get better answers from less. The people doing this work every day will all tell you the same thing… context management is king.
Source: Multi Needle in a Haystack (LangChain)
Originally posted on LinkedIn.
RationalAI helps organizations turn ideas like this into results: AI strategy and advisory services. Previous article: Only a Machine Catches a Machine
