Every RAG pipeline has the same blind spot. You chop a document into chunks, embed each one, and at query time pull back whichever chunks score highest on similarity. That's it. Nothing in the system knows chunk 47 came right after chunk 46. A top-k hit can drop you into the middle of a scene with no way to see what led up to it or what happened next.

Diagram: chunks 44 through 48 drawn as memory blocks scattered at different addresses, each holding only its own embedding with no pointer to any other block; chunk 46 is highlighted as the top vector-search hit, marked as unreachable from its true neighbors 45 and 47, and a panel below shows only chunk 46's isolated text returned to the LLM

I had a dumb-simple idea for fixing this: what if every chunk in the vector store carried prev and next pointers, like a linked list? Not swapping out the vector DB: just threading sequence through it, so a retrieved chunk doesn't come back alone. It comes back with its neighbors.

Turns out this isn't a new idea: LlamaIndex ships prev/next node relationships, there's sentence-window retrieval, auto-merging retrievers, GraphRAG. So instead of just writing about the idea, I built the simplest possible version from scratch and measured whether it actually earns its cost.

The setup

Everything here runs on-device. No external API, nothing rented: HuggingFace's all-MiniLM-L6-v2 for embeddings, FAISS for the index, and a local Ollama model (qwen2.5:7b) doing question-generation, answering, and judging.

The test document is Arthur Conan Doyle's A Scandal in Bohemia, a first-person mystery where facts get revealed in sequence and cause leads to effect a paragraph later. If neighbor-linking helps anything, it should help here.

  • Chunking: fixed 50-word chunks, 171 total, sequential IDs.
  • Index: one FAISS IndexFlatIP index, built once, shared by both strategies below.
  • Baseline retrieval: top-k=2 chunks, returned exactly as FAISS finds them.
  • Linked retrieval: the same top-2 hits, each expanded ±1 hop via its prev/next pointers, merged into one contiguous passage.

That last point matters: both strategies query the identical index and the identical embeddings. The only thing that differs is whether retrieval walks the links afterward. That isolates the one variable I actually wanted to test, instead of also mixing in "different database" as a confound.

Diagram: the same scattered memory blocks 44 through 48, now each carrying an explicit next-pointer that arcs across the diagram to wherever the following chunk actually lives; chunk 46 is still the top hit, and following its pointers reaches chunks 45 and 47, which are merged into one continuous passage returned to the LLM

To grade it, I generated 19 questions from sliding windows of the story, each one written (by the local model) to require multiple parts of its window, not answerable from one stray sentence. For every question, both strategies retrieve their context, a model answers using only that context, blind to the reference answer, and a second model call judges the answer against the reference.

What actually happened

Strategy                        Accuracy    Avg. context
Baseline (plain top-k)          37% (7/19)  98 words
Linked (prev/next expansion)    37% (7/19)  274 words

Dead tie. Linking used almost 3x the context and landed on the exact same accuracy.

If the story stopped there it'd be a clean "don't bother," except it's not the same 7 questions. Linking fixed 2 questions the baseline missed outright, and broke 2 the baseline had right. Both correct: 5. Both wrong: 10. The tie is real, but it's a tie made of trades, not a tie made of nothing happening.

Where it helped

"What is the purpose of the task given to the character, and why might Holmes have chosen a smoke-rocket?"

The baseline's top-2 chunks didn't actually contain the smoke-rocket scene, so it guessed, confidently and wrong: something about "leaving a message... at the Temple." The linked version's neighbor expansion happened to pull in the chunk baseline's raw top-k missed, and got it right: "use a smoke-rocket to create a diversion... produce a lot of smoke quickly, effectively masking their presence."

That's the failure mode this whole idea was meant to fix, actually showing up.

Where it hurt

"Where does Holmes think the photograph might be located, based on the woman's secretive nature?"

Baseline, correct: "the photograph might be with her." Linked, wrong, and not just wrong, inverted: "the photograph is likely not carried by the woman herself." The wider window pulled in an adjacent aside about the photo's cabinet size, and the model reasoned its way to the opposite conclusion.

More context isn't free. It's more surface area for a small model to get confused on, especially when a neighboring chunk offers a plausible-sounding but wrong reading.

Why I think it landed here

For this document, this chunk size, and a 7B model doing the reading: naive prev/next linking is a wash on accuracy at roughly 3x the token cost. It's not that the idea is wrong: it visibly fixed the "top-k missed the right chunk entirely" failure. It's that unconditional expansion is a blunt instrument. Every seed hit gets padded with neighbors whether it needs them or not, and sometimes those neighbors are noise instead of signal.

This is probably why the production versions of this idea (auto-merging retrievers, sentence windows) don't expand every hit unconditionally: they merge selectively, promoting to a wider span only when there's independent evidence it's warranted. The simplest version of the idea, the one I built, doesn't have that judgment. It just always says yes.

The honest caveats

  • n = 19. Each flipped answer is worth about 5 accuracy points. This is nowhere near enough data to call the tie meaningful, in either direction.
  • The judge and the answerer are the same small model, with nothing independent checking its calls.
  • One document, one genre: a first-person mystery is close to the best-case scenario for this idea. A technical manual or reference doc might behave completely differently.
  • Chunk size, k, and hop count were fixed choices (50 words, 2, 1), not swept. I'd expect the results to move if any of them did.

I still think the core intuition (a vector store that knows what came before and after a chunk should beat one that doesn't) is right. What this run actually shows is that the naive, always-expand version of it isn't automatically a win. The next version needs to be selective about when it reaches for the neighbors, not just capable of doing it.

Eco signing off