Why RAG Remained So Primitive

The “First Textbook” Trap Seen Before in the History of SOM

Retrieval-Augmented Generation, or RAG, has become widely known as a way to connect generative AI with internal corporate documents.

The commonly presented explanation of RAG is surprisingly simple:

Divide documents into short chunks.
Convert each chunk into an embedding vector.
Convert the user’s question into a vector as well.
Retrieve the chunks closest to the question.
Pass those chunks to an LLM and generate an answer.

LangChain’s introductory materials similarly present documents, text splitters, embeddings, vector stores, and retrievers as the basic components of semantic search. Major surveys of RAG research classify this structure—splitting documents into chunks, storing their embeddings in a vector database, and retrieving the most similar chunks—as Naive RAG.

The problem is not that this method is taught as an introductory model.

The problem is that an introductory model has come to be treated as though it were the completed form of RAG.

The Unnaturalness of Directly Comparing Questions with Answer Text

A user’s question and the answer-bearing text in a document usually have different linguistic forms.

Suppose a contract contains the following provision:

Unless written notice is provided at least 30 days before the expiration date, the agreement will renew automatically.

A user may ask:

How far in advance do I need to notify you if I want to stop the automatic renewal?

These two sentences concern the same subject, but their vocabulary, syntax, and communicative functions differ. One is a declarative rule. The other expresses a user’s intent in question form.

Naive RAG directly compares these different kinds of sentences in an embedding space and assumes that their meanings will be sufficiently close.

Modern embedding models can handle this to a certain extent. However, as the number of documents grows and more chunks contain similar language, the correct chunk may fall outside the top few retrieved results. Surveys of RAG have repeatedly identified problems such as irrelevant retrieval, failure to retrieve necessary evidence, and answers that are not properly grounded in the retrieved material.

In response, practitioners repeatedly adjust:

chunk size,
chunk overlap,
the number of retrieved results,
hybrid keyword search,
reranking,
query rewriting,
and prompts.

These techniques can improve performance, but they do not change the underlying structure. They continue to refine a system that directly compares a question with fragments of answer text.

Indexing “Answerable Questions” Instead of Documents

A more natural approach is to generate, in advance:

What kinds of questions can this passage answer?

From the contract provision above, a system could generate question patterns such as:

How far in advance must I give notice to terminate the contract?
How can I prevent the agreement from renewing automatically?
What is the notification deadline if I do not want to renew?
Will the contract renew automatically if I do nothing?

A user’s actual question is likely to resemble these generated question patterns more closely than it resembles the original legal provision.

The retrieval structure can therefore be redesigned.

Conventional RAG

User question
    ↓
Direct comparison with document chunks
    ↓
Original source text

Question-pattern routing

User question
    ↓
Comparison with generated question patterns
    ↓
Identification of the relevant Wiki page
    ↓
Retrieval of the original source text
    ↓
Grounded answer generation

The generated question patterns should not become the final source of truth.

They are routing devices that lead the system to the correct original evidence. The final answer should still be generated from the source document.

This distinction matters. Even if some detail is omitted when a Wiki page or question pattern is generated, the final system can return to the original passage before answering.

Question-Based Retrieval Is Not an Exotic Idea

The idea of generating synthetic questions from documents and using them for retrieval has already appeared in research.

Studies on enterprise RAG have proposed decomposing document chunks into atomic statements, generating synthetic questions from those statements, and matching user questions against the generated questions. In reported experiments, retrieval based on synthetic questions achieved better recall than directly retrieving the original document chunks.

Other work has likewise proposed generating questions that a document chunk could answer and indexing those questions rather than relying only on the chunk’s original wording.

The important conclusion is not that vector search is fundamentally wrong.

The real issue is:

What should be vectorized, and what should be compared?

It may be more rational to construct an “answerable question space” from a source document and route the user’s question through that space than to treat raw document chunks as the only retrieval units.

The Same Thing Happened with SOM

This situation closely resembles the history of the Self-Organizing Map, or SOM.

In many textbooks, SOM is presented as an online learning algorithm. Data records are shown one at a time, a winning node is selected, and the winner and its neighbors are gradually updated.

This explanation is intuitive and useful for understanding the basic principle of SOM. However, online SOM is affected by:

the order in which data is presented,
initial values,
the learning rate,
and the number of learning iterations.

Batch SOM, by contrast, processes the complete dataset in batch form. It does not require a conventional learning-rate schedule and can produce stable, reproducible models when its conditions are fixed.

Teuvo Kohonen himself later recommended the batch-learning version for practical use, describing it as faster and safer than sequential learning.

Viscovery SOMine also used high-performance batch SOM as its core technology for many years, together with two-dimensional hexagonal maps, variable weighting, missing-value handling, and clustering.

Nevertheless, when many people hear “SOM,” they still think primarily of the sequential algorithm described in introductory textbooks.

This reveals a recurring problem in the diffusion of technology:

The first version that is explained clearly becomes fixed in the public imagination as the standard form of the technology.

Even when more practical methods are later developed, it may take years for them to reach textbooks, training courses, sample code, and ordinary users.

Primitive Methods Are Often Easier to Popularize

Naive RAG did not become widespread because it was necessarily the most advanced approach.

It became widespread because it was easy to explain and easy to demonstrate.

The following story is immediately understandable:

Split a PDF, put the chunks into a vector database, retrieve the most similar passages, and send them to an LLM.

This structure also maps neatly onto commercial products:

embedding APIs,
vector databases,
RAG frameworks,
and cloud search services.

It is convenient for developers because they can build a demo quickly. It is convenient for vendors because the components are easy to sell. It is convenient for corporate buyers because they can explain internally that they are adopting a widely recognized RAG architecture.

The same was true of sequential SOM. The narrative of presenting an input, selecting a winner, and updating neighboring nodes was intuitive and closely resembled the familiar story of neural-network learning.

But ease of explanation and practical superiority are not the same thing.

Endless Tuning Can Make an Elementary Design Look Advanced

When Naive RAG fails to answer a question, developers begin adjusting:

how the document is divided,
how much overlap is used,
how many results are retrieved,
which similarity threshold is applied,
how reranking is performed,
and how the prompt is written.

Because these adjustments can become highly complex, RAG begins to look like an advanced technical discipline.

But there is another interpretation:

Much of the tuning may be required because the system began with an overly primitive retrieval design.

The system is trying to compensate for the mismatch between question language and document language through downstream parameter adjustments.

This resembles repeatedly changing the learning rate, initialization, and presentation order of an online SOM and then concluding that SOM is inherently unstable.

Batch SOM simplifies the problem by changing the learning procedure itself.

Similarly, redesigning what RAG retrieves may eliminate much of the tuning that otherwise appears unavoidable.

An LLM Wiki Is a Routing Layer, Not Merely a Knowledge Repository

When people hear that an LLM automatically creates Wiki pages from documents, they may imagine that the source text is merely being summarized into another knowledge base.

That is not the most important function.

Each Wiki page can contain:

the subject covered by the page,
references to the original source,
and generated question patterns representing what the source can answer.

The Wiki page then functions as a semantic router between the user’s question and the original evidence.

If a separate LLM Wiki is created for each source document, several additional problems become easier to manage:

Each document can be updated independently.
Each document can be tested independently.
A problematic document can be disabled without rebuilding everything.
Permissions can be assigned at the document level.
Only the Wiki sets available to a particular user need to be virtually integrated.

There is no need to place every corporate document into one enormous vector database and ask the LLM to decide afterward whether the retrieved information should be disclosed.

The system should first determine which Wiki sets the authenticated user is permitted to access. Routing and retrieval should then occur only within that authorized knowledge space.

This Is Not an Argument Against Vector Search

The point is not that embedding-based search should be abandoned.

Embeddings can still be used to compare a user’s question with generated question patterns.

The difference lies in what is being compared:

Do we directly compare the question with raw document chunks?
Or do we first generate the questions that the document can answer and search within that question space?

Vector search is a tool. It is not a complete knowledge architecture.

Yet the introduction of a vector database is often treated as equivalent to the construction of a knowledge base.

This confusion lies at the center of much of the current enthusiasm around RAG.

The Problem Is Not Simply a Lack of Intelligence

It would be too simple to say that developers continue using primitive RAG because they do not understand the technology.

The deeper problem is structural:

The first tutorial establishes the conceptual framework.
Frameworks turn that framework into reusable code.
Vendors sell products based on the same architecture.
Buyers prefer methods that are already widely recognized.
Educational institutions teach the methods most commonly used in the market.

Once this cycle is established, the standard architecture becomes difficult to challenge, even when more rational alternatives are available.

The slow diffusion of batch SOM and the continued dominance of raw-chunk vector retrieval can be understood as examples of the same phenomenon.

What Enterprises Need Is Knowledge Design, Not Merely Better Retrieval

Companies do not truly need systems that find chunks resembling a user’s query.

They need systems capable of managing questions such as:

What can each document answer?
Which knowledge domain should receive a particular question?
Which original passage should serve as evidence?
Which users are authorized to access which sources?
What must be regenerated when a document changes?
How should unanswered questions be incorporated into the knowledge system?

The central challenge of RAG is therefore not the speed or sophistication of a vector database.

It is the design problem of transforming documents written for humans into structures through which AI can reliably answer questions.

Questioning the Basic Form Presented in Textbooks

When a new technology appears, the first simplified explanation often survives for a long time.

In SOM, sequential learning came to be treated as though it were SOM itself.

In RAG, chunking documents and retrieving them by vector similarity has come to be treated as though it were RAG itself.

But a basic principle is a starting point, not a final form.

Just as SOM evolved in practical applications toward batch learning, variable weighting, missing-data handling, clustering, and stable two-dimensional visualization, RAG must move beyond raw-document retrieval toward: