How to Give an LLM Access to Internal Data Without Dumping Everything in a Vector Store

The standard advice is three steps long. Embed the documents, retrieve the top matches, put them in the prompt. It is good advice, it works, and a great many useful systems are built on exactly that.

If you are asking the question anyway, it is probably because you have seen where it stops, and cannot yet name what you are looking at. This article is mostly about naming it.

How do you give an LLM access to data?

Mechanically, three routes, and most systems use more than one.

Retrieval over an index. Embed content, search by similarity at question time, pass the matches into the prompt. Cheap to build, good at fuzzy recall over prose, and the default for good reason.

Direct query against systems of record. Give the model a tool that runs a query and returns rows. More precise than similarity search, bounded by what the query can express, and dependent on the model choosing the right tool and the right parameters.

Resolution against governed definitions. Before retrieving anything, resolve the terms in the question against definitions the organization has agreed, then retrieve within that. This is the one most systems skip, and skipping it is what produces the ceiling this article is about.

The three are not alternatives. A capable system uses similarity for recall, queries for precision, and governed resolution to decide what the words in the question actually refer to.

Where the default approach runs out

A vector store gives a model access without authority.

That sentence is the whole diagnosis, and it has three consequences worth separating, because they fail in different ways.

It can find text that mentions a thing. It cannot tell which mention the company stands behind. Search for the definition of an active customer across a decade of internal material and there are nine answers, written at different times by people with different purposes, all genuine, several contradictory. Similarity returns whichever is phrased most like the question. It has no notion that the others exist, and no basis for preferring one.

It cannot resolve two records to one entity. The same customer appears under three identifiers in three systems. To a retrieval index these are three unrelated strings that happen to look somewhat alike. Any question requiring completeness (everything about this customer, all exposure to this supplier) returns a confident partial answer, and nothing in the output indicates it is partial.

It cannot tell that the person asking is not cleared to see the answer. Embeddings do not carry permissions. If access control lives in the application and retrieval happens underneath it, eventually there is a path that returns governed content to the wrong caller, and it surfaces in an audit rather than in testing.

A memorable illustration of the first one appeared on Hacker News under the title your RAG still thinks the old CFO is the CFO. Nothing malfunctioned. Retrieval found a real sentence in a real document that had been accurate when written, and the system had no way to know it had been superseded.

What vectors are genuinely good at

Worth stating plainly, because the argument here is not that vector search is the wrong technology. Vector search is excellent at what it does.

It handles paraphrase and synonym without anyone enumerating them. It degrades gracefully when the question is phrased unlike anything in the corpus. It is cheap to stand up and cheap to run. Over a large body of unstructured prose with no reliable entity structure (support history, policy documents, meeting notes, research), it extracts value that a modelled approach would take months to match, if it ever recouped the modelling cost at all.

The honest line is this: a vector store is a retrieval mechanism, not a decision about what is true. The unease in the original question comes from having been handed the first as an answer to the second.

It is also worth checking whether the default is simply sufficient, because for a lot of systems it is. If the questions are answered by a passage that exists somewhere, if nobody would dispute the terms in them, and if every user is cleared to see everything in the corpus, then embedding the documents and retrieving the top matches is not a compromise. It is the correct architecture, and adding a governance layer to it would be effort spent on a problem the system does not have. The ceiling described above is real, and it is not universal. Establishing which side of it you are on is the first useful thing to do, and it is cheaper than either build.

What the alternative actually looks like

Not replacing the index. Adding the layer that decides what the retrieved material means.

Decide what has to be authoritative, and leave the rest inferred. Most content can stay exactly where it is, indexed the way it is. The set that needs an agreed answer is the terms and rules where two competent people would answer differently and both could defend it, and in most organizations that list is tens of items, not thousands.

Resolve entities before retrieving about them. Decide which records are one customer, one product, one legal parent. This is the step that converts a confident partial answer into a complete one, and it is a decision rather than a similarity computation.

Let retrieval resolve against governed definitions rather than instead of them. The order matters. When a question contains a governed term, resolve the term first, then search within that frame. Retrieval still does the finding; it is no longer also doing the deciding.

Attach access policy to the context. Enforce at the point the context resolves, so the same question from two callers can correctly return two different answers, and correctness never depends on the application layer remembering to check.

That is the shape. The detailed method (inventory, decide, model, wire, validate) is a separate article, and once you are at the point of building rather than diagnosing, grounding an LLM in enterprise data is the piece to go to. It picks up precisely where this one stops.

What to check before building either

Three questions, and they take an afternoon rather than a quarter.

Would two people here answer this differently? Pick the ten questions the system most needs to get right and ask two people each. Every disagreement is a term that needs deciding before any retrieval architecture will help.

Does the same thing appear under several identifiers? If yes, entity resolution is on the critical path and no amount of embedding quality substitutes for it.

Who is allowed to see each answer? If that varies by caller, access has to resolve with the context rather than around it, and retrofitting it later is considerably more expensive than designing it in.

For the discipline this all sits inside, see context management. For the comparison between retrieval approaches in more depth, knowledge graph versus vector database, and for the retrieval pattern that traverses structure rather than matching text, GraphRAG.