RAG development

  • Answers sourced from your own documents
  • Chunking and embeddings done deliberately
  • Updates when your documents change

The model answers from your documents, and can show you where

Retrieval-augmented generation is the difference between a model guessing about your business and a model quoting it. Your documents are split, indexed and searched at the moment of the question, and the relevant passages are handed to the model to answer from. Most of the engineering is not in the model - it is in how you split the documents, and that is where most RAG systems are won or lost.

Talk to us about your documents

How retrieval actually works

Four steps. The first one decides the quality of everything after it.

The pipeline, in order

Chunking

Your documents are split into passages small enough to be precise and large enough to make sense alone. Split mid-table or mid-clause and the retrieved passage is useless; split too coarsely and the model gets three pages to find one sentence in. This is the step that decides whether the system is good, and the step most tutorials spend two lines on.

Embedding

Each passage is turned into a vector - a numerical position that places similar meanings near each other - so a question about "shipping to the Gulf" can find a passage that says "delivery to UAE and Saudi" without sharing a word with it.

Retrieval

The question is embedded the same way and the nearest passages are fetched. In practice this is usually hybrid: vector similarity for meaning plus keyword search for the exact product code or name that vectors handle badly.

Generation, with a refusal

The passages go to the model with an instruction to answer only from them and to say so when they do not contain the answer. That last instruction is what separates a sourced answer from a confident invention.

Where RAG systems go wrong

Four failures we see most, three of which are decided before any code is written.

What actually breaks

Chunks that cut through meaning

A fixed 500-character split through the middle of a price table, a numbered clause or a step-by-step procedure produces passages that are individually meaningless. Splitting on the document's own structure - headings, clauses, rows - is more work and is most of the difference in answer quality.

Documents that contradict each other

Two versions of a policy, one of them superseded, both indexed. The system will retrieve one with complete confidence. Deciding which source is authoritative is not a technical task and cannot be skipped.

A stale index

Prices change, policies get revised, products are discontinued - and the index still holds last quarter. A re-indexing path has to exist from the start, or the system silently becomes a record of how things used to be.

No evaluation

Without a set of real questions and known-correct answers, nobody can tell whether a change made it better. Teams end up adjusting prompts on instinct. The question set is tedious to build and it is the only thing that makes improvement measurable.

RAG and chunking: common questions

Retrieval-augmented generation. Instead of relying on what a model absorbed during training, the system searches your own documents at the moment of the question and hands the relevant passages to the model to answer from. The answer can then be traced to a source, and it updates when your documents do.

Chunking is splitting your documents into the passages that get indexed and retrieved. It matters because retrieval can only return what chunking created: a passage split through the middle of a price table or a numbered clause is useless no matter how good the model is. Splitting on the document's own structure rather than a fixed character count is most of the quality difference in a RAG system.

They solve different problems. RAG gives a model facts it can cite and that you can update by editing a document - which is what almost every business case needs. Fine-tuning changes how a model writes or behaves, not what it knows, and it goes stale the moment your information changes. For "answer from our documents", RAG is the right tool.

The retrieved passages do, on each question, under commercial API terms that exclude business data from training. If your data cannot leave your own infrastructure at all, the answer is a self-hosted model and a self-hosted vector store - slower and more expensive to run, and the right choice for some industries.

A re-indexing path built in from the start: when a document changes, its chunks are replaced rather than the whole index rebuilt. Without that the system quietly becomes a record of how your business used to work, and nobody notices until a customer is quoted last year's price.

A question set - thirty to fifty real questions with answers you know are correct - run after every change. It is tedious to assemble and it is the only thing that turns "the prompt feels better now" into a measurement. We build one as part of the project rather than after it.

Related work

What retrieval feeds, and where it runs.

Talk to us about your documents