Building a RAG System: The Decisions That Actually Mattered

Part 1 of 2. Part 2 is about how I measure whether any of this works.

I've spent a good part of this year building a RAG system: an assistant that answers questions over long technical documents, for several customers sharing the same setup. It runs on a managed knowledge-base service from one of the big cloud providers, so I didn't write a vector database or a chunker myself. Most of the job was picking settings, then finding out which ones mattered.

This post is the one I wish I'd read before starting. It covers the choices that moved quality and cost, and the small gaps between what a managed service does by default and what a real product needs.

It started as a free offering. The company wanted two things from it: to see what questions customers actually ask, and to give them a reason to talk to us about the paid product. That changed what "good enough" meant. A demo can get by on a few good answers. A free tool that real customers use has to be right, and it has to be cheap enough to give away.

What RAG is for

A language model can't answer questions about a document it has never seen. You could paste the whole document into the prompt, but a 300-page manual may not fit, it's expensive to send with every question, and models get worse at finding a detail buried deep in a long context.

RAG gets around this by finding the few passages that matter and sending only those, along with the question.

At ingestion, each document is split into chunks, each chunk is turned into a vector, and the vectors go into an index built for similarity search. At query time, the question is embedded the same way, the closest chunks come back, and the model answers from those.

That's the whole idea. Everything interesting is in the settings.

The parser matters more than anything else

The parser turns a PDF into the text your chunker sees. Most managed services default to something close to plain text extraction. It's free and fast, and it wrecks tables. A table comes out as a run of numbers with the rows and columns gone, so the meaning is lost before retrieval even starts. No amount of chunking or prompting brings it back.

Somewhere between a third and half of my documents is tables. When I started measuring, table and figure questions were my weakest categories by a distance (the numbers are in Part 2).

The one that made it obvious was a simple question: how many incidents happened in 2023? The answer was sitting in a table. The default parser had flattened the table into a string of numbers, so the model had nothing usable to read, and it came back with a figure it had made up. It read fine, it was wrong, and nothing in the output hinted at either.

A document-understanding parser that recovers table structure and pulls content out of figures costs roughly a cent a page. If your documents carry information in tables, charts or diagrams, that's an easy trade.

Chunking

The usual options are fixed-size, semantic and hierarchical. Hierarchical is the one I'd start with.

          Document
         /        \
   Parent A      Parent B        (~600 tokens, sent to the model)
    /    \        /    \
 Child  Child  Child  Child      (~200 tokens, used for search)

Search runs over the small child chunks, so matches are precise. What gets sent to the model is the parent, so it has enough surrounding text to actually answer instead of matching on keywords.

The trade-off doesn't go away, though. Bigger chunks give the model more to work with but cost more and match more vaguely. Smaller chunks are cheaper and sharper but miss anything that spans paragraphs. Hierarchical chunking just lets you choose both sizes separately.

Vector store: decide by traffic, plan for hybrid search

The choice is mostly between an always-on search cluster and a pay-per-query index backed by object storage.

For a few thousand queries a month, pay-per-query is the obvious pick. The catch is hybrid search. Most pay-per-query stores don't support it, and switching later means re-ingesting everything. If you think you'll need keyword matching for part numbers, codes or exact phrases, factor that in now.

Pick the embedding model once

For English prose, the embedding model barely moved my scores. What it does affect is how painful change is. Switching models later means re-embedding every chunk you have, so choose carefully and expect to live with it.

Guardrails: start loose, tighten with real traffic

If you turn every content filter to its strictest setting on day one, you'll block normal questions. In my case a safety filter kept tripping on ordinary technical vocabulary from the domain.

Two things are worth being strict about from the start. The first is prompt injection: a denied-topic rule with example attack phrases, plus a cheap regex check in the application before anything reaches the model. The second is grounding: run grounding checks in log-only mode first, review the answers they flag, and only switch to blocking once retrieval and prompts are stable. Until then the false positives are too high.

Start with one API call, but add the orchestration layer early

My first version was a thin wrapper around the service's one-shot "retrieve and generate" call. No custom retrieval and no routing. It got a working system in days rather than weeks.

That shortcut comes with gaps you only notice once people use it:

• Session memory expires on a fixed timer. On the service I use it's an hour of inactivity, with no setting to change it. For a tool someone keeps open all day, losing the conversation after lunch is a real problem.
• Every message goes through retrieval, including "hello" and "thanks". Each one embeds the text, searches the index and calls the model. At test volume you don't notice the cost. At real traffic you're paying for small talk.
• Follow-ups are searched as-is. "What about the second option?" gets embedded on its own, with no idea what the previous question was about. Retrieval finds nothing useful, and the assistant seems to lose the thread. All three are fixed by an orchestration layer between your app and the knowledge base. Most platforms offer an agent abstraction for this. It rewrites follow-ups into standalone questions before searching, decides whether a message needs retrieval at all, keeps longer sessions, can call other APIs, streams answers, and gives you a trace of each turn.

The agent layer is a heavier thing to set up and version. Even so, if I started again I'd add it from the beginning. The one-call version was quick, but I've spent more time working around it than it saved me.

The gotchas are in the defaults

Most of my surprises had nothing to do with the model. They came from defaults in the managed service, which are documented, just easy to skim past.

One ingestion job at a time

The service only lets one ingestion job run per data source. If two documents arrive close together, the second start call fails with a conflict, and unless you handle that, the document is never indexed.

What works is a deferred retry. On conflict, mark the document as waiting with no job attached. A small worker checks every few minutes for waiting documents and starts their ingestion once the source is free. Anything stuck too long raises an alert. Build this before you have concurrent uploads, not after the first missing document.

Tenant isolation belongs in one function

Separate knowledge bases per customer don't scale, because most services cap how many you can have. So everyone shares one, and isolation is a metadata filter on every query:

filter = {"equals": {"key": "org_id", "value": current_org_id}}

The risk is where current_org_id comes from. It has to come from the authenticated identity on the server, never from the request. One endpoint that takes an org_id from the request body without checking it, and one customer can retrieve another's documents. From the outside, that looks exactly like a normal answer.

The fix is structural. Build the filter in one function that only accepts the verified identity object, and don't let any other code construct it. Relying on everyone remembering to check isn't enough.

Deleting a file doesn't delete its vectors right away

On the service I use, removing a document from storage doesn't remove its vectors until the next sync. In between, the document is gone from your UI but still retrievable by the model. Someone could delete a file and still get its content back in an answer.

Treat delete as two steps: remove the file, then remove its vectors. If your product can't live with the gap, trigger the vector deletion yourself rather than waiting for the sync.

Session filters can't change mid-conversation

If a session starts scoped to documents A, B and C, the next turn can't switch to a different set. The API rejects it. That rules out a UI where people add and remove documents mid-chat.

You have to choose: keep the conversation and fix the scope, or allow scope changes and start a fresh session each time. Either is fine, as long as you decide before you design the UI.

Keep anything that can change out of the vector store

My first instinct was to store document titles as vector metadata, so citations could show the title straight from the search results. The first rename would have broken it, because metadata can't be updated in place and re-ingesting is slow.

Now the vector metadata only holds IDs that never change, like document_id and org_id. Titles and everything else that can change live in the application database and get joined in at query time. A rename is one SQL update.

Log everything from day one

Every query, response, token count, guardrail trigger and set of retrieved chunks goes into the logs. It costs almost nothing, and it's useful in ways I didn't plan for. It seeds the evaluation set with real questions, it makes support tickets answerable, it makes cost per customer a single query, and it lets you notice when one customer's token use suddenly triples.

Where the money goes

Across providers the shape is the same. Generation at query time is most of the bill, mostly input tokens. Ingestion is close to free. A premium parser at a cent a page adds up at large volumes, but it's small next to query costs.

The levers, roughly in order:

1. How many chunks you retrieve (top-k). Cutting it is often a 30-40% saving for a small, measurable drop in quality.
2. Parent chunk size. Doubling it roughly doubles cost per query.
3. Model routing. Sending simple turns to a cheaper model saves a lot on those turns.
4. Parser. Real money at volume, still smaller than the above. The bigger point is that a fixed top-k is the wrong default. A simple factual question needs a couple of chunks. A question that pulls together several sections needs more, and probably a reranker. Choosing k per question type gets you both the savings and the quality.

If I started again

1. Use a managed knowledge base before writing your own pipeline. It gets you most of the way in a fraction of the time.
2. Spend your time on the parser before the chunker.
3. Build an evaluation set before the second feature.
4. Add the orchestration layer early.
5. Keep the parser, chunker, embedder and store behind interfaces you can swap. You will swap at least one of them. None of this is exotic. It's mostly reading the API reference carefully and writing the small pieces of glue that a managed service leaves to you. The part I'd single out is measurement, because without it every decision above is a guess. That's Part 2.