Anatomy of a Production RAG System

Ask an LLM about your own data and it will happily improvise. Retrieval-Augmented Generation (RAG) is the fix: before the model answers, the system retrieves the most relevant records from your own store and instructs the model to answer only from them. The model stops improvising and starts citing.

To show what that looks like in practice, we built fit-copilot, a fitness Q&A assistant. A user asks something like "which exercises target the triceps without equipment", the question is embedded, matched against an exercise dataset in Postgres, and Gemini answers strictly from the matched exercises. Every answer is judged for relevance, priced per token, and logged to a dashboard.

This post walks the full anatomy: the retrieval layer, the grounded generation step, the two evaluation loops that put numbers on quality, and the monitoring that keeps it honest in production. The complete code is open source at github.com/karancodes95/fit-copilot if you want to follow along.

The Architecture

Six pieces, each replaceable on its own:

• Dataset: 70 exercises generated once with Gemini structured output against a Pydantic schema, then frozen as the source of truth.
• Retrieval: every exercise embedded with gemini-embedding-001 (3072 dimensions) into Postgres + pgvector; queries ranked by cosine distance.
• Generation: Gemini 2.5 Flash answers from a grounded prompt that allows only the retrieved context.
• API: FastAPI with Pydantic validation at every boundary.
• UI: a Streamlit chat client with thumbs up/down feedback.
• Monitoring: every conversation logged to Postgres with tokens, cost, an inline relevance verdict and user feedback, visualised in Grafana.

The Dataset

Every RAG system starts with a corpus, and ours is deliberately small: 70 exercises, each with a name, activity type, equipment, body part, movement type, the muscle groups it activates and step-by-step instructions. We generated it once with Gemini using structured output: a Pydantic schema defines exactly what one exercise looks like, and the model must return JSON that validates against it, so there is no fragile parsing of free text.

The generated CSV is then frozen and committed as the single source of truth. A sanity pass confirmed 70 rows with zero duplicates and zero nulls before anything downstream consumed it. The database never becomes precious: ingestion drops and rebuilds the table from the CSV in one atomic transaction, so the whole retrieval layer is reproducible from a file you can read in any spreadsheet.

The lesson transfers directly to client work: whether the corpus is product docs, a menu or a policy manual, freeze a validated snapshot, keep it in version control and treat the vector store as disposable.

The Retrieval Layer

Retrieval is a meaning-matching problem: "something like running in place" shares no words with "Mountain Climber", yet it must find it. Embeddings solve that. Each exercise's fields are concatenated into one labelled text block and turned into a 3072-dimension vector with gemini-embedding-001; texts with similar meaning land near each other in that space. The one iron rule: documents and queries must be embedded by the same model, or they live in different spaces and nothing matches.

We store those vectors in Postgres with the pgvector extension rather than a dedicated vector database: at this scale a query is an exact scan in milliseconds, everything (vectors, conversations, feedback) lives in one transactional database, and there is no extra infrastructure to run. A dedicated vector store earns its cost at millions of vectors, not seventy.

Choosing hosted embeddings over a local model was a deliberate tradeoff. A local sentence-transformers setup would pull in torch (a roughly 106 MB wheel) and a model download, while the Gemini endpoint costs a fraction of a cent at this scale and adds zero dependencies. The reverse call is right when data cannot leave your infrastructure; the point is to make the choice consciously.

Putting a number on it

Most RAG demos stop at "it seems to answer well". We wanted numbers. For every exercise we had Gemini 2.5 Pro generate five realistic user questions that deliberately avoid naming the exercise, giving us 350 labelled test cases where the correct result is known in advance.

Running all 350 through retrieval at K=5 gave a Hit Rate of 0.9486 (the right exercise appears in the top five results 95% of the time) and an MRR of 0.7967 (it usually ranks first or second). The few misses were semantically close neighbours, a Triceps Dip ranking above a Push-up, which told us the remaining headroom is in ordering, not in finding.

Grounded Generation

Generation is where RAG earns its name. The top retrieved exercises are rendered into the prompt as clean labelled blocks (name, equipment, muscle groups, instructions), and the instruction to the model is blunt: answer the question using only the facts from the context. That one line is the anti-hallucination guardrail; the model's job shrinks from "know fitness" to "read these five entries and answer from them".

Gemini 2.5 Flash writes the answer. Alongside the text, the code captures response time and the full token breakdown from the usage metadata: prompt tokens, completion tokens and thinking tokens. That last one matters; 2.5 Flash reasons before answering by default, and those hidden thinking tokens bill at the output rate. Ignore them and every cost estimate you make is quietly wrong.

The failure path is handled too: if the model returns an empty or blocked response, the code raises a clear error instead of letting a null answer flow into the database and surface as a confusing crash three layers later.

The API and the UI

The backend is FastAPI with three endpoints: ask a question, leave feedback, health check. One Gemini client is created at startup and shared across requests, and the question handler runs in a threadpool so blocking LLM calls never freeze the server. Pydantic validates every boundary: blank or oversized questions are rejected with a 422 before any money is spent, and feedback is constrained to exactly +1 or -1 at the API layer and again by a CHECK constraint in the database.

Failure handling has a deliberate priority: if the answer was generated but the monitoring write fails, the user still gets their answer and the miss is logged. Burning two paid LLM calls and then showing an error because a log insert failed is the wrong trade.

The frontend is a small Streamlit chat client. It renders your question immediately, shows a spinner while the pipeline runs, then prints the answer with its relevance verdict and estimated cost underneath, plus thumbs up and down that accept one vote per answer. It holds no logic of its own; it just talks to the API, which is exactly what makes the API swappable under any other frontend later.

Monitoring: Every Answer Leaves a Paper Trail

Two Postgres tables carry the whole story. Every answered question becomes a row in conversations: the question, the answer, which model ran, response time, the full token breakdown including thinking tokens, a computed dollar cost and an inline relevance verdict from an LLM judge. Thumbs from the UI land in a feedback table keyed to the conversation id, so quality signals and usage data live next to the vectors they came from.

Cost is computed per call from published per-token pricing, with thinking tokens billed at the output rate. In live use an answer costs around $0.002 to $0.0025, and because every row carries its own price tag, "what did this feature cost this month" is a SQL query, not a guess.

A Grafana dashboard reads those tables directly: questions per day, total spend, response times, the relevance breakdown, token usage over time and net feedback. When quality drifts or cost spikes, it shows up here first.

Judging the Answers, Not Just the Retrieval

Perfect retrieval can still produce a bad answer, so a second evaluation layer scores the output itself. An LLM judge reads each question and generated answer and returns a structured, schema-validated verdict: RELEVANT, PARTLY_RELEVANT or NON_RELEVANT, with a one-line reason. On a 50-question sample the system scored 98% RELEVANT, 2% PARTLY_RELEVANT and zero NON_RELEVANT.

Two honest caveats. The generator and the judge are the same model family, and models grade their own family generously, so we read 98% as a ceiling rather than gospel. And a judge that runs only in offline evals goes stale, which is why the same judge also scores every live answer inline; the number on the dashboard is the same metric as the number in the eval report.

What We'd Tell You About RAG

• Pick the vector store your app actually needs. Small corpus: your existing database with a vector extension is fine. Millions of vectors: get a dedicated store.
• Measure before you tune. Build a ground-truth question set first, so every change is judged by numbers, not feelings.
• Ground it, then check it stayed grounded. Tell the model to answer only from context, and have a judge verify it did.
• Count the hidden tokens. Thinking models bill their reasoning as output; skip it and your cost math is wrong.
• Fail in the user's favour. Validate input before spending money, and never drop a paid answer because a log write failed.
• Rebuild your index atomically. An interrupted ingestion should leave the old data live, never an empty table.

See It Yourself

The whole project is open source and reproducible end to end: github.com/karancodes95/fit-copilot. If your business wants an assistant grounded in your own data, with numbers instead of vibes, book a free discovery call.