How to Build an AI Chatbot That Reasons: Hybrid RAG Explained Simply

Five words explained

• RAG (retrieval-augmented generation): before the AI answers, the system searches your documents and hands the AI the most relevant passages. The AI answers from those passages, like an open-book exam.
• Keyword search: finds passages containing the exact words you typed. Great for product codes, names and numbers.
• Meaning search (vector search): finds passages that mean the same thing even with different words. “How do I get my money back?” finds a passage titled “Refund process”.
• Re-ranking: a second, more careful pass that reorders the search results so the truly useful passages come first.
• Reasoning: the bot breaks a question into steps, gathers evidence for each, and checks its own answer before replying.

Why most company chatbots disappoint

The usual first version takes a question, runs one meaning search, and passes the top results to a model. It demos well and fails in daily use, for four reasons:

1. Meaning search misses exact terms. Ask about part “XR-220” and it may return XR-210, because the two look alike in meaning.
2. One search can’t answer a two-part question. “Which plan is cheaper for 50 users, and does it include support?” needs at least two lookups and a calculation.
3. Follow-ups lose the thread. “What about the second one?” means nothing to a search engine.
4. Nothing stops a guess. If the right passage isn’t found, the model often answers anyway, confidently.

The architecture, step by step

1. Understand the question. Use the chat history to rewrite follow-ups into full, standalone questions. “What about the second one?” becomes “Does the Business plan include phone support?”

2. Plan. For simple questions, go straight to search. For complex ones, split the question into sub-questions and decide which sources or tools each needs: documents, a database, or a calculator.

3. Retrieve with hybrid search. Run keyword search and meaning search at the same time, merge the two result lists, and re-rank them. Filter by metadata (product, region, date) and, most importantly, by who is asking: the bot must only see documents that this user is allowed to open.

4. Answer, then check. The model writes an answer using only the passages it was given and cites each one. A final check confirms that every sentence is backed by a passage. If something isn’t, the bot removes it or says it doesn’t know.

The top row turns a message into a clear plan. The shaded band is the hybrid search: two searches run side by side and are merged into one short, ranked, permission-checked list. The bottom row writes, checks and returns the answer with its sources.

How the reasoning loop works

Reasoning sounds mysterious, but in practice it is a short, controlled loop. After each search the bot asks itself: “Do I have enough evidence to answer every part of the question?” If not, it rewrites its search and tries again. After drafting, it asks: “Is every claim backed by a source?”

Two loops keep the bot honest. The top loop searches again when evidence is thin. The bottom check removes any line the sources don’t support. Capping the search loop at a few rounds keeps answers fast and costs predictable.

An architect’s note: you don’t need the most expensive “reasoning model” for every message. Most questions are simple lookups. Send those down the fast path, and switch on deeper reasoning only when the plan step finds a multi-part question. This keeps response time and cost under control.

Build it in four phases

1. Prepare the knowledge. Clean, structured documents with page numbers and section titles. If your source documents are large or messy, start there (see How to Extract Data from Large, Complex Documents).
2. Get retrieval right first. Before writing a single prompt, test whether the right passage appears in the top results for 100 real questions. If retrieval is weak, no model can fix it.
3. Add the answer and check steps. Require citations. Teach the bot that “I don’t know, here is who can help” is a good answer.
4. Add reasoning and tools last, only for the question types that need them.

Guardrails you can’t skip

• Access control in retrieval, not just in the user interface. If a document is hidden from a user, it must never reach the model for that user.
• Treat documents as data, not instructions. A document that says “ignore your rules” must not change the bot’s behaviour. This attack is called prompt injection.
• Mask personal data in logs, and keep an audit trail of what was retrieved for each answer.

How to measure it

Track four numbers on a fixed set of real questions: how often the right passage is retrieved, how often answers are fully supported by their sources, how often answers are correct, and how often the bot rightly says “I don’t know”. Re-run them after every change.

Common mistakes

• Using meaning search alone.
• Judging the bot by a demo instead of a test set.
• Letting the bot answer without citations.
• Running a slow, expensive reasoning model on every message.

How SHVX can help

Our Agent Engineering team builds conversational assistants on hybrid retrieval, with access control, citations and evaluation built in from day one.

Book an assessmentinfo@shvx.in

Learn more about Agent Engineering.

FAQ

What is hybrid RAG in simple terms?

It’s an open-book AI that searches your documents two ways, by exact words and by meaning, and combines the results before answering.

Do I need a vector database?

You need meaning search, which usually runs on a vector index. Many search engines now offer keyword and vector search together, so you may not need a separate product.

Is fine-tuning better than RAG?

For company knowledge that changes often, RAG is usually better: you update documents, not the model. Fine-tuning helps more with tone and format.

How do I stop the chatbot from making things up?

Require citations, check every claim against the retrieved passages, and reward the bot for saying “I don’t know”.