# when was he born? 2023-10-16 · 4 min read · #ai · #langchain Source: https://jopcmelo.com/writing/when-was-he-born a chatbot that answers questions about your own documents is mostly three pieces: a vector database that finds the parts of your documents that matter, a model that writes the answer, and a memory of what's been said. I built one in 2023 with [LangChain](https://docs.langchain.com/oss/javascript/langchain/overview), [Pinecone](https://docs.pinecone.io/guides/get-started/overview) and [Upstash Redis](https://upstash.com/docs/redis/overall/getstarted), in a Next.js app streaming with the [AI SDK](https://ai-sdk.dev/docs). every one of those has changed its API since, so this is about the ideas, which haven't. ## search, then answer the usual setup is called [retrieval-augmented generation](https://www.pinecone.io/learn/retrieval-augmented-generation/): turn the question into an [embedding](https://developers.openai.com/api/docs/guides/embeddings), find the stored chunks of your documents with the closest embeddings, and hand those chunks to the model along with the question. the model answers from them instead of from whatever it remembers. that works for the first question. the second one is where it breaks. ## who's he? *(sketch: a chat. the user asks who wrote War and Peace, the AI says Leo Tolstoy, the user asks when was he born, with who? pointing at he, and the AI's reply is three dots, bouncing while it types)* search for "when was he born?" and you get chunks about anyone who was ever born. the search has no idea who he is. **the question only makes sense with the conversation attached**, and embeddings don't do pronouns. so before searching, a model rewrites the question into one that stands on its own, using the history: "when was he born?" becomes "when was Leo Tolstoy born?". that's the question that gets embedded and searched. LangChain's `ConversationalRetrievalQAChain`, which I used back then, did exactly this in two model calls: one to rephrase, one to answer. *(sketch: question and history go into rephrase, done by a small model, which gives when was Leo Tolstoy born?. that goes to vector search, the top matches go to answer, done by a big model, and the reply is streamed back)* ## two models the two calls don't need the same model. rephrasing a question is a small job, and nobody reads its output: it's time spent before the first word of the answer shows up. **use a small, fast model to rephrase, and save the big one for the answer.** and stream only the answer. the rephrased question is an internal step, so it shouldn't appear in the chat. ## where the history lives the memory has to come from somewhere on every request. the chat UI already has every message, so it's tempting to send them all with each question. but those messages live in the browser, and **anyone can edit what lives in the browser**: they could hand your model a conversation that never happened. keep the history on the server, keyed by session. I used Redis on Upstash: every message gets appended under the session's key, and the rephrase step reads it from there. *(sketch: two boxes of messages. the browser's is crossed out: anyone can edit this. the other is Redis, by session id: the history you trust)* ## when the answer is in the conversation one more problem showed up. when the answer was something said earlier in the chat, not something in the documents, the bot couldn't find it. the rephrase step kept only the question, so the search went looking in the documents for something that only existed in the chat. the fix went in the rephrase prompt: ask for the relevant part of the history along with the standalone question, and pass both on. ``` given the conversation and a follow-up question, return the part of the conversation that's relevant to the question, if there is one, and rephrase the follow-up as a standalone question. chat history: {chat_history} follow-up: {question} answer in this format: relevant history: question: ``` with that, the model writing the answer sees both the documents the search found and the bit of conversation that matters.