How to Add AI Chat to Your Product Documentation in an Afternoon

Most product documentation is a graveyard of search queries that returned nothing. Users type "how do I rotate my API key" and your Algolia index shrugs. An AI chat layer over your docs turns those zero-result searches into answered questions, and you can ship one in an afternoon if you stop overthinking the architecture.

This walkthrough assumes you already have a documentation site (Markdown, Docusaurus, Nextra, VitePress, whatever) and an OpenAI-compatible API key. Everything else is plumbing.

What You're Actually Building

You're not building a chatbot. You're building a retrieval-augmented generation (RAG) endpoint that takes a user's question, finds the 3-5 most relevant chunks of your docs, and asks an LLM to answer using only those chunks. The complexity lives in three places: chunking, retrieval, and the system prompt.

Skip the temptation to fine-tune a model. Retrieval is faster, cheaper, and you can update it by editing a Markdown file.

Step 1: Get Your Docs Into a Shape the Model Can Read

If your docs live in a GitHub repo (they do), you're already most of the way there. The job is to split each page into chunks small enough to fit in a context window but large enough to be self-contained.

A good rule of thumb: 200-500 tokens per chunk, with a 50-token overlap between consecutive chunks so you don't cut a sentence in half at a boundary. Don't chunk by character count or word count; chunk by structure. Split on H2 headings first, then on paragraphs if a section is still too long.

Here's a minimal Python chunker that respects headings:

import re
from typing import List

def chunk_markdown(text: str, max_tokens: int = 400) -> List[str]:
    sections = re.split(r'(?m)^## ', text)
    chunks = []
    for section in sections:
        if not section.strip():
            continue
        # Re-add the heading if we split one off
        if not section.startswith('#'):
            section = '## ' + section
        # Split long sections on paragraph breaks
        paragraphs = section.split('\n\n')
        buffer = ''
        for para in paragraphs:
            if len(buffer) + len(para) > max_tokens * 4:
                chunks.append(buffer.strip())
                buffer = para + '\n\n'
            else:
                buffer += para + '\n\n'
        if buffer.strip():
            chunks.append(buffer.strip())
    return chunks

Step 2: Embed and Index

Once you have chunks, embed them. OpenAI's text-embedding-3-small is the default choice; it's cheap, fast, and good enough for documentation retrieval. For most products under 5,000 pages of docs, you don't need a vector database. A SQLite table holding embeddings as BLOBs plus a cosine similarity scan at query time is plenty.

import sqlite3
import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model='text-embedding-3-small',
        input=texts
    )
    return [e.embedding for e in response.data]

def build_index(chunks: list[str], db_path: str = 'docs.db'):
    conn = sqlite3.connect(db_path)
    conn.execute('CREATE TABLE IF NOT EXISTS chunks (id INTEGER PRIMARY KEY, text TEXT, embedding BLOB)')
    embeddings = embed(chunks)
    for chunk, emb in zip(chunks, embeddings):
        conn.execute('INSERT INTO chunks (text, embedding) VALUES (?, ?)',
                     (chunk, np.array(emb, dtype=np.float32).tobytes()))
    conn.commit()
    conn.close()

When you outgrow SQLite, migrate to pgvector or a hosted option like Turbopuffer. The migration is mostly moving bytes around; the logic stays the same.

Step 3: Write a Retrieval Function

At query time, embed the user's question, score every chunk against it, and take the top five. Cosine similarity is fine; don't bother with cross-encoders for a docs chatbot.

def retrieve(query: str, db_path: str = 'docs.db', top_k: int = 5) -> list[str]:
    conn = sqlite3.connect(db_path)
    rows = conn.execute('SELECT text, embedding FROM chunks').fetchall()
    conn.close()

    query_emb = np.array(embed([query])[0], dtype=np.float32)
    query_emb /= np.linalg.norm(query_emb) + 1e-10

    scored = []
    for text, emb_bytes in rows:
        emb = np.frombuffer(emb_bytes, dtype=np.float32)
        emb /= np.linalg.norm(emb) + 1e-10
        score = float(query_emb @ emb)
        scored.append((score, text))

    scored.sort(reverse=True)
    return [text for _, text in scored[:top_k]]

If your docs are large enough that this scan takes more than 200ms, switch to an approximate nearest neighbors index (HNSW or IVF). For under a few thousand chunks, the brute force is faster to build and easier to debug.

Step 4: The System Prompt Is the Product

Most teams blow the launch by treating the system prompt as an afterthought. It's the single most important file in your chatbot. Spend an hour here.

The prompt needs three sections:

1. Role and scope: Tell the model what it is and what it's allowed to do. Example: You are a support assistant for AcmeDB. Answer only using the documentation provided. If the docs don't contain the answer, say so and point the user to support@acme.dev.
2. Format rules: Force short paragraphs, code fences with the correct language tag, and never invent API endpoints. If you have a style guide for your docs, codify it.
3. Citations: Require the model to cite the chunk it pulled each fact from. You can render these as superscript numbers or footers; the point is the user can verify the answer.

A working prompt skeleton:

You are the AcmeDB documentation assistant. Use only the context below.
If the answer is not in the context, reply: "I don't have that in the docs —
email support@acme.dev."

Format answers in Markdown. Wrap code in triple backticks with the
correct language. Cite sources like [^1] using the provided [n] markers.

Context:
[1] {{chunk_1}}
[2] {{chunk_2}}
[3] {{chunk_3}}
[4] {{chunk_4}}
[5] {{chunk_5}}

Step 5: The API Endpoint

Wrap retrieval and generation in a single POST endpoint. Streaming is worth the extra 30 minutes — users perceive answer bots as broken when they wait 4 seconds for a wall of text.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()

class AskRequest(BaseModel):
    question: str

@app.post('/ask')
def ask(req: AskRequest):
    chunks = retrieve(req.question)
    context = '\n\n'.join(f'[{i+1}] {c}' for i, c in enumerate(chunks))
    prompt = SYSTEM_PROMPT.replace('{{context}}', context)

    def stream():
        response = client.chat.completions.create(
            model='gpt-4o-mini',
            messages=[
                {'role': 'system', 'content': SYSTEM_PROMPT},
                {'role': 'user', 'content': f'Context:\n{context}\n\nQuestion: {req.question}'}
            ],
            stream=True
        )
        for chunk in response:
            delta = chunk.choices[0].delta.content
            if delta:
                yield delta

    return StreamingResponse(stream(), media_type='text/plain')

Step 6: The Frontend Widget

The widget is a floating button bottom-right that opens a panel. You can build this in 100 lines of React, but don't. Use a library like Vercel's ai-sdk UI components or an open-source option like CopilotKit. Both give you streaming, message history, and a chat-bubble UI out of the box.

import { useChat } from 'ai/react';

export default function DocsChat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat({
    api: '/api/ask',
  });

  return (
    <div className="fixed bottom-4 right-4 w-96 h-[500px] bg-white shadow-xl rounded-lg flex flex-col">
      <div className="flex-1 overflow-y-auto p-4">
        {messages.map(m => (
          <div key={m.id} className={m.role === 'user' ? 'text-right' : 'text-left'}>
            <div className="inline-block p-2 rounded bg-slate-100 my-1">
              {m.content}
            </div>
          </div>
        ))}
      </div>
      <form onSubmit={handleSubmit} className="border-t p-2 flex">
        <input
          value={input}
          onChange={handleInputChange}
          className="flex-1 border rounded px-2 py-1"
          placeholder="Ask the docs…"
        />
        <button type="submit" className="ml-2 px-3 py-1 bg-blue-600 text-white rounded">
          Send
        </button>
      </form>
    </div>
  );
}

Drop the component into your docs layout. Done.

Step 7: Keep It in Sync

The trap with docs chatbots is staleness. The day after you ship, someone edits a page and the bot keeps answering with the old text. Two ways to handle this:

1. Webhook on push: Add a GitHub Action that runs build_index.py on every push to main and writes to your DB. Takes 2 minutes to set up.
2. Lazy rebuild: On every page render, check if the page's mtime is newer than the indexed version; if so, re-embed and update. Simpler, slightly slower at query time.

Pick the webhook. Your CI already runs; piggyback on it.

What to Skip on Day One

You will be tempted to add features. Resist them. Specifically, do not build:

• Conversation memory across sessions — useful, but privacy is a tar pit. Add it in v2.
• Multi-language support — translates the docs, not the answers. Worth a quarter of work, not an afternoon.
• Fine-tuned embeddings — your retrieval quality is bounded by chunk quality and prompt quality, not by the embedding model.
• A Slack integration — your support team will ask. Say next sprint.

Cost Reality Check

A docs chatbot on gpt-4o-mini with text-embedding-3-small costs roughly $0.10 per 1,000 questions at this scale. For a mid-sized SaaS with 50,000 doc interactions a month, you're looking at $5/month in API costs plus the vector storage. The expensive part of the project is your engineering time, not the bill from OpenAI.

The Afternoon Is Done

You now have a streaming, retrieval-grounded AI chat sitting on top of your docs. It cites its sources. It updates on every push. It cost you one afternoon and a few dollars in API credits.

The rest is iteration: read your chat logs weekly, find the questions the bot answered badly, fix the chunks or the prompt, redeploy. Quality compounds.

If you'd rather skip the chunker, the retrieval code, and the system prompt gymnastics, DocPilot generates the entire backend, including embeddings, a streaming endpoint, and a chat widget, from your existing GitHub repo in about five minutes. Connect your repo, pick which folders to index, and you get the same architecture you just built, plus the webhook-driven rebuild and a hosted widget you can paste into any docs site. Try DocPilot for free and have the chat live by tomorrow morning.