How to Use RAG to Answer Questions About Your Codebase

Most AI Tools Hallucinate About Your Code

Ask ChatGPT "what does the billing module do in our internal platform" and you will get a confident, plausible, and completely fabricated answer. The model has never seen your repo, so it does what language models always do. It pattern-matches against public code that looks similar.

Retrieval-Augmented Generation (RAG) fixes this. Instead of asking an LLM to remember your codebase from training, you retrieve the relevant chunks of your codebase at query time and pass them to the model as context. The LLM then answers based on what you actually gave it. No hallucination. No guessing. Just grounded answers.

This article walks through how to build a RAG system for your codebase, what to watch out for, and why it matters more than a chatbot bolted onto a vector DB.

What RAG Actually Does

RAG is a two-step pattern:

1. Retrieval — Given a question, find the most relevant pieces of your repo (functions, files, docs, comments).
2. Generation — Feed those pieces to an LLM along with the question, and ask it to answer using only that context.

The retrieval step is the hard part. The generation step is just a well-prompted LLM call.

The Retrieval Problem

Your codebase is not a flat pile of text. It is a graph of files, functions, modules, imports, and dependencies. Naive approaches fail in obvious ways:

• Keyword search (grep) misses semantic matches. Searching for "how do we retry failed jobs" will not find a function called scheduleCompensation even though it is exactly what you want.
• Whole-file embeddings are too noisy. A 500-line file dumped into a single embedding dilutes the signal.
• Static chunking breaks at arbitrary line counts and splits functions in half.

Good RAG systems chunk along semantic boundaries — functions, methods, classes — and embed those units independently.

A Minimal Architecture

Here is the shape of a working system:

GitHub repo
   │
   ▼
Indexer (parse AST → chunk into functions/classes)
   │
   ▼
Embedder (OpenAI / Voyage / Cohere)
   │
   ▼
Vector store (pgvector / Qdrant / Pinecone)

At query time:

User question
   │
   ▼
Embed question
   │
   ▼
Vector search → top-K chunks
   │
   ▼
Prompt: "Answer using only this context..."
   │
   ▼
LLM → answer

The whole thing is ~300 lines of Python if you skip the bells and whistles. Production systems add reranking, hybrid search, query rewriting, and conversation memory, but the core loop stays the same.

Step 1: Chunking Code Well

Chunking is where most DIY implementations fall apart. Splitting a 2000-line routes.py into 500-token windows produces chunks that are meaningless on their own.

Use AST Parsing

Parse each file into an abstract syntax tree and emit one chunk per function, method, class, or top-level constant. Include the file path, the surrounding imports, and a docstring (if present) as metadata.

For Python, tree-sitter works across most languages. For TypeScript and JavaScript, it is also solid. For Go and Rust, you are in good shape. For legacy languages, fall back to regex-based splitting by class and function declarations and accept some noise.

Include Context, Not Just Code

A function with no imports is not useful. Include the immediate context:

• The enclosing class (for methods)
• The file header or top-of-file docstring
• Relevant type definitions used in the chunk

A good chunk might look like:

# File: src/billing/invoices.py
# Imports: datetime, sqlalchemy.orm

def issue_invoice(order_id: str, customer_id: str) -> Invoice:
    """Create and persist an invoice for a fulfilled order. Triggers the
    payment capture flow if the customer has a default payment method."""
    order = Order.get(order_id)
    if order.status != "fulfilled":
        raise InvalidStateError(order.status)
    # ... rest of function

This is the unit that gets embedded. A question like "how do we handle fulfilled orders" can match this chunk directly.

Step 2: Embeddings

Embeddings turn code into vectors such that semantically similar code ends up close together in vector space. "How do we handle errors" should land near try/catch blocks and error-handling utilities.

Which Model to Use

For English-heavy repos with prose comments:

• text-embedding-3-small from OpenAI is cheap and good enough for most use cases.
• voyage-code-2 is specifically tuned for code and noticeably better for codebases with sparse comments.

For code-heavy repos with minimal comments:

• voyage-code-2 or jina-embeddings-v2-base-code outperform general-purpose models.

Test with your own queries. Embedding quality varies by language and codebase style.

Vector Store

For a small team (under ~100K chunks), pgvector inside your existing Postgres is the right starting point. No new infrastructure. Single source of truth.

For larger repos or multi-repo setups, Qdrant or Weaviate give you better filtering and hybrid search out of the box.

Step 3: Retrieval

When a user asks a question, embed the question, and pull the top-K (typically 5–10) most similar chunks from the vector store.

Hybrid Search Beats Pure Vector

Vector search misses exact matches. If someone searches for processPayment, you want the function with that name to rank first, not a semantically similar one called handleCharge.

Combine BM25 (keyword) scores and vector scores:

final_score = 0.7 * vector_score + 0.3 * bm25_score

Tune the weights per codebase. Documentation-heavy repos favor vector. Code-heavy repos with precise naming favor BM25.

Reranking Helps

After the initial top-K retrieval, pass the chunks through a reranker (Cohere's rerank-3, or a cross-encoder). Rerankers are slower but more accurate. Pull 50 chunks, rerank to top 5, send those to the LLM. Latency goes up by ~200ms; answer quality goes up significantly.

Step 4: Prompting the LLM

The prompt matters more than people think. A weak prompt:

Answer this question about the codebase: {question}. Here is the context: {chunks}

A strong prompt:

You are an engineering assistant answering questions about a private codebase.
Use ONLY the context below. If the answer is not in the context, say
"I cannot find this in the codebase" rather than guessing.

For each claim, cite the file path and line range from the context.

Context:
---
{chunk_1}
---
{chunk_2}
---
...

Question: {question}

Three things to enforce:

1. No guessing. If the answer isn't there, the model must say so.
2. Citations. Force the model to point at the evidence. You can verify.
3. Scope. The model answers about this codebase, not the world.

Model Choice

• Claude Sonnet 4 handles long contexts (200K tokens) and is reliable about citing.
• GPT-4o is comparable and slightly faster.
• Local models (Llama 3.1 70B, Qwen 2.5 Coder 32B) work for sensitive codebases that cannot leave your infrastructure. Quality is close but not quite at the frontier.

What RAG Gets Wrong (and How to Fix It)

Even with a solid setup, RAG has failure modes.

Stale Indexes

The moment you merge a PR, the vector store is wrong. Solutions:

• Re-index on every push (slow, expensive).
• Re-index on webhook from GitHub (incremental, fast).
• Re-index nightly (cheap, lags by hours).

Pick based on how quickly answers need to reflect new code.

Cross-File Reasoning

A question like "how does data flow from the API to the database" requires stitching together multiple files. Vector search finds individual chunks well but does not reason across them.

Mitigations:

• Include related chunks (callers, callees, importers) in the context.
• Use graph-aware retrieval that follows the AST.
• Increase top-K and rely on the LLM to synthesize.

Permissions

If different engineers should see different code, RAG adds a filter layer: only retrieve chunks from repos the user has access to. Easy to add, easy to skip, critical for enterprise use.

When RAG Is Not the Answer

If your repo fits in a model's context window (under ~200K tokens), you might just give it the whole thing and skip retrieval. This works for small libraries but scales badly. Retrieval is the right default for real-world codebases.

If your questions are mostly "write me a function that does X," RAG helps with style and conventions but the model still does the heavy lifting. Retrieval-augmented generation shines on questions about your code: how it works, where things live, why a decision was made.

The Practical Path

If you are building this yourself, the order of work is:

1. Start with one repo and one language.
2. Get chunking right before chasing model quality.
3. Use pgvector or an in-memory store for the prototype.
4. Add reranking once the basic loop works.
5. Wire up incremental updates via GitHub webhooks.
6. Layer in permissions before opening it to the whole org.

You will have something usable in a week. Production-grade takes longer, but the core loop is small.

---

DocPilot applies this same pattern to documentation. It parses your repo, generates accurate READMEs, architecture docs, and API references grounded in your actual code, and keeps them updated through GitHub webhooks. If you want RAG-quality answers about your codebase without building the pipeline yourself, DocPilot does the indexing, the embedding, and the synthesis so your team can ask questions in plain English and get grounded, cited answers back. The free tier covers public repos; the paid tier handles private ones with permissions and incremental sync.