How We Cut Docs Generation Costs by 80% Using Local Embeddings

The Bill That Woke Us Up

Our monthly embedding invoice hit $4,200 and we hadn't even onboarded our first 100 repositories. Each documentation sync against a mid-sized codebase was running between 200,000 and 800,000 tokens through a hosted embedding API, and the unit economics made the path forward look bleak. At our projected growth, we'd be paying more for embeddings than for compute.

We had two options: pass costs to users or rebuild the embedding pipeline. We chose the latter. Over six weeks, we migrated from a cloud-hosted embedding model to locally hosted alternatives and reduced our per-repository embedding cost by roughly 80%, while keeping latency within a few hundred milliseconds of the hosted version. Here's exactly how we did it, what worked, what didn't, and the benchmarks that justified the move.

Why Embeddings Are the Hidden Tax on Documentation Tools

Before diving in, it's worth understanding why embeddings dominate the cost structure of any retrieval-augmented documentation system.

The Token Multiplier Problem

Every chunk of source code, every README paragraph, and every architectural decision captured during indexing gets converted into a vector. A single 50,000-line TypeScript repository can produce 15,000 to 25,000 chunks depending on your splitter. Multiply that across users syncing daily, and you're not running an embedding job, you're running an embedding factory.

Cloud providers charge per token, and tokens in code are dense. A function signature with generics, JSDoc, and a body can easily consume 300-500 tokens. At $0.0001 per 1K tokens, that looks cheap—until you do it 100,000 times per day.

Why We Couldn't Just Pass Costs Through

We briefly considered per-document pricing. The problem: documentation is meant to be regenerated frequently. Webhooks fire on every push to main. A typical active customer triggers 8-15 syncs per week. Usage-based pricing for embeddings punishes the exact behavior we want to encourage: keeping docs fresh.

The only sustainable answer was to drive the unit cost toward zero.

Evaluating Local Embedding Models

We tested four candidates across three dimensions: retrieval quality (measured by recall@10 on a labeled eval set), throughput on our target hardware, and operational complexity.

The Candidate Shortlist

Model	Dimensions	Size	Notes
all-MiniLM-L6-v2	384	80 MB	Fast baseline, lower recall
bge-small-en-v1.5	384	130 MB	Stronger semantic capture
e5-base-v2	768	440 MB	Best quality/size tradeoff
bge-large-en-v1.5	1024	1.3 GB	Highest recall, heaviest

We use bge-base-en-v1.5 (768 dimensions) as our default. The 384-dim options saved memory but our retrieval eval showed a consistent 6-9 point drop in recall@10 on code-adjacent queries—queries like "where is auth handled" or "explain the retry logic." Code vocabulary is too specialized for the smallest models.

Hardware Reality Check

We run on a mix of A10G and L4 instances. A single L4 handles roughly 180 documents per second through bge-base-en-v1.5 with batching, which is comfortable headroom for our workload. The 1.3 GB large model only fit on the A10G and delivered diminishing returns for the 3x memory footprint.

The Architecture: Batch, Cache, and Quantize

Migrating to a local model was the easy part. The cost reduction came from three engineering decisions that compounded.

1. Aggressive Caching with Content Hashing

The single biggest win. Roughly 70% of chunks across syncs are unchanged—function signatures that haven't moved, docstrings that haven't been edited. We hash each chunk's content with sha256 and check the vector store before re-embedding:

import hashlib
from typing import Optional
import numpy as np

class EmbeddingCache:
    def __init__(self, store):
        self.store = store  # Redis or similar KV store

    def _hash(self, text: str) -> str:
        return hashlib.sha256(text.encode("utf-8")).hexdigest()

    def get_or_embed(self, text: str, embed_fn) -> np.ndarray:
        key = self._hash(text)
        cached = self.store.get(f"emb:{key}")
        if cached is not None:
            return np.frombuffer(cached, dtype=np.float16)
        vec = embed_fn(text)
        self.store.set(f"emb:{key}", vec.astype(np.float16).tobytes())
        return vec

Casting to float16 cuts storage in half with negligible retrieval impact at 768d.

2. Dynamic Batching

The hosted API effectively forces sequential or small-batch requests over HTTP. Locally, we can saturate the GPU. We batch chunks into groups of 32-64 with a 50ms timeout:

class Batcher:
    def __init__(self, model, max_batch=64, timeout_ms=50):
        self.model = model
        self.max_batch = max_batch
        self.timeout_ms = timeout_ms
        self.queue = []

    async def embed(self, text: str) -> np.ndarray:
        future = asyncio.Future()
        self.queue.append((text, future))
        if len(self.queue) >= self.max_batch:
            await self._flush()
        return await future

    async def _flush(self):
        if not self.queue:
            return
        texts = [t for t, _ in self.queue]
        vectors = self.model.encode(texts, batch_size=len(texts))
        for (_, future), vec in zip(self.queue, vectors):
            future.set_result(vec)
        self.queue.clear()

Batching alone gave us a 5-6x throughput improvement versus one-at-a-time inference.

3. INT8 Quantization

We quantized bge-base-en-v1.5 using ONNX Runtime's static quantization. Model size dropped from 440 MB to 120 MB, inference latency fell by ~35%, and recall@10 degraded by less than 1 point on our eval set. For code documentation retrieval, the precision loss is invisible.

The Cost Math

Here's the actual breakdown on a per-repository basis for a typical sync of 500,000 tokens.

Approach	Cost per Sync	Notes
Hosted API (before)	$0.050	$0.0001 / 1K tokens
Local, uncached	$0.012	GPU amortization only
Local, with cache	$0.003	70% cache hit rate
Local, cached + quantized	$0.002	Reduced GPU minutes

The "local with cache" row is the steady-state number. Our measured 80% reduction is the delta between row one and row three. The GPU time amortizes across many concurrent syncs, so the marginal cost per repository approaches the cost of the hash check and the storage write.

What We Got Wrong

The migration wasn't smooth. Two calls in particular cost us a week.

First, we underestimated cold-start latency. A quantized ONNX model loads in ~3 seconds, but loading plus tokenizer warmup pushed our first-sync latency from 800ms to 4-5 seconds. We solved this with a daemon-style worker pool that keeps models resident. In Kubernetes terms, we run a dedicated embedding Deployment with a fixed replica count and a queue between the API tier and the workers.

Second, we initially used the same embedding model for code and prose. Code chunks have structural patterns—braces, semicolons, type annotations—that the model treats as noise. We now run code chunks through a lightweight preprocessing pass that joins identifiers with underscores and strips comments, which lifted recall@10 on code-specific queries by 4 points. Prose goes through unchanged.

Benchmarking We Trust

Vendor benchmarks are useless for this decision. We built a 2,000-query eval set from real DocPilot customer queries, hand-labeled with the correct chunk ID. Three things matter:

• Recall@10 stayed within 1.5 points of the hosted model
• p95 latency for a single embedding call: 47ms (hosted was 180ms)
• Throughput: 180 docs/sec on a single L4, versus ~15 docs/sec against the hosted API given our rate limits

We didn't beat the hosted model on quality—we weren't trying to. We matched it within a tolerance and won decisively on cost and latency.

The Operational Checklist

If you're considering the same move, here's what we'd validate before committing:

• Build a labeled retrieval eval set of at least 500 queries from real user traffic
• Measure cache hit rates on a sample of repeated syncs to confirm your chunking is stable
• Quantize and benchmark on your actual hardware, not your laptop
• Plan for cold-start with a warm worker pool
• Keep the hosted API as a fallback for the first week in production

Where DocPilot Fits

DocPilot runs this exact pipeline today. When you connect a GitHub repository, the platform spins up an embedding worker, caches aggressively against your prior sync state, and only re-embeds what actually changed. The cost savings we described here are what let us keep a generous free tier and avoid charging per-sync. If you're maintaining documentation for an active codebase, the technical question isn't whether embeddings are expensive—it's whether the tool you're using is doing the work to keep them cheap.

If you want to see the pipeline in action, connect a repository to DocPilot and watch what gets re-embedded on your second sync. The cache hit rate is usually the moment it clicks.