Documentation for AI Products: A Complete Handbook

Most AI product docs fail because they document the SDK, not the model. This handbook covers what to actually write — and how to keep it honest.

DocPilot Team

AI documentation research and engineering from Onzira Labs.

5 September 2026 12 min read

Why AI Product Documentation Is a Different Discipline

Your team ships a new feature: a vector search endpoint that returns ranked chunks from a customer knowledge base. The backend is clean, latency is under 200 ms, and the eval suite shows 11% better recall than the previous BM25 baseline. Everyone is happy. Then someone asks: "What do we put in the docs?"

The honest answer is: nothing you have written so far is sufficient. The README covers the HTTP request and the response schema. But it does not cover the model behind the endpoint, the prompt template it uses, the failure modes you have observed in evaluation, the data the customer needs to know about, or the cost characteristics of a query that fans out to a reranker and an embedding model. For most software, that level of detail would be overkill. For AI products, it is table stakes.

AI products have properties traditional software does not: non-deterministic outputs, prompt sensitivity, training data lineage, evals as first-class artifacts, model versioning, and a much wider blast radius when behavior drifts. Documentation is the only durable mechanism you have to communicate these properties to developers integrating your product, to reviewers approving it, and to your own team six months from now when nobody remembers why the temperature is 0.2 instead of 0.0.

This handbook is the structure I wish someone had handed me the first time I shipped a model behind an API. It is opinionated, written for engineering leads and developer-founders, and assumes you already know what an embedding is and how RAG works. We will move from principles to specific doc types, then close with the operational workflow that keeps everything in sync.

The Four Audiences You Are Actually Writing For

Before you write a word, name the audiences. Each one reads a different layer of your docs.

The Integrator

This is a backend engineer at a customer company who has three days to evaluate your product. They want to know what it does, what it costs, what the latency profile looks like, and whether they can swap it for a competitor without rewriting half their stack. The integrator docs are the README, the quickstart, the API reference, and the changelog.

The Evaluator

This is a technical decision-maker, often a staff engineer or an ML lead. They want to see your evals, your model card, your known limitations, and your safety posture. Evaluators are the audience that decides whether your product makes it into a vendor review document. They will read things the integrator will skip.

The Operator

This is your own team, plus the customer's MLOps or platform team. They want runbooks, observability docs, rate limits, cost projections, and version migration guides. Operator docs are the most neglected category in early-stage AI products, and the most expensive to skip.

The Reviewer

This is anyone who needs to justify the use of your product inside a regulated environment: a security team, a procurement officer, a compliance lead. They need a model card, a data processing addendum summary, and clear statements about retention, PII handling, and subprocessor usage.

If you try to write one document that serves all four, you will write something that serves none of them. The trick is to write four overlapping documents with cross-links, not one omnibus.

The Doc Stack for an AI Product

A mature AI product has roughly seven documentation surfaces. You will not ship all of them on day one, but you should know they exist and have a plan.

README and Quickstart

The README is the integrator's first impression. For AI products it has to do more than the standard template. In addition to installation and a 30-line usage example, the README needs a "What is this?" paragraph that names the model class, the task it solves, and the input/output contract in plain language. It needs a cost paragraph that gives a concrete example: "$0.0008 per 1k input tokens at list price, so a 2k-token summarization costs about $0.0024." It needs a limitations paragraph with at least three honest entries. A README that says "works great for everything" is a red flag to evaluators.

The quickstart should be a complete, copy-pasteable path from zero to a working integration in under fifteen minutes. Use a real provider (OpenAI, Anthropic, or your own endpoint), a real SDK, and a real example payload.

API Reference

For an AI product the API reference must include, for every endpoint, the model identifier used, the prompt template version (if exposed), the maximum context length, the supported modalities, the streaming behavior, and the error codes that are specifically tied to model behavior (context too long, safety filter triggered, rate limit on the upstream model, etc.). A bare REST reference copied from the SDK is not enough.

Model Card

The model card is borrowed from the research community and is now industry standard. It covers the model's intended use, the training data at a high level, the evaluation results across a defined benchmark set, the ethical considerations, and the known failure modes. Mitchell et al. introduced model cards in 2019 and they have aged well. If you are fine-tuning a base model, your model card should explicitly call out what changed and what did not — the gap between "the base model and our fine-tune" is the most common place where customer expectations break.

For a thin wrapper around a frontier model, you still need a model card. The card documents your system prompt, your retrieval pipeline, your reranker, and any guardrails. Call it a "system card" if you prefer; the principle is the same.

Evaluation Documentation

Most AI products under-document evals. You should publish, at minimum:

The benchmark suite you run, including the version of each benchmark dataset
The metrics you track, with definitions
The eval prompts and the eval harness code, or a stable pointer to them
The latest results, with the date and the model version
A "regression" section listing any metrics that got worse and the explanation

A simple markdown table is usually enough. The point is not pretty charts; the point is reproducibility and accountability.

Metric	Baseline model	Current model	Delta
Recall@10 on internal eval	0.71	0.79	+0.08
Hallucination rate (human eval)	4.2%	3.1%	-1.1pp
P95 latency (ms)	240	285	+45
Cost per 1k queries	$0.18	$0.22	+$0.04

If a customer asks "how do you know this works?" your eval docs are the answer.

Prompt and Configuration Reference

If your product exposes prompt templates, temperature, top-p, system messages, or tool definitions, document each one. Even if the customer never changes them, they need to know what the defaults are and what happens when they deviate. This is also where you document prompt caching behavior, structured output modes, and any reasoning effort parameters.

A common mistake is to document the configuration but not the constraints. If you say "max temperature 2.0" but do not mention that values above 1.0 frequently produce malformed JSON, the customer will find out the hard way.

Runbooks and Operator Docs

Operator docs cover the things that go wrong in production. For an AI product this includes:

How to roll back to a previous model version
How to interpret a drift in eval scores run on production traffic
How to handle a safety filter regression
How to investigate a spike in cost (usually: a customer found a way to trigger long-context requests, or a prompt injection attack is fanning out)
On-call contacts for upstream model providers

These are internal-first but they shape customer-facing incident communications. A runbook that does not exist will produce a customer-facing incident communication that lies.

Compliance and Trust Documents

This is the layer most early-stage AI products skip entirely. It includes a data processing summary, a subprocessor list (which upstream model providers you call), retention defaults, PII handling, and the SOC 2 or ISO posture if you have one. You do not need a 40-page DPA to start; a one-page trust portal with clear bullets is enough at seed stage and signals seriousness to enterprise evaluators.

The Six Principles That Hold the Stack Together

Principles are easy to write and easy to ignore. These six are concrete enough that you can audit them.

1. Document the System, Not the Code

The model, the prompt, the retrieval index, the reranker, the guardrails, and the cache form a system. The README should describe that system, not the HTTP handler. If a developer cannot answer "what does this endpoint actually do?" from your docs, you have failed.

2. Pin Everything Versioned

Every numeric claim in your docs needs a version pointer. "Our model returns 0.79 recall on the internal eval" is meaningless without "on eval set v3, model build 2025-03-14." Versioning is also what lets you rewrite the model without rewriting the docs.

3. Show Limits Before Strengths

Evaluators notice. Integrators forgive. Put limitations near the top of the README and at the top of the model card. "Not recommended for use cases involving medical triage" is a sentence that builds more trust than a feature list.

4. Make Evals a First-Class Artifact

Evals should live in the same repo as the docs, with their own version, their own changelog, and a clear owner. The eval set is part of the product, not a research side quest.

5. Separate the Stable From the Volatile

The HTTP contract should be stable for years. The model behavior will drift as you retrain, re-prompt, or swap upstream providers. Document them in different places with different churn rates.

6. Write for the Customer Six Months From Now

You will forget why the reranker threshold is 0.62. The customer will never know. Write it down. Add a "Why this value?" line under every non-obvious default.

A Worked Example: Documenting a RAG Endpoint

Let's walk through one example end-to-end. The product is POST /v1/search, which takes a query string and returns ranked chunks from a customer-uploaded corpus.

The system is: a query embedding model (text-embedding-3-large, pinned), an ANN index (Pinecone serverless, namespace per customer), a BM25 hybrid index, a cross-encoder reranker, and a top-k=10 final list.

README excerpt

## What this is
A hybrid retrieval endpoint that combines dense embeddings and lexical
matching, then reranks with a cross-encoder. Returns up to 10 chunks.

## Cost
$0.0009 per query at list price. A typical customer runs ~120k queries/day,
so about $108/day or ~$3,200/month.

## Limitations
- English-only. Multilingual queries return lower-quality results.
- Recency is not modeled. The index snapshot is at most 15 minutes stale.
- Tables and figures in source documents are not extracted; only text.

Model card excerpt

## Components
- Embedding: text-embedding-3-large (provider: OpenAI, pinned 2025-02-01)
- Reranker: cross-encoder/ms-marco-MiniLM-L-12-v2 (self-hosted)
- Hybrid weights: 0.7 dense / 0.3 lexical (see "Why" below)

## Why hybrid weight is 0.7
Empirically tuned on internal eval set v3. Weight of 1.0 (dense-only)
loses 4pp recall on keyword-heavy queries. Weight of 0.5 over-prioritizes
lexical matches on long natural-language queries.

## Eval summary (2025-03-14 build)
- Recall@10: 0.79
- MRR@10:    0.61
- P95 latency: 285 ms
- Cost per 1k queries: $0.90

Notice that every non-obvious number has a "why." Notice that the limitations live next to the strengths. Notice that the eval summary has a date and a build identifier.

The Operational Workflow

Documentation drifts. Models drift faster. The only durable defense is a workflow that forces reconciliation.

Generate from Source Where Possible

API references, SDK references, and configuration tables should be generated from code, not hand-written. If a parameter changes in the SDK, the docs change with it. This is the lowest-effort, highest-leverage thing you can do.

Regenerate Eval Tables from Eval Runs

Treat your eval suite like a CI job. Every model build runs the suite, and the results table is published automatically. If a number in the docs differs from the latest run, the docs are wrong; fix the docs.

Tie Doc Updates to Code Reviews

A pull request that changes a prompt template, a model identifier, or an eval set should not merge without a doc diff. Make this a checklist item. Many teams add a CODEOWNERS rule that requires a docs reviewer to approve any change touching the model layer.

Version the Docs

GitHub branches are fine for now. When you ship a v2 API, ship v2 docs. Old versions stay accessible, with a clear "deprecated" banner and a migration guide.

Audit Quarterly

Every quarter, walk through the four audiences and ask: if I were this person today, would I find what I need? It takes an afternoon and surfaces gaps that daily work hides.

Common Failure Modes

Documenting the Wrapper Instead of the Model

The README describes the SDK. The model card, if it exists, is a copy-paste from the upstream provider. The customer integrates, the model changes, the customer is surprised. This is the most common failure mode for thin wrappers around frontier models. The fix is a system card that documents your prompts, your retrieval, your rerankers, and your guardrails — the parts you actually own.

Static Evals That Nobody Trusts

A results.md file checked in once, never updated, with no version pin. Evaluators ignore it. Fix it by wiring evals into CI and regenerating the table on every run.

Cost Docs That Lie

Cost numbers go stale as soon as upstream providers change pricing. If your cost paragraph is older than three months, assume it is wrong. Either regenerate from a billing query or add a date stamp and a caveat.

Limitations Buried in a Footer

If the limitations are six scrolls down in a 4,000-word README, they are not really documented. Move them up.

One Doc to Rule Them All

The "complete guide" that tries to serve integrator, evaluator, operator, and reviewer at once. It ends up serving none. Split it.

Tooling and Where DocPilot Fits

The hard part of AI product documentation is not writing it. It is keeping it synchronized with code, prompts, evals, and configuration that change weekly. DocPilot is built specifically for this problem: connect a repo, point it at the components of an AI product (the API handlers, the prompt templates, the eval suite, the model card), and it generates the README, the API docs, the model card, and the configuration reference, then keeps them in sync via webhooks on every merge. The eval table can be regenerated on every eval CI job, so the docs you ship are the docs that match the code you shipped. For teams that have outgrown a hand-maintained docs/ folder but are not ready to staff a docs team, this is the right tool for the moment.

Closing Checklist

Before you ship your next AI product doc update, run this list:

Documentation for AI products is not a chore at the end of the project. It is part of the product surface, and for many buyers it is the product surface they evaluate you on. The teams that treat it that way ship faster and lose fewer customers to misaligned expectations.

If you want to stop hand-maintaining AI product docs and let the tool regenerate them from your code, prompts, and eval results, try DocPilot free at docpilot.io — it takes about ten minutes to connect a repo and produce a first draft you can actually ship.