RAG over documentation: chunks, an index and answers with sources

RAG over documentation is an index of chunks with their embeddings plus a prompt that receives only the relevant passages: it answers with the file cited instead of guessing. Quality is decided in how you cut the text, before the model.

The chunk is the unit of work

A chunk that is too large brings noise into the prompt and costs tokens; one that is too small loses the subject of the sentence. Cutting at 1,200 characters with 200 of overlap keeps a paragraph together and lets you cite the file.

• 1,200 characters: about 300 tokens, so five chunks fill 1,500 tokens of context.
• 200 characters of overlap: a sentence across the cut appears whole in at least one chunk.
• One file, many chunks: LIBRARIES.md alone produces 2,470 chunks and should be split by library or dropped.

The index in Postgres

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id bigserial PRIMARY KEY,
  source text NOT NULL,
  chunk_index int NOT NULL,
  content text NOT NULL,
  embedding vector(1536) NOT NULL
);

CREATE INDEX doc_chunks_embedding_idx
  ON doc_chunks USING hnsw (embedding vector_cosine_ops);

The schema lives in schema.sql and is applied with psql "$DATABASE_URL" -f schema.sql. The vector(1536) column has the dimension of the embedding: change the model and you change the column and rebuild the index. HNSW makes the search approximate, so fast but not exact.

The cut, in code

const SIZE = 1200;
const OVERLAP = 200;

export function chunk(text: string): string[] {
  const out: string[] = [];
  for (let i = 0; i = text.length) break;
  }
  return out;
}

The function is deterministic: the same file produces the same chunks, so you can compare an old index with a new one. Store source and chunk_index on every row: without those two fields the answer cannot be checked.

The answer cites its source

• Five chunks, not twenty: the prompt grows linearly and so does the cost.
• A threshold you pick, 0.3 of cosine distance for example: below it the model gets “I do not know” instead of context.
• Source in the clear: every sentence of the answer carries the file and the chunk index.

The limitation

The index ages: if you update a file without recomputing its chunks, RAG answers with the old version and cites it with confidence. You need a file fingerprint per row and a job that rebuilds only the changed files; in our case that job is not automated yet. Cutting by characters also ignores sentence boundaries.

Where to continue

For where to keep the vectors, the comparison between SQLite and Postgres in a small project; the prompt that receives the chunks is in prompt engineering for agents; the protocol for exposing the search to a client is in what MCP is. Ready-made parts for a chat with sources are in chat components, for example Jahed AI chat input and chatgpt prompt input, inside 111,985 components.