The knowledge layer decides whether your answers are right
Ingestion, chunking, hybrid retrieval and evaluation for systems answering from your own documents — permission-aware, kept fresh, measured.
Retrieval
The passage that would have answered was never retrieved: the PDF was scanned and nobody ran OCR, a fixed-size chunker cut the table from its header, or the index was last rebuilt in March. We measure retrieval on its own first.
Ingestion
Scanned PDFs, old Word files, spreadsheets with meaning in the layout. OCR and layout-aware extraction keep tables and headings intact, and changes are picked up incrementally.
Chunk size
256–1,024
the token range we sweep; the right value is measured, never a default
Permissions
0
documents retrievable that the user could not already open at source
Search
Dense embeddings handle paraphrase and miss exact part numbers; BM25 does the reverse. Fusing both is the largest win in most systems, and a cross-encoder reranker the second.
Chunking
Cutting every document at 512 tokens severs a clause from the section that qualifies it. We split on headings, sections and tables, with parent-child retrieval for context.
Vector and search
Parsing
Pipelines
Embeddings
Evaluation
Who we do this for
2–3 weeks
We build a labelled question set against your corpus, measure retrieval quality and show where the answers are lost.
Labelled evaluation set, retrieval baseline, ranked findings
6–14 weeks
Ingestion, parsing, chunking, indexing, hybrid retrieval, permission enforcement and an evaluation harness in your environment.
Pipelines as code, configured indexes, permission model, runbook
Ongoing
We run the pipelines under an SLA and keep connectors working as the source systems change.
Freshness monitoring, index maintenance, quarterly retrieval review
When the knowledge is small and stable, put it in the prompt rather than operating a pipeline. When the requirement is aggregate reporting over structured data, you want SQL and a semantic layer, not a vector index. And when the documents are out of date, retrieval surfaces that faithfully.
If you already run Postgres and expect under roughly ten million chunks, pgvector is usually right and removes a system from your estate. Qdrant, Milvus or Weaviate earn their place at larger scale or with heavy filtering. It matters far less than chunking, hybrid retrieval and reranking.
Yes, routinely. Mixed-language corpora need multilingual embeddings, script-aware OCR and evaluation sets in each language — English-only test questions hide a large share of the failures. We also test cross-lingual retrieval directly, since asking in one language about documents in another is a common silent failure.
Send us the constraint you are actually up against — budget, latency, regulator, deadline.
