Benchmarks

Vector search retrieves what sounds similar. It does not know what you are doing.

The same question means different things to a billing analyst and a data engineer. FactRule understands what you are working on — not just the words you typed. Below is what that is worth, measured the hard way: weights frozen, then evaluated once on questions and entities the system had never seen. We publish the losses alongside the wins, because a benchmark you can only quote selectively is marketing, not measurement.

Held-out benchmark: +49% recall over plain vector search

1,112 questions in a wording register the system was never tuned on, over a real 39,830-document data estate. Recall@10 — how often the right memory is in the top ten.

Plain vector search
the industry baseline
0.329
FactRule Deep Context
the shipping configuration, frozen before the test
0.491

Both systems saw identical questions, an identical corpus and an identical context budget. On the development set the gap holds across two independently-written question wordings at p < 10⁻⁷, paired per question — it is not a lucky phrasing.

On the entities your embeddings barely know, similarity is nearly useless

96 questions about entities no tuning run ever touched — the long tail every real schema has. This is where retrieval products actually differ.

Plain vector search
0.023
FactRule retrieval stack
0.264

Plain similarity finds 2% of the right memories; FactRule finds a quarter. Not because it embeds better — because it understands your estate and what you are working on, not just the words in the question.

Does it change the answers? Same model, live database, one variable

24 executable questions where a bare table name exists in two schemas and only the analyst's stated context says which — the ambiguity every warehouse has. Answers graded by running the SQL and comparing result signatures. No judgement calls.

Plain vector search
10/24
FactRule Deep Context
16/24

Exact McNemar p = 0.031, and the discordant count is 0/6: plain search never wins a question FactRule loses. It even picks the empty table when the user's context calls for it — the case every popularity heuristic gets wrong.

Where it does not help

An evaluator will find these in their first hour, so here they are first.

Ordinary questions gain nothing
On 148 recorded production questions that name the table they need, FactRule is level with plain search (0.85 vs 0.85). When a question already says what it wants, there is no ambiguity for state to resolve.
The answer study is small
24 gradable cases is what one real estate yields. The result is significant and the mechanism is demonstrated — but we call it a demonstration, not a product-scale benchmark.
One estate, one embedder
The retrieval numbers come from one 39,830-document corpus and one embedding model. The holdout protocol is what travels; the absolute numbers may not.
Some families were tested on wording only
Two question families exhausted their fresh entities during development, so for those the holdout proves robustness to rewording, not to unseen entities. The headline families carry the fresh-entity claim.

The only benchmark that matters is your schema

Our numbers were measured on our estate. Yours has its own homonyms, its own empty decoy tables, its own tribal knowledge. The collector runs inside your network — Windows or Linux — builds a portable memory from your schema, code and documents, and tells you the cost before spending a call: on our estate it projected 119 model calls and the build made exactly 119. Nothing leaves your network until you decide it should.

Start a free trial See how it works Book a results walkthrough

sign up → download the collector from your account → preflight → estimate → build — no admin rights, nothing leaves your network

Method: DEV/TEST split by entity hash · paraphrase control (worse of two wordings reported) · frozen weights before holdout · answers graded by result-signature equality on a live read-only database · retrieval harness fully deterministic.