Search does four jobs in my agent workspace. It finds the right note among 123 markdown memory files. It finds the old conversation where something was decided, in 2,961 transcripts. It picks which of 28 repositories a ticket belongs to. And inside that repository it suggests which files to open. One engine did all four: a Rust code-search binary with multilingual-e5-small vectors (384 dimensions) and a tantivy BM25 index, fused with reciprocal rank fusion. A hybrid.
Vectors were there for the language gap. The queries are Korean and the code is English, and I had already measured that the choice of embedding model moves Pass@1 from 1.8% to 32% on this kind of data (the model post). A lexical index cannot connect a Korean sentence to an English identifier, so a lexical index was not the plan.
Then I built gold sets for all four jobs and a SQLite FTS5 table matched or beat the hybrid on every one of them, at 28 ms instead of 750. That is the smaller of the two things I found.
The measurement that said do not switch
Four days earlier I had looked at exactly this question and concluded the opposite.
Someone suggested moving search to SQLite full-text for speed. I timed a query and broke the 800 ms down:
And it did not depend on corpus size. 768 chunks took 870 ms, 22,282 took 847, 76,968 took 828, 145,971 took 893. The cost was embedding the query, which happens once per query regardless of what it is searched against. Changing the storage engine would not touch it. I wrote that down as no reason to switch to SQLite and moved on.
Accuracy I checked with 14 questions I had written myself: the vector engine got 11 of 14 at rank one and 14 of 14 in the top three, and an FTS5 table over the notes' one-line descriptions got 12 and 13. Close enough. Two days later I rebuilt that gold set from 39 questions the user had actually typed, and the description index lost: it missed 9 questions entirely where the full-text index missed 4, and there was not a single question it found that the full text did not. My 14 questions had been written by someone looking at the descriptions, and they matched the descriptions.
So I threw out the description idea, kept the vector engine, and did not notice two things. I had never measured the vector engine's accuracy on a gold set. I had measured its speed and assumed the accuracy. And the 3.1 ms attributed to the BM25 half of a hybrid engine was not the cost of running BM25. It was the cost of not running it.
The hybrid that was vector-only, again
This blog has already told the early-termination story three times, so briefly: the engine skips the lexical search when the top five vector hits look confident, the confidence threshold was set for a previous model, and with the current model the condition is true on every query. I found that, wrote it up, and set CODESEARCH_EARLY_TERMINATION=0 for the caller I was measuring at the time, the one that suggests files.
The caller that searches memory notes never got the variable. It ran the default. So the measurement two days earlier, the one that rejected description indexing, had measured a vector-only engine and called it hybrid.
This time a number caught it. The gold-set harness ran the default binary and a --vector-only flag as separate engines, and the top-five inclusion table between them came out like this:
Thirty-eight questions in the table after the leak adjustment described below, and identical ranks on all thirty-nine before it. Two code paths that differ cannot produce that row. The flag post ends with a rule: for each component you believe is running, find a number that proves it ran. This is the first time I had a number that did.
Four gold sets
Four engines on the same data: the hybrid with early termination turned off, vector-only, SQLite FTS5 with the unicode61 tokenizer, and FTS5 with trigram. The FTS5 tables are Node's built-in SQLite, ranked by bm25(). The query is the space-separated tokens joined with OR, each also added as a prefix with its last one or two characters stripped, the cheap way to survive Korean particles, which an earlier post measured on its own. For the code repositories FTS5 indexed the same file list the vector index had, cut into fixed 25-line windows, because FTS5 does not know where a function ends.
The gold sets:
- Memory notes. 39 questions the user actually typed, each with one correct note. Three notes written after the gold set quoted gold-set questions as examples, so I removed them from every engine's ranking; one question's answer was among them, leaving 38.
- Repository routing. 228 distinct human-filed tickets, gold = the repository where the fix branch was cut, eight candidate repositories. 42 tickets have more than one correct repository.
- File finding. 273 cases: the same 228 tickets, counted once per gold repository, so a ticket whose fix spanned two repositories appears twice. Gold = the files the fix touched, searched within that repository.
- Conversation history. 39 questions against 2,961 transcripts. Strict scoring credits only the session that wrote the memory note (18 questions have one); loose also credits any session containing the question verbatim (27).
One run per engine per task, no repeats. Rankings are deterministic, so a second run would reproduce them, but a single run is a point and not a band, and the tables should be read that way. The vector engine's ranks matched the run from two days earlier on 28 of 39 questions, and the other eleven mostly moved one or two places; the larger moves were on notes that had been edited in between.
| task | n | hybrid (ET=0) | vector-only | FTS5 unicode61 | FTS5 trigram | FTS5 + .vue |
|---|---|---|---|---|---|---|
| memory notes, top-1 | 38 | 26% (10) | 26% (10) | 37% (14) | 34% (13) | — |
| memory notes, top-3 | 38 | 45% (17) | 50% (19) | 61% (23) | 53% (20) | — |
| repository routing, top-1 | 228 | 39% (89) | 36% (82) | 43% (99) | 46% (106) | 47% (107) |
| repository routing, top-3 | 228 | 62% (141) | 69% (158) | 79% (180) | 76% (173) | 81% (185) |
| file finding, any gold file in top-5 | 273 | 25% (68) | 23% (62) | 25% (67) | 20% (54) | 37% (100) |
| same, only cases with a gold file in the index | 191 | 36% (68) | 32% (62) | 35% (67) | 28% (54) | 34% (64) |
| one query, fresh process, median | 720–790 ms | 720–755 ms | 28–39 ms | 28–29 ms | same |
The totals are less informative than the inclusion tables, which show the questions each engine gets that the other does not:
Three different answers. On memory notes FTS5 is ahead, but 38 questions cannot order two engines; the honest reading is not the same, and the direction is FTS5. On routing the asymmetry is 14 against 53 and that is a direction. On file finding the two engines tie: each finds twenty cases the other misses, and the union (87 of 273) is better than either alone, which I note without having measured a fused version.
Conversation history was the fourth task, and there the answer was the same:
| engine | strict top-1 / top-3 (n=18) | loose top-1 / top-3 (n=27) | median query | index time | index size |
|---|---|---|---|---|---|
| hybrid | 67% / 78% | 85% / 93% | 760 ms | 56 min on GPU (1 h 33 m on CPU) | 800 MB |
| vector-only | 56% / 78% | 78% / 93% | 728 ms | same index | same |
| FTS5 unicode61 | 67% / 89% | 85% / 96% | 28 ms | 15 s | 119 MB |
| FTS5 trigram | 67% / 89% | 85% / 96% | 12 ms | 15 s (both built together) | 273 MB |
| a dedicated transcript-search tool | 67% / 89% | 85% / 96% | 4,851 ms | 50 s | 269 MB |
The last row is a third-party tool built for exactly this corpus, a Go inverted index with an optional vector layer that was off. In the strict top-five, the three lexical engines have an inclusion row of 0 / 0 / 16 / 2: they find the same sessions and miss the same sessions. The hybrid trades one question each way. It finds one the lexical engines miss and misses one built around an upper-case identifier, which is the kind of thing the embedding model reads past and a tokenizer does not. Eighteen questions cannot rank these engines and I am not going to pretend they can. What is left after the same is cost, and the cost column is not close.
A colleague reproduced the conversation comparison on a different machine and corpus with 27 questions, one run: FTS5 24 of 27 at rank one, 27 of 27 in the top three, 3 ms; the dedicated tool 21 and 24 at 900–2,500 ms. That corpus had only seven substantive sessions, so the near-perfect numbers say more about the corpus than the engines.
The larger finding: files nobody indexed
The file-finding gold set has 543 correct files. Before ranking anything I checked how many of them the vector index contained.
A third of the answers could not be returned by any engine reading that index. And the missing files are not spread evenly: two of the eight repositories are legacy admin apps written in Vue single-file components. In legacy-admin, 26 of 76 cases have at least one gold file in the index. In partner-admin, 10 of 42. The other five repositories are at 100%. For weeks the agent's file suggestions and diagnosis hints had been effectively blind in those two repositories, and nothing said so. The searches returned results, just not from the files that mattered.
Adding the 521 tracked .vue files to the FTS5 corpus took two seconds of indexing (19.9 s became 21.9 s) and 33 MB. That one change moves file finding from 25% to 37% at top-5 and routing from 79% to 81% at top-3, the last column of the table. And the row below it is the control: on the 191 cases whose gold files were already indexed, the .vue column does not move (35% to 34%). The gain is entirely corpus. No ranker was changed.
The same fact explains part of the routing gap. At the ticket level, 48 of the 228 routing tickets have no gold file in the index in any repository. That is the 82 file-finding cases with nothing indexed (273 minus 191), collapsed to tickets. On those, the hybrid puts the right repository in the top three 5 times; FTS5 does it 31 times, with the same file list. The gold file is missing, but the repository's other files (service modules, translation tables, configuration) still contain the ticket's words, and BM25 finds the repository through them. The vector engine drifts instead: of the 118 tickets belonging to the two legacy admin apps, it sends 70 to the API repository, because a product name in the ticket is closest in embedding space to controller classes named after that product. FTS5 sends 49 of the same tickets to the newer admin app that replaced the legacy one. Both are wrong in the direction of a sibling repository; they are wrong differently.
I want to put this plainly because it is the point of the post. When I concluded no reason to switch on the ninth, and when I concluded FTS5 is ahead on the thirteenth, the largest loss in the system was the same on both days: 170 files that no engine had read. The ranking experiments were second-order to that, and I ran them first.
What it costs
| corpus | vector index | FTS5 index | one query |
|---|---|---|---|
| memory notes, 123 files, 695 KB | 36.0 s (3.1 GB resident) · 6.8 MB | 0.13 s · 7.8 MB | 720 ms vs 28 ms (0.75 ms once the process is up) |
| code, 8 repositories, 21,605 files, 105 MB | ≈107 min estimated · 2,089 MB | 19.9 s · 664 MB | same |
| transcripts, 2,961 files, 67.9 MB | 56 min on GPU, 1 h 33 m on CPU · 800 MB | 14.9 s · 119 MB | 760 ms vs 28 ms |
The 107 minutes is an estimate, not a measurement: I reused the existing code index rather than rebuilding it, and computed the figure from a throughput of 35 chunks per second measured on another day against the 225,149 chunks it contains. The other two rows are measured.
The latency finding from the ninth stands. 97% of a query is embedding the query, and it does not depend on corpus size. What was wrong was not the decomposition. It was assuming that the accuracy I had not measured was the same.
What moved and what stayed
Memory notes moved to FTS5. The vector index for them is retired. The production module reproduces the benchmark exactly, 39 of 39 questions at identical ranks with section-sized chunks. A 25-line-window variant finds three more in the top five (26 to 29) and is still being checked.
Conversation history moved to FTS5, and neither the vector engine nor the dedicated tool is used. The dedicated tool indexes tool output unconditionally (the setting that is supposed to turn that off had no effect), and five sessions that had read the gold set took the top five for all 39 questions until I removed them. Five seconds per query did not help. The FTS5 index refreshes every thirty minutes, 0.4 s per incremental pass, text only, with tool output and reasoning excluded.
Repository routing is in progress. FTS5 with the .vue files is being tried as the candidate generator, A/B against the current engine inside the agent harness rather than on the gold set alone. The hybrid still finds 13 to 29 tickets FTS5 does not, depending on the cutoff, so the union is in the trial too. Not decided.
File finding waits on the corpus. Making the indexer read .vue, whether with a parser in the fork or the FTS5 side-table, comes before any ranking experiment there. On the shared file list the engines tie, and the union is larger than either; that is arithmetic from the inclusion table, not a measurement of a fused ranker.
Where the vectors still earn their 750 ms, one or two questions in 38 each way: a note whose body never uses the query's word (the question asks about budget, the note says token six times and budget zero, and only the filename has the English), and Korean transliterations of English names, where the query says the name the way people say it and the note only has it in Latin letters. The vector engine ranks those where a tokenizer returns nothing: sixth place, in the transliteration case, which is not the top five either. The reverse case is the upper-case identifier above. Neither list is long enough to decide anything, and I am noting them so the next person does not read this post as embeddings are useless.
What I would do differently
Read the inclusion table before the totals. Two engines with the same total can be finding different things or exactly the same things, and the total cannot tell you which. A 0 / 0 row between two configurations means one code path, whatever the flags say.
Count the corpus before ranking it. How many of the gold answers are in the index at all? Sixty-five percent, in my case, and the number was available in ten minutes from the index's own file list. It is the ceiling for every engine, and I ran comparisons under it for days.
Do not infer accuracy from a latency breakdown. The breakdown was correct. The conclusion drawn from it was about a quantity it did not contain.
Write n next to every number. Thirty-eight cannot order two engines. Two hundred twenty-eight can say which direction. Eighteen can say the same, and the same is a result. It is the one that let cost decide.
Frequently Asked Questions
Does this mean BM25 beats embeddings?
No. On the shared file list the two engines tie on file finding, 68 against 67 with twenty cases each that the other misses, and their union is larger than either. It means my hybrid was not hybrid, and that a plain FTS5 table gives the same accuracy at a twenty-fifth of the latency and a tiny fraction of the indexing cost. Whether a properly fused hybrid beats FTS5 alone is a question these tables do not answer.
Why unicode61 rather than trigram?
Because unicode61 is a superset here: on memory notes its top-five contains everything trigram's does plus five more (5 / 0), on file finding 14 / 1. Trigram has a slightly higher routing top-1 (46% against 43%) and loses at top-3. Korean particles are handled by the prefix query, not the tokenizer.
Would turning early termination off have saved the vector engine?
It was off. That is the hybrid (ET=0) column. Memory notes 45% and routing 62% at top-3, against 50% and 69% for vector-only. On prose, fusing BM25 in was a three-for-three trade with no net gain: the top-five inclusion row between hybrid and vector-only is 3 / 3 / 20 / 12. That is not a contradiction of earlier results where fusion helped: on a code index at file level, fusion beat both rankers alone (MRR 0.323 vector, 0.357 lexical, 0.448 fused, p < 0.05 by randomisation), and on a description index for routing it cost eighteen first places (the one-case post). Fusion's value depends on the corpus. What this post can say is that on all four corpora, FTS5 alone matched or led.
How wrong was the no reason to switch conclusion?
The latency decomposition was right, 97% is query embedding independent of scale, and it is still the reason vector search costs what it costs. The mistake was treating an accuracy I had not measured as known.