Reference
Benchmarks
Every measurement on this site, in one place. Each table links to the post that produced it, where the setup, the caveats and what was inferred rather than measured are written down.
- Tables
- 7
- Rows
- 27
Reranker comparison — Pass@k, MRR, wall clock
42 Korean issue-title queries · hybrid dense+BM25 index · top-50 candidates reranked
| Reranker | Pass@1 | Pass@5 | Pass@10 | MRR | Wall clock |
|---|---|---|---|---|---|
| none | 26.2% | 45.2% | 47.6% | 0.333 | 33s |
| jina-reranker-v1-turbo-en | 23.8% | 42.9% | 47.6% | 0.326 | 54s |
| jina-reranker-v2-multilingual | 26.2% | 42.9% | 47.6% | 0.330 | 282s |
| bge-reranker-v2-m3 | 28.6% | 42.9% | 47.6% | 0.352 | 514s |
Pass@10 is identical in all four runs. Reranking reorders a candidate set; it cannot add to it.
Embedding models — Korean queries against English-identifier code
56 issue titles · same corpus, chunking and binary · random guessing lands near 2%
| Model | Pass@1 | Pass@10 | MRR | Index time |
|---|---|---|---|---|
| embeddinggemma-q4 | 32.1% | 42.9% | 0.356 | 656s |
| multilingual-e5-small | 26.8% | 42.9% | 0.332 | 261s |
| paraphrase-minilm | 1.8% | 16.1% | 0.065 | — |
| bge-small | 3.6% | 7.1% | 0.042 | — |
| jina-code | did not finish in 19 min | — | — | — |
English-only models do not degrade gracefully; they stop working. Multilingual is necessary but not sufficient — the model must also be trained for retrieval rather than sentence similarity.
Source · Picking an embedding model when the queries and the code are in different languages →
Removing 89 prose documentation files from a code index
42 issue-title queries · same model, chunking and queries · docs were 3% of chunks
| Metric | With docs | Docs excluded |
|---|---|---|
| Pass@1 | 9.5% | 26.2% |
| Pass@5 | 19.0% | 45.2% |
| Pass@10 | 35.7% | 47.6% |
| MRR | 0.142 | 0.333 |
The largest single improvement measured on this corpus came from deleting 3% of it, not from changing the model.
Source · Three percent of my index was eating the top five results →
Embedding throughput — CPU vs Metal, same model and corpus
1,024 chunks averaging 688 bytes · Apple Silicon
| Configuration | chunks/s |
|---|---|
| candle, CPU f32 | 3.8 |
| ONNX, CPU int8 | 19.8 |
| Metal f16, batch 256 | 20.1 |
| Metal f16, token-sorted, batch 4 | 64.6 |
The GPU lost three times before it won. All three losses were measurement setup, not hardware.
Source · I measured Metal wrong three times before it beat the CPU →
Batch size vs throughput — padding is the cost
same 1,024 chunks every run · token-sorted · Apple Silicon Metal
| Batch | Order | Wall clock | chunks/s | Padded tokens |
|---|---|---|---|---|
| 4 | token-sorted | 15.9s | 64.6 | 196,688 |
| 64 | token-sorted | 21.0s | 48.8 | 204,736 |
| 256 | token-sorted | 29.9s | 34.2 | 230,656 |
Batch 4 beat batch 256 by 1.9x. A large batch pads every sequence to its longest member, and the model is bandwidth-bound at this size. The post also reports the unsorted runs, which show how much of the loss sorting recovers.
Production indexing — ONNX int8 vs Metal f32
one repository, 406 files · retrieval quality identical to three decimals
| Metric | ONNX int8 | Metal f32 |
|---|---|---|
| Cold index | 294s | 152s |
| CPU time consumed | 1,736s | 7.2s |
| Pass@1 / Pass@5 / Pass@10 | 26.8 / 42.9 / 42.9 | identical |
| MRR | 0.332 | identical |
The CPU-time row is the one that changed how the machine feels. Wall clock halved; CPU time fell by 99.6%.
Source · I measured Metal wrong three times before it beat the CPU →
A strategy that never reads the input
20-question sample from 117 labelled tickets · only the sampling method changed
| Sample | Strategy | Exact | Recall |
|---|---|---|---|
| top 20 by repo count | no judgement at all | 95% | 100% |
| top 20 by repo count | the actual system | 79% | 92% |
| round-robin, 8 answer shapes | no judgement at all | 15% | 55% |
On the broken sample, answering the same three repositories every time beat the actual system on every column. Fixing the sample dropped that strategy from 95% to 15%.
Source · The baseline that beat my agent was answering the same thing every time →