The queries are Korean issue titles. The documents are TypeScript and Java with English identifiers, Korean comments, and Korean UI strings. Every retrieval decision in this system is downstream of a model that has to bridge those two.
I benchmarked five embedding models on 56 real issue titles against the same corpus, same chunking, same binary. Two of them scored under 4%.
| Model | Pass@1 | Pass@10 | MRR | Index time |
|---|---|---|---|---|
| embeddinggemma-q4 | 32.1% | 42.9% | 0.356 | 656s |
| multilingual-e5-small | 26.8% | 42.9% | 0.332 | 261s |
| paraphrase-minilm | 1.8% | 16.1% | 0.065 | n/a |
| bge-small | 3.6% | 7.1% | 0.042 | n/a |
| jina-code | did not finish in 19 min |
Random guessing over this corpus lands somewhere near 2%. Two of these models are not doing retrieval.
English-only models don't degrade, they stop
This is the part that surprised me, and it's the reason I'd write the post even though "use a multilingual model for multilingual data" sounds like advice nobody needs.
I expected an English-only model to be worse. Maybe half as good, since it still sees the identifiers, the file paths, the English keywords in the code, and those carry some signal. A query about product form might still land near a file called ProductForm.tsx.
That's not what happens. bge-small scores 3.6% on Pass@1 and 7.1% on Pass@10, which means that nine times out of ten the correct file isn't in the top ten of a corpus it has fully indexed. The tokenizer sees Korean text as a run of unknown or near-unknown pieces, the embedding it produces carries almost nothing from them, and what remains is not enough to rank anything.
There's a practical consequence: you cannot detect this by spot-checking. A model at 3.6% still returns ten plausible-looking files for every query: files from the right repository, often the right directory, with familiar names. Nothing about the output announces that the ranking is arbitrary. I ran queries against these models by hand before I had a gold set and did not notice anything wrong.
It took 56 labelled queries to see it. That's the entire argument for building a gold set even when it feels like overkill for a small tool.
The one that never finished
jina-code is trained specifically on code, which made it the model I most expected to win. Code on both sides of the comparison, purpose-built.
I killed it after nineteen minutes without an index. The other models had finished the same corpus in four to eleven.
I never found out how well it retrieves, and I'm comfortable with that. A model that takes more than three times as long to index is not a candidate for a corpus that gets rebuilt whenever the code changes, because quality would have to be extraordinary to buy back that much, and "extraordinary" was not on the table given that two other models were already producing usable results at a quarter of the cost.
There's a version of benchmarking where you wait it out for completeness. I've stopped doing that. If a candidate is disqualified on operational grounds, measuring its accuracy is collecting a number you will not act on.
Multilingual is necessary and not sufficient
The second thing I got wrong is subtler and cost more time.
paraphrase-multilingual-MiniLM handles Korean fine. It's in the name. I assumed it belonged in the "should work" group and was surprised when it landed at 1.8%, worse than the English-only model next to it.
The problem isn't language coverage. It's that the model is trained for sentence similarity, which is a symmetric task: two pieces of text, how alike are they. Retrieval is asymmetric: a short query on one side, a long document on the other, and the question isn't "are these similar" but "does this document answer this query."
Models trained for retrieval learn that asymmetry, usually with distinct handling for the two roles. The E5 family literally prefixes them (query: and passage:) during training. A symmetric model has no such notion; it embeds the issue title and the code chunk into the same space with the same assumptions, and a five-word query is simply not similar to a forty-line function no matter how related they are.
So the filter is two-stage: multilingual, and trained for retrieval. Missing either one puts you near random, and "multilingual" is the one that's advertised in the model name while "trained for retrieval" is the one you have to go read about.
Choosing between the two that worked
Both remaining models are usable. The difference is not quality:
The gap is 5.3 points of Pass@1 on 56 queries, three queries' worth. That's inside the noise for a sample this size, and Pass@10 is identical at 42.9% for both.
Indexing time is not inside the noise. embeddinggemma takes 2.5× longer, and across the whole corpus that's the difference between a two-hour reindex and a five-hour one. The dimension count compounds it: 768 versus 384 means the stored index is twice the size and every query does twice the vector arithmetic.
I took the smaller, faster one. When two options are tied on the thing you care about, decide on the thing you can measure without arguing.
The transition is not incremental. Vectors from different models are not comparable, so changing the model means rebuilding every index, because searches against a half-migrated corpus return nothing useful rather than degraded results. Budget for it as one operation.
The prefix that did nothing
E5 models are trained with query: and passage: prefixes, and the library wasn't applying them. I found this while reading the tokenizer path, added them, and expected a clear improvement since this is the documented usage.
The difference was noise. Well within run-to-run variation on 56 queries.
My guess, and it is a guess, is that the prefixes teach the model to distinguish a question from a prose passage, and my documents are not prose passages. They're code chunks. Whatever the passage: prefix means to the model, a TypeScript function is not what it means it about. I left the prefixes in because they're correct usage and cost nothing, but I stopped expecting anything from them.
Worth recording as a negative result: the documented best practice for a model can be irrelevant for a corpus that doesn't look like its training data.
What the ceiling looks like
Something worth naming: the two working models both top out at 42.9% Pass@10, identically.
Five points separate them at Pass@1 and nothing separates them at Pass@10. Whatever limits this system is not the embedding model. Both models find the same set of candidates and differ only in how they order them.
That reframed where I spent the next round of effort. Model comparison is the visible knob, and it's where I'd been putting my time because there are model cards to read and leaderboards to check. But if the ceiling doesn't move when you change models, the ceiling is somewhere else: chunking, what's allowed into the index, how the hybrid weighs lexical against dense.
The largest single improvement I found on this corpus afterwards came from excluding 89 documentation files, and it moved MRR from 0.142 to 0.333, more than double what any model swap had produced. Nothing about comparing model cards would have pointed there.
Model selection matters at the bottom of the range, where a wrong choice puts you at random. Above that it matters much less than the rest of the pipeline, and it's the part that feels most like progress while you're doing it.
What I'd do on a new corpus
Build a gold set first, even a small one. Fifty labelled queries is a weekend afternoon if you mine them from closed tickets: the query is the ticket title, the answer is the files its closing commit touched. Without it you cannot tell a 3.6% model from a 30% model by looking, and I have tried.
Test the models you assume will fail. bge-small was in my comparison as a control, and it produced the most useful number in the table. It showed me what broken looks like. Every other result is interpreted relative to it.
Check "trained for retrieval," not just "multilingual." The model card says which. Symmetric similarity models are common, well-regarded, and wrong for this.
Decide the tie on operations. Two models within noise of each other on quality are not tied on index time, index size, or query latency. Those are measurable without a statistical argument.