Question 1

Can a reranker improve recall?

Accepted Answer

No. Pass@10 was 20/42 in all four reranker configurations, identical to the decimal — a reranker reorders the candidate set it was handed and cannot add to it. That the candidate set beyond rank ten held nothing useful was inferred from four models agreeing, not measured directly — recall@50 was never logged. A postscript to the post later found the cause: an early-termination threshold tuned for an older embedding model was skipping the lexical half of retrieval on every query checked, so the candidate set came from one retriever, not two.

Question 2

What does an English-only reranker cost on non-English queries?

Accepted Answer

4.8 points of Pass@1 on 42 Korean queries, and 5.4 on a second set of 56. The English-only default scored below running no reranker at all. Recall did not move — the loss is entirely in ranking.

Question 3

Does fusing two rankers with RRF always help?

Accepted Answer

No. When one ranker cannot see 11 of 28 candidates, fusion penalises exactly those — two test cases fell from 1st to 14th and 6th to 17th. RRF is sound when both rankers have comparable coverage. The failure is structural, not a tuning problem.

Question 4

How much can removing files from an index improve search?

Accepted Answer

Excluding 89 prose documentation files — 3% of chunks — moved MRR from 0.142 to 0.333 and Pass@1 from 9.5% to 26.2%. Same model, same queries, same chunking. Those files matched the vocabulary of the queries almost exactly. Prose in a code index competes on the model's terms, not yours.

Question 5

How badly do English-only embedding models do on non-English queries?

Accepted Answer

1.8% and 3.6% Pass@1 where random guessing lands near 2%. They do not degrade — they stop working, and the output still looks reasonable. Multilingual is necessary but not sufficient: a multilingual sentence-similarity model scored 1.8%, below the English-only model it was meant to beat.

Question 6

Why is my GPU slower with a larger batch size?

Accepted Answer

Padding and bandwidth. Batch 4 beat batch 256 by 1.9x on the same corpus — 196,688 padded tokens versus 230,656 for identical input. This is a small-model, variable-length-input result. Large models are compute-bound and the usual batching advice holds.

Question 7

Is half precision worth it?

Accepted Answer

It depends on batch size, which means it cannot be evaluated in isolation: 19% at batch 256, about 3% at batch 4. Precision helps where there is more data in flight. Measuring it alone on a default batch size produces a confident wrong recommendation.

Question 8

My accelerator is slow — is it working hard or waiting?

Accepted Answer

Check utilisation, not wall clock. A run at CPU 21% with the GPU idle for twelve minutes is a plumbing problem; none of the usual optimisations touch it. The cause here was CoreML recompiling for every new input shape. A compilation cache did not help — the cost was crossing 39 segment boundaries, not compiling them.

Question 9

What does moving embeddings to the GPU actually buy?

Accepted Answer

Cold index 294s → 152s, and CPU time 1,736s → 7.2s, with retrieval quality identical to three decimals. The CPU-time row is the one that changed how the machine feels. On a dedicated server that distinction disappears.

Question 10

What does running embeddings locally cost?

Accepted Answer

672,487 chunks in about five hours as a one-time migration, then two minutes a day incremental. The expensive part was a week of tuning and two hundred lines of maintained code, not machine time. Changing embedding models means re-indexing everything — vectors from different models are not comparable.

Question 11

How do I know my benchmark is measuring anything?

Accepted Answer

Score a strategy that ignores the input. On one 20-question sample, answering the same three repositories every time scored 95% exact match while the real system scored 79%. Fixing the sampling method dropped that same strategy to 15%. The sample, not the system, was the problem.

Question 12

How do I know my agent benchmark isn't contaminated?

Accepted Answer

Diff the output against the reference. An empty diff — 239 lines matching line for line, filename included — is a smoke alarm, not an achievement. Blocking git commands does not help. The agent has a file-reading tool and the prompt usually hands over the repository path.

Question 13

Is ground truth mined from git history reliable?

Accepted Answer

It records what one person changed once, not what needs changing now. Of 117 mined tickets, 14 had labels larger than five files, 21 pointed at renamed paths, and 31 were already fully deployed. Still worth doing — hand-labelling a hundred examples is a week most people will not spend. Just know which way it leans.

Question 14

My agent is inconsistent — is that variance or a bad rule?

Accepted Answer

Run the set twice and record whether each case gave the same answer. Of 19 failures, 11 varied between runs and 8 were identically wrong every time — and only the second kind responds to prompt edits. Two runs distinguish 'always the same' from 'not always the same' and nothing finer. It is triage, not a statistical treatment.

Question 15

Should I benchmark against a copy of my index or the real one?

Accepted Answer

The real one. A copy stops representing production the moment anyone changes an ignore rule, and nothing tells you it has drifted. The exception is agent benchmarks, where the system can read the answer off disk. There you need an isolated clone at an older commit.

Question 16

What fills up an agent's context window?

Accepted Answer

Tool output, not conversation. Every one of the ten largest entries in the transcript was a single command's result — a 1.2 MB result is roughly 300,000 tokens, close to a third of a million-token window. The cost is not paid once. The transcript is the context, and it is re-sent on every subsequent request.

Question 17

Is a long-running agent session good for context?

Accepted Answer

Only until compaction starts. A 80-day session had been compacted 50 times, every one triggered at the one-million-token ceiling. After fifty compactions what the session 'knew' was a summary of a summary. Written notes preserve more and cost nothing per request.

Question 18

Where should I instrument an agent pipeline?

Accepted Answer

Where the decisions happen, which is usually a subprocess. A tracing integration wired into the orchestration framework produced a full dashboard and zero traces from the three components that make choices. Framework SDKs integrate where the framework has hooks. That is the right layer for orchestration questions and one or two levels too high for judgement questions.

Question 19

How do I stop a session from hitting the context ceiling?

Accepted Answer

Make the number visible. A status line reading the transcript tail costs 63 ms on a 566 MB file and turns a rule you forget into a percentage that turns amber. The instrumentation is the cheapest of the three fixes and the least effective. Not dumping large output into the transcript matters more.

Question 20

Do I need an agent framework?

Accepted Answer

Depends which layer is yours. Durable state and retry flow are worth a library; provider abstraction is not, if you invoke one specific agent binary rather than calling a model. Message formats are worth matching as a specification without importing the library that popularised them.

Question 21

Which repository does this ticket belong to?

Accepted Answer

Unmeasured in the public literature. Every published issue-localisation benchmark hands the system a repository and asks which file to change — in a 28-repository organisation, choosing the repository is the step that wastes everything downstream if you get it wrong. Part of it is structurally unwinnable: near-duplicate repositories produce textually identical tickets, so the honest target is a shortlist of three rather than rank-1 accuracy.

Question 22

Why would a hybrid retrieval setup return vector-only results?

Accepted Answer

An early-termination shortcut skipped the lexical query whenever the top five vector results were closer than a distance threshold of 0.15. That threshold was tuned against an earlier embedding model; the newer one returned cosine similarities around 0.93, so the shortcut fired on every query checked. A similarity threshold is a property of the model's output distribution, not configuration that survives a model swap. Measured on eighteen queries from one corpus, found while investigating something else. The AND-joined lexical query was a second, independent cause on the same path.

Question 23

Why does the biggest repository always rank first?

Accepted Answer

Averaging the top three chunk scores is a max-of-k statistic, and max-of-k grows with sample size. Across 28 repositories, log(chunk count) correlated with mean rank at r = -0.784 — bigger repository, higher rank, for every query. Chunk counts ranged from 629 to 145,971. Correlation over 28 repositories: enough for the sign and rough magnitude, not for the third decimal. One repository was excluded because it had been indexed with a different embedding model, putting its scores on another scale.

Question 24

Can score normalization remove collection-size bias?

Accepted Answer

No — three standard corrections all did worse than leaving it alone. Per-collection z-scores dropped first place from 28% to 12% because small collections have narrow distributions. Scoring the margin moved the correct repository's mean rank from 3.9 to 16.4. ReDDE-style counting has the size effect by design. The unit was wrong, not the comparison. One corpus, one embedding model, 90 balanced queries over 16 repositories.

Question 25

Do richer descriptions make a description index rank better?

Accepted Answer

Only when what you add is unique to one document. Measured one change at a time into the same directory, on 90 tickets: descriptions as they ship take first place 47 times, adding a hand-written glossary of shared product names drops it to 44, LLM-generated tags to 39, and filtering those tags to the rare ones only recovers to 42. IDF discounts common terms at query time; in a dense index the term has already shaped the vector, so the filtering has to happen when you write the document. Five builds in one sitting. An earlier report of this put the glossary cost at four times the real figure by comparing a build carrying glossary and tags against one carrying neither. Absolute counts are specific to the directory the index was built in.

Question 26

Why would a perfect keyword match return nothing at all?

Accepted Answer

Because the ceiling can sit below the floor. A document found only by the lexical path maxes out at its fusion weight of 0.35; multiplied by a retrieval weight of 0.58 that is 0.2030, against a candidate cutoff of 0.25. A title matching the query exactly could not clear it. No amount of better retrieval moves a number that is capped below the threshold. One production search pipeline. The arithmetic is specific to that weighting, but the check is general: multiply each path's maximum reachable score through the pipeline and compare it to every cutoff downstream.

Question 27

Should I fuse rankers by rank or by score?

Accepted Answer

It depends on whether your scores separate, and you can measure that in one query. Where per-item scores clustered inside 0.001, ordering was the only trustworthy signal and rank fusion helped. Where the candidate band was 0.048 to 0.094 wide, converting to ranks threw the separation away — reciprocal rank fusion cut rank-1 hits from 698 to 672 even though both paths covered the same corpus. Symmetric coverage is necessary for rank fusion, not sufficient. Two systems, two spreads. The threshold between 'clustered' and 'separated' is not established — these are 0.001 and roughly 0.05, with nothing measured in between.

Question 28

How much of a zero-result rate is actually a search defect?

Accepted Answer

On one customer-facing search, 8.7% of zero-result volume. Decomposing 231 query kinds that returned nothing: a sellable product existed and search missed it for 8.7%, the product existed but was sold out for 8.3%, and for 83.0% no matching product had ever been in the catalogue. Query kinds and volume disagree — 209 of 231 kinds is 90.5% by count but 83.0% by volume. One catalogue, measured twice a day apart. The first two slices moved between runs as inventory rotated (5.2/11.9 became 8.7/8.3); only the 83.0% held. Treat the split as a snapshot with a date on it.

Question 29

Why do inventory estimates from a top-N sample come out so high?

Accepted Answer

Because the head of a skewed distribution is not the distribution. Sampling 25 product codes chosen by search volume, descending, overestimated reachable inventory by 6.4× to 13.6×. A random sample of 60 gave a median of 28 against a mean of 58, with 15% returning zero. A separate expansion estimate drawn the same way was 22.7× over. One catalogue, one supplier API. The random sample was 60 codes, which fixes the sampling bias but not the small-sample error.

Question 30

Why would a GPU path be slower than doing nothing?

Accepted Answer

Because it was not the GPU path. The binary in PATH had been compiled without GPU support, and the device constructor fell back to CPU without a log line. On the same 1,840-chunk corpus: 306s on the silent fallback, 130s on the default runtime, 51s on a binary actually built with GPU support. The configured-for-speed path was two and a half times slower than doing nothing. One machine, one corpus, one library. The pattern is the fallback being silent, not the specific numbers.

Question 31

How do I know a component I configured is actually running?

Accepted Answer

Find a number that is different when it runs and when it does not, and print it where someone will see it. A green pipeline proves nothing — every command in these failures succeeded and every output was well-formed. Device name in the startup line, model name in the index metadata checked against what was requested, and one line asking whether all indexes report the same model. These are checks I added after being caught three times in one week, not a scheme validated against future failures.

Question 32

Is a bug report with a correct diagnosis enough to act on?

Accepted Answer

Not on its own. A reported routing failure came with a correct mechanism — a rare internal nickname that dense retrieval averages away — and turning on the lexical ranker moved that ticket from 9th to 1st. Run against all 90 tickets, the same switch dropped answer-in-top-3 from 80 to 68 and cost 18 tickets their first place to gain 12. The case was strong evidence for a claim nobody was making. One router, two gold sets (90 balanced tickets and 10 real ones). Per-query lexical contributions were not instrumented, so why any individual ticket moved is unmeasured.

Question 33

Does a small net change in a ranking metric mean nothing moved?

Accepted Answer

No — a net can be two large opposite movements cancelling. Top-3 fell by twelve, but 53 of 90 tickets changed rank: 34 worse, 19 better. First place moved for 30 of them, 18 lost and 12 gained. Reading the summary row as noise on 90 samples would have been wrong in both directions at once. Sixteen of the 34 regressions were a single rank. The distinction needs per-case ranks, which no aggregate row carries.

Question 34

What does a silent CPU fallback cost when you asked for the GPU?

Accepted Answer

Six times the wall clock, and worse than doing nothing. On 1,840 chunks: 306s with the GPU requested on a binary built without GPU support, 130s on the default runtime, 51s with the GPU actually available. The deliberately configured fast path was two and a half times slower than the default one, silently, for weeks. One machine, one corpus, one library. The fallback was a single `unwrap_or(Device::Cpu)`; the fix makes an unavailable device an error rather than a downgrade.

Question 35

Why does a place name with a facility suffix return nothing?

Accepted Answer

Because the exact string is not in the corpus. A hotel listing says the place name; it never says the place name plus "station". Queries ending in a facility suffix were 4.3% of typed query kinds, and 29.6% of that class returned zero results. Retrying with the suffix removed took that to 5.4% and raised place-name-at-rank-1 from 49.2% to 67.4%. Korean, where the suffix attaches with no space and is one or three characters. The retry is gated: it only replaces the original when it does not return fewer results, which took regressions from 0.4% to 0.0%.

Question 36

Can you evaluate a list of query-rewriting rules as one change?

Accepted Answer

Not safely. Measured together, nine suffixes give +32% and every one looks worth keeping. Measured one at a time, two of them regress almost exactly as much as they improve — 0.8% against 0.6%, and 1.0% against 1.0% — because stripping them turns a place into a different place or into a non-word. Their combined contribution is +0.1% and −0.0%, invisible in the total. One corpus, one language. The rejected pair was found by measurement, not by any linguistic property that separates them from the seven that worked.

Question 37

Is a tie in an A/B test evidence that the two options are equal?

Accepted Answer

No — it can be evidence the test cannot tell them apart. A parameter swept over 400 queries scored 283, 283, 283 and 284 rank-1 hits across a four-fold change, with an identical 1.92% zero-result rate. Read as "no difference" and shipped as "the cheaper one is free," the same change cost seven answers when re-run over 1,500 queries — six improvements against eight regressions. One search pipeline. That the losses fall in the tail is inferred from where the top-400 set sits in the distribution, not from opening the seven lost queries.

Question 38

What should I check before running a parameter sweep?

Accepted Answer

That changing the parameter changes the output. The recommendation service cached results per query, so the second arm of every comparison would have replayed the first arm's answers — eighteen combinations returning identical numbers, read as "none of these axes matter." The inverse also happens: two arms agreed to the decimal because both cleared the same arithmetic cutoff, which was real. The all-identical outcome was not run to confirm; the cause was clear enough that reproducing it had no value.

Question 39

Can the same files indexed in a different folder produce different scores?

Accepted Answer

Yes, by about one part in ten thousand, and that was enough. Twenty-eight byte-identical description files indexed under two paths returned cosine similarities differing at the fourth decimal — 0.915857 against 0.916047 for one document — which reordered 8 of 15 queries. Re-indexing the same folder reproduces every score to seven decimal places, so the pipeline is not simply non-deterministic. Found the next morning: the first breadcrumb in the chunk's context field is the absolute file path, so every chunk was embedded with a prefix naming the checkout. Fixing it makes the two indexes agree to eight decimals (0.9052133 against 0.9052133, from 0.89013606 against 0.89162624). A second bug sat underneath — the embedding cache keys on content while the embedded text includes the context, so two files with the same body in different directories can be served each other's vector.

Question 40

Should each configuration in a comparison get its own working directory?

Accepted Answer

Not if rebuilding is part of what you are measuring. Giving four configurations four folders put a per-folder displacement worth about two points inside a treatment effect worth two to four. The messier-looking version — one directory, contents swapped between runs — is the controlled one, because then the only thing that changes is what you changed. Specific to pipelines where the artefact is rebuilt per arm. The check is to build one configuration twice in both places and diff the rankings, not the summary counts.

Question 41

How do I know a trace field is measured and not computed?

Accepted Answer

Check two things it cannot fake. If the sum of span durations equals the wall-clock window to three significant figures — 881s against 882s, 538 against 538, 524 against 523 — those two quantities are not independent. And if a thousand values land on exactly one number, that is a constant, not a measurement. Both came from spans whose end was pinned to the start of the next call. One codebase, recognised after the fact. Turning the two signals into a rule and running it against other traces has not been done.

Question 42

Does an agent run its tool calls in parallel?

Accepted Answer

Measure the gap between consecutive call start times, which is the one value nothing derives. Across 1,093 consecutive gaps in one day's traces: 5% under 0.3s (dispatched together), 39% between 0.3 and 3s, 57% at 3s or more, median 3.94s. Span durations cannot answer this if the span end is computed from the next call's start. The 5% is a floor, not a rate — the 0.3s cutoff is a judgement about what counts as the same dispatch, and the 39% middle band is genuinely ambiguous. A `gap < 3600s` filter is applied; without it one trace-boundary artefact reports 1.7 billion seconds. An earlier version of this post carried 1,025/49/383/593 and could not be re-derived.

Question 43

My agent runs are slow and inconsistent — what should I target?

Accepted Answer

The spread, not the median. Telling a review agent which search command to use moved the median from 13.1 to 11.4 minutes and the spread from 8.8 to 1.0, worst case 18.0 to 12.2. The tool-call count did not move: 70 against 72. The agent investigates as much as before; one investigation got cheaper. Three interleaved pairs. That an uninstructed run is a coin flip on tool choice is inferred from one control run that picked the fast command by itself.

Question 44

Will putting more in the prompt reduce an agent's tool calls?

Accepted Answer

Not if the step was optional. Telling the agent it already held the contributing guide went 73 calls to 72; precomputing the caller list it would have grepped for went 57 to 58. Injecting the diff body — a step without which there is no review at all — went 83 to 69. Removing a mandatory fetch is what helps. One pair per condition, same sample replayed. An alternative explanation is that the amount to investigate is set by diff size and is independent of context; the experiment separating those was not run.

Question 45

Does confirming a fix in the diff verify it?

Accepted Answer

No. Two correct fixes to a candidate-folding bug both left the output unchanged — `-m 5 --per-file 1` returned 3 distinct files before and after each. A third layer capped the candidate pool at exactly the number of results requested, so folding had nothing to fold. Found because a second party re-ran the command and reported the old number, not by any test. The flag had been parsed, documented, and inert on the JSON path since at least June 1 — possibly longer, as that history predates the fork.

Question 46

Is a bracketed prefix in a query title noise?

Accepted Answer

No, it was the strongest signal. Stripping it dropped top-3 hits from 26/29 to 8/29 — the prefix identified which system the ticket belonged to. The hypothesis came from a single query where stripping looked better, with a plausible mechanism attached. It reversed under the full set.

Question 47

Does a wider candidate pool cost wall clock?

Accepted Answer

Not measurably. Blocked runs said the wider pool cost 81ms; six alternating pairs reversed the sign, with the wider pool faster by 10ms on medians and 16ms on means, and the two ranges overlapping across 815-855ms. Three separate errors sat on this benchmark and none of them changed that conclusion: the first measurement was blocked rather than interleaved; the median helper sorted in place, destroying the pairing the design depended on; and that helper returned the upper-middle value of an even-length list rather than the median, so the reported 867/844 were neither means nor medians. The last two were found by a second party checking the summary against the raw values, not by rerunning anything.

Question 48

What does a containment scoring rule reward?

Accepted Answer

Guessing wide. Under expected-subset-of-answered, a router scored 10/10 with 11 correct and 7 wrong repositories named. The same data gives 61% precision and 50% exact match. Adding a repository to an answer can never lower a containment score, so the rule cannot separate a good router from one that answers everything. Ten tickets, twelve distinct repositories.

Question 49

What does a fixed answer score without reading the input?

Accepted Answer

40% on the same ten tickets, by naming three constant repositories. Four constant names score 50%. A separate router's gold set was worse: 19 of its top 20 tickets shared one combination, so that combination alone scores 95%. Computed from the raw expected/answered pairs, not from the reported summary. The two failures differ in kind, one in the scoring rule and one in the evaluation set, and neither is visible from the score. Answering every repository scores 4.1% precision (11/270), where 270 counts the 27 repositories that appear as expected answers across the full ticket set, not the 12 that appear in these ten; 30 are registered in configuration and 29 active, so three defensible denominators exist and they differ by more than 2x.

Question 50

Why did the same change score 0 on one set and +4 on another?

Accepted Answer

Headroom. The first set was at 28/29 on top-5, leaving one point available, so no change could have shown more than +1. The second was at 85/90, leaving five, and the change took four of them. Both sets measure the same pipeline with the same metric. The zero was not evidence the change did nothing, only that the instrument was nearly full.

Question 51

Should you pick the metric with the most headroom?

Accepted Answer

No. Rank 1 had 49 slots free against top-5's five, but nothing downstream consumes rank 1 - the agent searches the whole candidate set, so reordering within it changes no behavior. The metric must have slots left and measure what the consumer depends on. Here that intersection was top 3, at 76/90. The intersection can be empty. A mature system whose consumer-facing metric sits at ceiling has no honest way to show incremental gains on it, and reaching for a looser metric that still moves is the failure this describes.

Question 52

How compressed are this model's similarity scores?

Accepted Answer

Twenty-nine unrelated documents scored between 0.9021 and 0.9277 against one query, a band 0.0256 wide. The mean of a few thousand unit-length vectors has norm 0.926 to 0.945 across 23 indexes, and the means of different indexes agree with each other at cosine 0.945 to 0.999. That last figure is a property of the model rather than of what was indexed: the indexes hold completely different codebases and their centres of mass point almost the same way.

Question 53

Does centering the embeddings improve ranking?

Accepted Answer

No. It widened the score band by 15.5x, from 0.0256 to 0.3976, and rank 1 went from 49/90 to 47/90 with MRR from 0.6949 to 0.6848. Sixteen queries moved, seven up and nine down. Only this model, this corpus, and 29 documents scored exhaustively rather than reranked. Centering is one endpoint of the soft-ZCA family and the rest is untested. Published work reports the same direction for contrastively fine-tuned models, but that agreement is deliberately not counted as support.

Question 54

Does restoring the suppressed lexical half improve results?

Accepted Answer

Three measurements gave three answers. On 29 short descriptions, turning the shortcut off was worse (rank 1 49/90 to 46/90). On one code index with a hand-built answer key it was decisively better (top 1 9/42 to 15/42, MRR 0.363 to 0.479). On a second code index in another language it did nothing (top 1 7/40 to 8/40, and 10 of 18 changed tickets got worse). The second repository did not replicate because the comparison there was mostly measuring recall: the answer file is absent from the candidate set entirely on 15 of 40 tickets, against 4 of 42 on the first. Those tickets cannot be won by any ranker. The obvious explanation, that Korean queries do not match a non-Korean codebase, was measured and rejected - the backend holds more Korean than the front end, 9.4% of chunks against 7.2%. Whether lexical retrieval helps there remains unmeasured.

Question 55

How often does a cosine threshold fire in an anisotropic space?

Accepted Answer

Always, once the model moved. The 0.85 cutoff was chosen against a 768-dimensional embedding model where it selected something; swapping to a multilingual model put every score near 0.93 and the condition became true. It fired 270 times out of 270 across three indexes at 90 queries each, so the fusion it gates has never executed here. No code changed when the condition died, so it appears in no diff, review or changelog. Firing rate was first read as 0/90 because the JSON output mode suppresses the logs it was counted from; the contradiction that exposed it was 0% firing alongside 85 of 90 queries changing when the shortcut was disabled.

Question 56

When does lexical retrieval earn its place next to embeddings?

Accepted Answer

On rare tokens. Reading eight queries by eye on a real code index, enabling lexical retrieval clearly won twice and clearly lost once. Both wins were queries naming a rare proper noun that the embedding smoothed away; the loss promoted generated API schema and end-to-end test specs over the component that handled the query. Eight cases judged by a person on one repository, not a measurement. Three of the five draws were the same generated schema file trading places with itself at different offsets, meaning both settings were wrong. It is what you do when no answer key exists and the alternative is deciding on nothing.

Question 57

Does making a parameter configurable count as fixing it?

Accepted Answer

No, and it is hard to tell apart. The commit that diagnosed the always-firing threshold traced it to a model swap, wrote "This is a value you have to revisit when you change the model", added an environment variable to override it, and left the default unchanged. Production behaviour was identical before and after. Measuring the two positions a day later showed the untouched default was the better one, so the outcome was correct. A default arrived at by deferring the decision is correct next time only by accident, and the commit history records a deferral as a decision.

Question 58

Can a threshold be a correct value and still never fire?

Accepted Answer

Yes, by reading the wrong unit. A 0.02 low-confidence cutoff was written for fusion scores, which top out near 0.05. The upstream shortcut always fires, so the pipeline runs vector-only and returns raw cosine instead, and the check receives 0.93. It was under 0.02 on 0 of 270 queries, with the lowest observed score 0.9104, forty-five times the cutoff. The zero has a denominator: all 270 queries produced a score. The same investigation had earlier produced a zero that meant the logging was off, so the count of values present is reported alongside it. Nothing in the file containing 0.02 has ever been wrong; what killed it is a branch in a different file, connected by one runtime decision.

Question 59

What does "this cannot be measured" usually mean?

Accepted Answer

Often, that nobody has priced it. A comparison declared unmeasurable for lack of a file-level answer key was measured the same evening, once someone built the key from issue titles and ticket-linked commits, and the result reversed the conclusion. The tell was available beforehand and had already been written down: the comparison that was affordable was also the one that mattered least. Cheap and important coming apart that neatly is the shape of a budget constraint, not of reality.

Question 60

Did the hand-built answer key leak its answers into the queries?

Accepted Answer

Yes on four of forty-two tickets, and it changed nothing. Those four score 4/4 at rank 1 with MRR 1.000 under both settings, contributing zero to the difference. The whole gain comes from the other 38 plain-language queries, where rank 1 goes from 5 to 11 and MRR from 0.296 to 0.424. The leak was not a later title edit, which is what the change history was checked for and cleared. Those tickets were filed after the investigation with its conclusion as the title, and the same description listed the answer file paths, so no edit history could show it. Three things are true together: the leak is real, it inflates absolute scores, and it does not touch the A/B result.

Question 61

What happens when you keep paying for a better measurement?

Accepted Answer

The answer can keep changing. The same question about one threshold produced three different results in two days at three price points, and the variable separating them was which index was affordable at the time, not anything about the threshold. This is not an argument for measuring less; the first and cheapest result would otherwise still be standing as the conclusion. It is an argument against the moment after a number lands, when stopping feels like rigour rather than the end of a budget. The default was left unchanged, not because leaving it was shown correct but because nothing has been shown.

Question 62

Were the untested fusion constants actually wrong?

Accepted Answer

No. A parameter sweep against both answer keys puts the value the engine uses, k=20, at the optimum on one corpus and within 0.002 of it on the other. The constant that had never executed is the right one as far as two corpora can say, and finding that out took three seconds with a standard library. Two corpora, and 'within 0.002' is a statement that the difference is not measurable here rather than that the values are identical.

Question 63

Would using standard evaluation tooling have prevented these mistakes?

Accepted Answer

One of four. Re-running everything through a retrieval evaluation library reproduces the hand-built counts exactly and supplies, as named arguments, the significance test and the parameter sweep that took a day of argument. But three of the four errors were in the engine: a comparison moving two variables at once, a threshold that changed which scale the next one read, and a counter reading a suppressed log as a missing event. No evaluation library observes any of those three. The one it would have caught is speaking with more confidence than the arithmetic supported, which was also the cheapest to fix.

Question 64

Is a calculated number safer than a measured one?

Accepted Answer

No, it is more dangerous. A saving computed as round trips times 14.3 seconds each came to 85 hours and was wrong by a factor of thirty: the round trip is almost entirely model thinking and the tool inside it takes 0.2 seconds. The real figure was 48 minutes. A measurement carries visible noise that argues with you; a product of two defensible inputs arrives with clean edges and no error bars. The four other errors in the same investigation were measurement errors and eventually announced themselves.