I index 28 repositories locally: 672,487 chunks, rebuilt whenever the code moves. No API calls, no per-token billing, everything on one machine.
People usually ask what that costs. The interesting answer is that the expensive part was none of the things I expected.
The dollar figure is zero and that is not the point
The obvious framing is API cost avoided. At current rates a corpus this size costs somewhere in the low tens of dollars to embed once. I rebuild constantly. Over a few months the arithmetic favours local by a wide margin.
But I did not choose local to save money. The corpus is proprietary source code, and sending all of it to an external service is a conversation with several people and a security review. Local was the default because the alternative had a process attached to it.
If that constraint does not apply to you, the money argument alone is weaker than it looks. A hosted embedding API is fast, requires no tuning, and has no failure mode where you spend an afternoon discovering that your reranker is English-only.
What it actually cost
Three things, in descending order of how much they surprised me.
A week of tuning, most of it wasted. I measured Metal wrong three times before it beat the CPU. Chunk sizes that did not match my corpus, an f16 crash that was my own dtype bug, sorting by bytes instead of tokens. None of that work exists if you call an API.
Two hundred lines of maintained code. A Metal backend, batching, sorted padding, a cache keyed per backend. It works and it is mine to keep working when the framework updates.
An afternoon on a dependency's padding strategy. The library pads each batch to its longest sequence, which makes CoreML recompile constantly. Understanding that took a day; the eventual answer was to route around the library entirely.
Against which the machine-time cost is almost nothing: 7.2 seconds of CPU for a full repository index that used to take 1,736. That number is the one I would put on a slide, and it is the least important one here.
The number that changed how the laptop feels
The wall clock halving is nice. The CPU column is what changed the experience of using the machine.
The ONNX path saturated every core for the duration. You could start an index and then not use the laptop, with editor lag, fans, and everything else queued behind it. The Metal path uses seven seconds of CPU across a two-and-a-half-minute run. The work happens on a device that was idle anyway. The rest of the machine does not notice.
For a batch job on a server that distinction is meaningless. For a tool that runs on the machine you are working on, it is most of the value. I would take the Metal path even if wall clock were identical.
The scale it holds at
Full rebuild of all 28 repositories, everything cold:
Five hours is not fast. It is also something I have done exactly twice (once when adopting the tool and once when changing embedding models) because everything else is incremental.
The steady state is what matters and it is much cheaper:
An index tracks file hashes, so a rebuild only touches what moved. The daily job runs at 04:00 and I have never noticed it. The five-hour number is a migration cost, not an operating cost, and conflating those makes local inference look far more expensive than it is.
Worth pricing separately: changing embedding models means re-indexing everything, because vectors from different models are not comparable. That is the five-hour operation, and it is the reason to be deliberate about model choice rather than trying things casually.
The lever I had not set
What indexing costs is decided by how many chunks there are, and how many chunks there are is decided by which files you let in. I had no upper bound on file size at all. That did not feel like a decision until one file turned it into one.
A 182 MB JSON fixture in one repository produced 106,170 chunks, 48% of that repository's index. A rebuild that normally took sixteen minutes had not finished after two and a half hours. Nothing in the output said why; the progress counter was simply moving very slowly through a file nobody would ever search for.
I dealt with it the first time by naming the file in an ignore list, which handles exactly the files you already know about. The job that runs at 04:00 does not know about the next one.
So before picking a threshold I measured the corpus, which took ten minutes and settled an argument I would otherwise have had with myself for an hour:
At a 1 MB limit, 34 files are excluded, 0.138%, and every one of them is generated: bundler output, tokenizer vocabularies, a font encoded as JavaScript, an SVG logo pasted inline into a component, a generated GraphQL schema. Not one hand-written file in 28 repositories comes near it.
The measurement is what lets me say that. Between p99 and p99.9 there is a factor of twelve with nothing human in it, so anywhere in that gap would have worked, but picking from the distribution means I can state what the limit excludes instead of hoping.
Then a second-order cost I did not expect. Excluding a file does not remove what it already put in the index. The deletion check asked whether a file was gone from disk, and these files are not gone from disk. They stopped being indexed, which is a different thing the code had no word for. Their chunks stayed, and the repository kept answering queries out of content it was no longer supposed to hold.
Fixing that surfaced an older one underneath it. A run that only deletes was not saving its file list, so the next run found the same chunks and deleted them again. Reproduced, the second, third and fourth runs each removed the same 1,001 chunks and each reported success.
Disk, since nobody mentions it
The storage cost is small but not zero, and it scales with a decision most people make without thinking.
Dimension count is the lever. I use a 384-dimensional model; the 768-dimensional alternative I benchmarked would double all of that, and it would double the arithmetic on every query too. The quality difference between them was inside the noise on my gold set, so the larger model would have cost twice the disk and twice the query work for nothing measurable.
One trap worth naming: the on-disk file does not shrink when you remove things. Rebuilding an index after excluding files leaves the old pages allocated, so a repository whose content dropped by 92% still occupied its original footprint until I deleted the index directory and built fresh. If disk matters to you, remove rather than rebuild.
There is also a model cache, a few hundred megabytes per model, kept locally. Trying four embedding models and three rerankers means holding all of them. My models directory is larger than most of the indexes it serves.
The failure modes are yours now
An underrated part of the trade: when a hosted API misbehaves, you file a ticket. When a local pipeline misbehaves, you are the one who has to notice.
The failures I hit were all silent. The reranker that was English-only produced worse rankings with no error. The embedding cache did not distinguish backends, so a rebuild handed back another model's vectors. Two "different" configurations produced byte-identical results. For an hour I believed I had discovered something. The incremental rebuild above reported success four times while removing the same chunks four times.
None of those raised an exception. All of them were caught by measurement, and only because I had a gold set to measure against.
That is the hidden line item. Running inference locally means owning a class of quiet failure that hosted services absorb for you, and the only defence is a way to check that the output is still good. Budget the gold set alongside the code. It is not optional infrastructure here, it is the thing that tells you the pipeline is still working.
Where the money would have gone instead
If I had used a hosted API, the costs would have moved rather than disappeared:
Per-rebuild billing. Incremental indexing means small daily charges, but a model change or a chunking change means re-embedding the whole corpus. Locally that is five hours of a machine that is on anyway; hosted, it is a line item every time I want to try something.
Latency on the query path. Search happens interactively and a network round trip per query is a different feel from a local model. I run queries through the CPU path deliberately, since for a single query the GPU's advantage disappears into model load time, and it is fast enough that search feels instant.
A dependency on someone else's availability. Minor, but code search that stops working when a provider has an incident is a worse tool than one that does not.
Against those, the hosted version buys you: no tuning, no maintained code, better models than you can run locally, and no afternoon spent learning what your library's padding strategy does to CoreML.
Would I do it again
For this corpus, yes. The privacy constraint made it the only option, and the operating cost turned out to be two minutes a day.
If the constraint were absent I would think harder. The honest accounting is not "free versus paid." It is a week of engineering and two hundred lines of ongoing maintenance, against a per-token bill and a network dependency. At my scale the local side wins. At a tenth of the scale, with no privacy requirement, I do not think it would.
What I would not do again is treat local inference as a performance decision. It is a control decision: you get to choose the model, the batching, the padding, and where the data goes, and you pay for that in the time it takes to make those choices correctly. Three of my four wrong turns were choices a hosted API would have made for me, correctly.