Four wrong measurements, in order
The GPU investigation
An afternoon's work that took a week, because the first three measurements were of things I had set up badly. Read in order: the mistakes are more transferable than the result, and each one looked reasonable while I was making it.
5 posts
Where this stands
- Held to be true
- Every apparent GPU problem here was a plumbing problem, not a capacity one, and the two want opposite responses. Twice the accelerator was idle while something upstream fed it badly, and once it was not running at all despite being configured.
- What would break it
- A case on this hardware where the device is saturated and more parallelism is the answer.
- Still open
- Whether any of this survives a different runtime. The findings are about one accelerator path on one platform, and the fixes were mostly going around a dependency rather than through it.
- 1Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.
- 2Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.
- 3Inference9 min
Bigger batches made my GPU slower
Embedding batch size on a GPU: batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- 4Inference10 min
What running embeddings locally actually costs
Local embeddings at scale: 672,000 chunks across 28 repositories, indexed on a laptop. The bill was not money and not really time either.
- 5Inference7 min
The flag was set and the feature was not there
GPU not being used, silently: the variable that turns on GPU embedding was set, but the binary was built without GPU support, so it ran on CPU. 306 seconds, not 51.