Category
Inference
Local inference, Metal/GPU, and batch tuning.
5 published
- Inference7 min
The flag was set and the feature was not there
GPU not being used, silently: the variable that turns on GPU embedding was set, but the binary was built without GPU support, so it ran on CPU. 306 seconds, not 51.
- Inference10 min
What running embeddings locally actually costs
Local embeddings at scale: 672,000 chunks across 28 repositories, indexed on a laptop. The bill was not money and not really time either.
- Inference9 min
Bigger batches made my GPU slower
Embedding batch size on a GPU: batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.