Topic
apple-silicon
3 posts
- Inference10 min
What running embeddings locally actually costs
Local embeddings at scale: 672,000 chunks across 28 repositories, indexed on a laptop. The bill was not money and not really time either.
- Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.