Topic
metal
2 posts
- Inference9 min
Bigger batches made my GPU slower
Embedding batch size on a GPU: batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.