Topic
metal
2 posts
- Inference9 min
Bigger batches made my GPU slower
Batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
The GPU looked slower than the CPU in every early run. All three of those runs were measuring something I had set up badly.