Topic
performance
2 posts
- Inference9 min
Bigger batches made my GPU slower
Batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Inference8 min
The GPU was not slow, it was waiting
CoreML made indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.