Topic
performance
2 posts
- Inference9 min
Bigger batches made my GPU slower
Embedding batch size on a GPU: batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.