Topic
inference
2 posts
- Inference7 min
The flag was set and the feature was not there
GPU not being used, silently: the variable that turns on GPU embedding was set, but the binary was built without GPU support, so it ran on CPU. 306 seconds, not 51.
- Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.