Batch size is the first knob anyone reaches for on a GPU. More work per dispatch, better occupancy, fewer launches. I set mine to 256 and moved on.
When I finally swept it, the smallest batch I tried was the fastest by a factor of 1.9.
Same 1,024 chunks, same device, same weights, same precision. Only the batch size changed, and the throughput fell by half as it grew.
The padding column is the answer
A batch pads every sequence up to the length of its longest member. That is unavoidable, since the tensor has to be rectangular.
What changes with batch size is how long the longest member is likely to be. Draw four chunks and the longest might be 400 tokens. Draw 256 and you will almost certainly catch one near the model's 512-token limit, and now all 256 are padded to that.
So the corpus is fixed at 1,024 chunks, and the amount of computation is not:
That extra 17% is padding. It gets embedded, multiplied by an attention mask that zeroes it, and thrown away. You pay full price for arithmetic whose result is discarded by construction.
The effect compounds with variance. A corpus of uniformly sized chunks barely notices batch size. Mine is source code, where a chunk might be a three-line helper or a 300-line component, so the maximum in a batch grows quickly with batch size and the padding grows with it.
Bandwidth is the rest of it
Padding does not explain all of the 1.9×, and the remainder is worth naming because it points in the opposite direction from usual advice.
Large batches are the right answer when you are compute-bound, when the arithmetic dominates and you want the device saturated. This model is small, 384 dimensions, a few hundred megabytes of weights. At these sizes the run is bandwidth-bound: most of the time goes to moving weights and activations rather than multiplying them. A larger batch moves more activation data through the same memory bus and buys back less compute than it costs in traffic.
You can see this in an unrelated number from the same investigation. Half precision, which halves the bytes moved, gave 19% at batch 256 and about 3% at batch 4. If the run were compute-bound, precision would help roughly equally at both. It helps where there is more data in flight, which is the definition of bandwidth-bound.
That is also the trap: someone benchmarks f16 at batch 256, sees 19%, and concludes f16 is worth it. At the batch size that actually turns out to be fastest, it is worth almost nothing. The optimisations are not independent, and measuring them one at a time on a default batch size produces confident wrong recommendations.
Why I had 256 in the first place
I did not choose 256. It was the library default, and defaults for batch size in embedding libraries tend to assume server hardware processing uniform records: a queue of user queries, a table of product descriptions, things that are roughly the same length as each other.
Source code is not that. The chunker splits on syntactic boundaries, so a chunk is whatever a function or a class happens to be. In this corpus that ranges from about 40 bytes to the truncation limit, and the distribution has a long tail of very large chunks that are exactly what sets the padding ceiling for any batch containing them.
There's a general version of this: a default encodes assumptions about the input, and library authors could not have known yours. The batch size default was written by someone whose inputs were more uniform than mine. Nothing about that is a mistake on their part, and nothing about my having kept it was reasoning.
The same was true of my padding strategy, my precision, and, as I found out later, the reranker model. Four defaults, all sensible for someone, none chosen for this corpus.
Where it stops helping
Batch 4 was the best of the values I tried, but smaller is not universally better and I want to be clear about where the curve turns.
Below some point, per-dispatch overhead starts to dominate: each kernel launch has a fixed cost, and at batch 1 you pay it once per chunk with no parallelism to amortise it against. I did not sweep below 4 carefully. I checked that 1 and 2 were not obviously better and stopped, because I had a result that was already 1.9× the default and other things to do.
So "4" is not a magic number. What generalises is the shape: on a small model with variable-length inputs, the optimum is far lower than the defaults suggest, and you find it by sweeping rather than by reasoning about occupancy.
If you take a rule from this, take this one: sweep three orders of magnitude before trusting any batch size. Four, sixty-four, two hundred and fifty-six took fifteen minutes and moved the number by a factor of two.
Sorting helps, and helps differently at each size
Grouping similar-length sequences before batching reduces padding, since the longest member of a batch is closer to the rest. The effect is much larger at big batch sizes, because that is where the padding was:
Sorting recovers a large fraction of what a big batch loses. It does not recover all of it, since sorted batch 256 (34.2) is still well behind unsorted batch 4 (54.1), because the bandwidth cost is unaffected by ordering.
Two independent effects, and you need both measurements to tell them apart. Padding responds to sorting; bandwidth responds to batch size.
Sort by tokens, not by bytes
One detail that cost me a run. Sorting by byte length made padding worse on this corpus.
The code is commented in Korean. UTF-8 spends three bytes on most Hangul syllables while the tokenizer sees roughly one token per two syllables: long in bytes, short in tokens. Identifiers do the reverse: getUserPreferenceByChannelCode is cheap in bytes and expensive in tokens.
So byte length and token length are only loosely correlated here, and sorting by the first scrambles the second. If your corpus is entirely English prose the two are close enough that it does not matter. If it is not, sort by the thing the model actually pads on.
Checking whether this applies to you
Three things determine whether any of this transfers, and all three are cheap to measure on your own corpus.
How variable are your input lengths. Sum the token counts and look at the spread: mean, median, maximum. If the maximum is close to the mean, batching costs you almost nothing in padding and you can stop reading. If the maximum is five times the median, as mine is, padding is the dominant term.
Whether you are bandwidth-bound. Halve the precision and measure. A large improvement means data movement is your limit and large batches will hurt; a small one means you are compute-bound and the usual batching advice applies.
What your padded token count actually is. Not the number of chunks, but the number of tokens after padding, which is what the device computes. If your stack does not report it, sum the padded sequence lengths per batch. The gap between that and the unpadded total is your waste, expressed in the only unit that matters.
I ran none of these before setting a batch size, and all three afterwards while trying to explain a result that made no sense to me.
What I'd do on a new setup
Sweep the batch size before assuming. Three values (small, default, large) is fifteen minutes and it told me the default was 1.9× off. I had never questioned 256 because it looked like a sensible default, which it is, for a different shape of workload.
Instrument padded tokens, not just wall clock. The token count is what turned "batch 4 is faster, oddly" into an explanation. Most inference stacks can report it; if yours cannot, summing sequence lengths after padding is a few lines.
Measure combinations. Batch size, sort order and precision interact strongly here. Each measured alone gives a defensible recommendation that is wrong in combination.
Check the variance of your input lengths. All of this is proportional to how much your sequence lengths differ. A corpus of uniform records will not show it; a corpus of source code shows it dramatically.
One caveat about scope
This is a small-model result and I would not carry it to a large one. Large models at these batch sizes are compute-bound, where the usual advice about filling the device holds and padding is a smaller share of a bigger number.
The tell for which regime you are in is the precision measurement. If halving precision buys you a lot, you are moving more bytes than you are multiplying, and batch size will behave the way it did here. If it buys you almost nothing, ignore this post and turn the batch up.