Years of suboptimal mannequin coaching?
When fine-tuning massive language fashions (LLMs) regionally, utilizing massive batch sizes is usually impractical because of their substantial GPU reminiscence consumption. To beat this limitation, a method known as gradient accumulation is usually used to simulate bigger batch sizes. As a substitute of updating the mannequin weights after processing every batch, gradient accumulation includes summing the gradients over a number of smaller mini-batches. The mannequin weights are up to date solely after a predetermined variety of these mini-batches have been processed. This methodology successfully mimics coaching with a bigger batch dimension with out the reminiscence overhead usually related to it.
As an example, setting a mini-batch dimension of 1 and accumulating gradients over 32 mini-batches must be equal to coaching with a full batch dimension of 32. Nevertheless, I found that gradient accumulation usually ends in considerably degraded efficiency in comparison with coaching with bigger precise batch sizes with well-liked deep-learning frameworks like Transformers.
After sharing this concern on X and Reddit, Daniel Han from Unsloth AI replicated the issue. He discovered that it was affecting not solely gradient accumulation but additionally multi-GPU setups. In such…