Estimating GPU reminiscence for deploying the newest open-source LLMs
For those who’re like me, you in all probability get excited concerning the newest and best open-source LLMs — from fashions like Llama 3 to the extra compact Phi-3 Mini. However earlier than you soar into deploying your language mannequin, there’s one essential issue you want to plan for: GPU reminiscence. Misjudge this, and your shiny new net app would possibly choke, run sluggishly, or rack up hefty cloud payments. To make issues simpler, I clarify to you what’s quantization, and I’ve ready for you a GPU Reminiscence Planning Cheat Sheet in 2024— a helpful abstract of the newest open-source LLMs available on the market and what you want to know earlier than deployment.
When deploying LLMs, guessing how a lot GPU reminiscence you want is dangerous. Too little, and your mannequin crashes. An excessive amount of, and also you’re burning cash for no cause.
Understanding these reminiscence necessities upfront is like understanding how a lot baggage you may slot in your automotive earlier than a street journey — it saves complications and retains issues environment friendly.