Understanding how a lot reminiscence you must serve a VLM
vLLM is at the moment one of many quickest inference engines for big language fashions (LLMs). It helps a variety of mannequin architectures and quantization strategies.
vLLM additionally helps vision-language fashions (VLMs) with multimodal inputs containing each pictures and textual content prompts. As an example, vLLM can now serve fashions like Phi-3.5 Imaginative and prescient and Pixtral, which excel at duties corresponding to picture captioning, optical character recognition (OCR), and visible query answering (VQA).
On this article, I’ll present you learn how to use VLMs with vLLM, specializing in key parameters that influence reminiscence consumption. We’ll see why VLMs devour way more reminiscence than commonplace LLMs. We’ll use Phi-3.5 Imaginative and prescient and Pixtral as case research for a multimodal software that processes prompts containing textual content and pictures.
The code for operating Phi-3.5 Imaginative and prescient and Pixtral with vLLM is offered on this pocket book:
In transformer fashions, producing textual content token by token is sluggish as a result of every prediction depends upon all earlier tokens…