From LLaVA, Flamingo, to NVLM
Multi-modal LLM growth has been advancing quick in recent times.
Though the industrial multi-modal fashions like GPT-4v, GPT-4o, Gemini, and Claude 3.5 Sonnet are probably the most eye-catching performers nowadays, the open-source fashions corresponding to LLaVA, Llama 3-V, Qwen-VL have been steadily catching up by way of efficiency on public benchmarks.
Simply final month, Nvidia launched their open-source multi-modal LLM household referred to as NVLM. The household includes three architectures: a) decoder-based, b) cross-attention-based, and c) hybrid. The decoder-based mannequin takes each the picture and textual content tokens to a pre-trained LLM, such because the LLaVA mannequin. The cross-attention-based mannequin makes use of the picture token embeddings because the keys and values whereas utilizing the textual content token embeddings because the queries; because the consideration is calculated utilizing completely different sources, it’s referred to as “cross-attention” as within the authentic transformer decoder relatively than the self-attention as in decoder-only fashions. The hybrid structure is a singular design merging the decoder and cross-attention structure for the advantage of multi-modal reasoning, fewer coaching parameters, and taking high-resolution enter. The 72B decoder-based NVLM-D mannequin achieved a formidable efficiency, beating state-of-the-art open-source and industrial fashions on duties like pure picture understanding and OCR.
On this article, I’m going to stroll by means of the next issues:
- the dynamic high-resolution (DHR) imaginative and prescient encoder, which all of the NVLM fashions undertake
- the decoder-based mannequin, NVLM-D, in comparison with LLaVA
- the gated cross-attention mannequin, NVLM-X, in comparison with Flamingo
- the hybrid mannequin, NVLM-H
Ultimately, I’ll present the NVLM-D 72B efficiency. In comparison with state-of-the-art open-source and industrial fashions, the NVLM-D mannequin exhibits stability over text-based duties and superior efficiency on pure understanding and OCR duties.
