Important developments in giant language fashions (LLMs) have impressed the event of multimodal giant language fashions (MLLMs). Early MLLM efforts, equivalent to LLaVA, MiniGPT-4, and InstructBLIP, show notable multimodal understanding capabilities. To combine LLMs into multimodal domains, these research explored projecting options from a pre-trained modality-specific encoder, equivalent to CLIP, into the enter house of LLMs, enabling multimodal understanding and reasoning throughout the transformer spine. Though there are numerous design selections for MLLMs, equivalent to imaginative and prescient encoders, characteristic alignment adapters, and datasets, the coaching for many of those fashions adheres to the autoregressive era paradigm, which has confirmed efficient for textual content era in LLMs. Regardless of their sturdy multimodal understanding capabilities, these fashions primarily deal with visible notion and lack the power to generate multimodal outputs past textual content.
Transformer fashions have demonstrated nice success in autoregressive modeling in pure language processing. Impressed by such progress, earlier research have immediately utilized the identical autoregressive modeling to be taught the dependency of picture pixels for picture and video era. As an example, VideoPoet employs a decoder-only transformer structure to synthesize high-quality movies from multimodal inputs. Extra just lately, LlamaGen has proven that a big language mannequin structure like Llama can autoregressively mannequin picture tokens, reaching first rate efficiency in class-conditional picture era.
On this article, we’ll talk about Present-O, a unified transformer that integrates multimodal understanding and era. In contrast to absolutely autoregressive fashions, Present-O unifies autoregressive and discrete diffusion modeling to adaptively deal with inputs and outputs of assorted and combined modalities. The unified mannequin flexibly helps a variety of vision-language duties, together with visible query answering, text-to-image era, text-guided inpainting/extrapolation, and mixed-modality era. Throughout varied benchmarks, Present-O demonstrates comparable or superior efficiency to present particular person fashions with an equal or bigger variety of parameters, highlighting its potential as a next-generation basis mannequin.
On this framework, the mannequin is tasked with predicting Gaussian noise added to the continual latent representations. In distinction, different fashions like D3PM, Masks-predict, ARDM, and MaskGIT use a discrete corruption course of as an alternative choice to Gaussian diffusion. Particularly, a picture is represented as a sequence of discrete tokens utilizing picture tokenizers, with every token related to a categorical label. The token-wise distribution is reworked right into a uniform distribution by way of a stochastic sampling course of. Throughout coaching, a portion of those tokens is randomly masked, and the mannequin is educated to foretell the unique values of the masked tokens. On this work, Present-O adopts discrete diffusion modeling for visible era.
Over the previous few years, vital developments have emerged within the two key pillars of multimodal intelligence: understanding and era. For multimodal understanding, Multimodal Giant Language Fashions (MLLMs) like LLaVA have demonstrated distinctive capabilities in vision-language duties equivalent to visible question-answering (VQA). For visible era, denoising diffusion probabilistic fashions (DDPMs) have revolutionized conventional generative paradigms, reaching unprecedented efficiency in text-to-image/video era.
Given these achievements in particular person fields, it’s pure to discover the potential of connecting them. Latest works have tried to assemble professional fashions from these two completely different domains to kind a unified system that may deal with each multimodal understanding and era. Nevertheless, present makes an attempt typically contain separate fashions for understanding and era. As an example, NExT-GPT employs a base language mannequin for multimodal understanding however requires a further pre-trained diffusion mannequin for picture era. This raises the query: can one single transformer deal with each multimodal understanding and era?
Not too long ago, Chameleon demonstrated that that is potential. Particularly, Chameleon permits the fusion of various modalities to generate each textual content and picture tokens by way of autoregressive modeling. Whereas it is smart to mannequin textual content tokens autoregressively, it’s much less clear whether or not modeling picture patches or pixels in the identical method is perfect. A key bottleneck of autoregressively predicting a picture is the big variety of sampling steps required, particularly when coping with larger decision photographs. Steady diffusion fashions have proven superior efficiency in visible era in comparison with autoregressive ones.
This leads us to discover whether or not a single transformer can combine each autoregressive and diffusion modeling. Present-O envisions a brand new paradigm the place textual content is represented as discrete tokens and modeled autoregressively, whereas steady picture pixels are modeled utilizing denoising diffusion. Nevertheless, integrating these two distinct methods right into a single community is non-trivial as a result of variations between discrete textual content tokens and steady picture representations. Moreover, diffusion fashions sometimes depend on two distinct fashions: a textual content encoder and a denoising community.
To handle this, Present-O introduces a novel unified mannequin able to dealing with each multimodal understanding and era duties utilizing combined autoregressive and diffusion modeling. Present-O is constructed upon a pre-trained LLM and leverages its autoregressive modeling capabilities for text-based reasoning. Impressed by different works, Present-O employs discrete denoising diffusion to mannequin picture tokens as an alternative of steady representations. Furthermore, Present-O inherently encodes textual content conditional info, eliminating the necessity for extra textual content encoders. By using textual content and picture tokenizers, Present-O can course of numerous enter information and duties, offering solutions autoregressively for vision-language duties and producing photographs utilizing discrete denoising diffusion.
Present-O demonstrates comparable, and in some instances higher, efficiency than particular person fashions with an equal or bigger variety of parameters throughout varied benchmarks. In contrast to autoregressive picture era, the Present-O framework requires about 20 instances fewer sampling steps, making it inherently sooner. Moreover, the Present-O framework helps downstream purposes like text-guided inpainting and extrapolation with out requiring fine-tuning, as demonstrated within the following picture.
Present-O additionally has the potential for mixed-modality era, equivalent to interleaved video keyframe era with textual content descriptions, exhibiting promise for long-form video era. Moreover, the Present-O framework investigates the affect of discrete and steady picture representations on multimodal understanding, providing insights for future unified mannequin designs.
The next determine presents a comparability of mannequin traits between the Present-O framework and present strategies throughout varied domains. Present-O stands out as a unified mannequin that integrates superior methods for each multimodal understanding and era.
In abstract, the primary contributions of this paper are as follows:
- Present-O is a unified mannequin that integrates multimodal understanding and era utilizing a single transformer.
- Present-O unifies autoregressive and discrete diffusion modeling inside one transformer, dealing with each textual content and pictures successfully.
- The Present-O framework outperforms or matches particular person baseline fashions with equal or bigger parameters throughout multimodal understanding and era benchmarks.
- Present-O helps downstream purposes like text-based inpainting and extrapolation with out fine-tuning and demonstrates potential for mixed-modality era.
- Present-O explores the affect of several types of representations, offering worthwhile insights for enhancing multimodal understanding in unified fashions.
Lately, an rising variety of research have targeted on unified multimodal language fashions able to each comprehension and era. Some efforts use steady representations interleaved with textual content tokens for autoregressive modeling to generate photographs. SEED-X proposes a unified and versatile basis system able to dealing with each multimodal understanding and era duties. On this method, steady picture representations from the CLIP ViT encoder are mixed with textual content tokens and fed into a big language mannequin (LLM) to carry out next-word prediction and picture illustration regression. Chameleon introduces a household of token-based mixed-modal fashions able to each comprehending and producing photographs. This method represents all modalities as discrete tokens, using a unified transformer-based structure and coaching the mannequin from scratch in an end-to-end method. Compared, Present-O additionally adopts discrete tokens to characterize all modalities however makes use of a discrete diffusion course of as an alternative of autoregressive modeling for visible era.
SHOW-O: Methodology and Structure
The first goal behind the Present-O framework is to develop a unified mannequin that integrates autoregressive and diffusion modeling for joint multimodal understanding and era. Creating such a unified mannequin poses vital challenges, with core points revolving round: i) defining the mannequin’s enter/output house; ii) unifying varied kinds of enter information from completely different modalities; iii) integrating each autoregressive and diffusion modeling right into a single transformer; and iv) successfully coaching such a unified mannequin.
Present-O addresses these challenges with the next options:
- Present-O constructs the enter/output house by tokenizing textual content and picture information into discrete tokens.
- Present-O introduces its default structure and a unified prompting technique to construction enter information and modalities.
- Present-O demonstrates the best way to incorporate each autoregressive and diffusion modeling inside a single transformer.
- Present-O presents a three-stage coaching pipeline to successfully prepare the unified mannequin.
Tokenization
On condition that the proposed Present-O is constructed upon pre-trained LLMs, it’s pure to carry out unified studying within the discrete house. By sustaining a unified vocabulary that features discrete textual content and picture tokens, Present-O is tasked with the identical studying goal: predicting discrete tokens.
Textual content Tokenization
Present-O relies on a pre-trained LLM, and the identical tokenizer is used for textual content information tokenization with none modifications.
Picture Tokenization
Following MAGVIT-v2, Present-O trains a lookup-free quantizer utilizing round 35M picture information. The quantizer maintains a codebook of dimension 8,192 and encodes photographs of 256×256 decision into 16×16 discrete tokens. MAGVIT-v2 is chosen for its ease of fine-tuning, making it appropriate as a video tokenizer with temporal compression functionality, a facet Present-O plans to discover sooner or later. Another method is to make use of completely different tokenizers for understanding and era, respectively. Impressed by present research, Present-O additionally extracts steady picture representations from the pre-trained MAGVIT-v2 and CLIP-ViT encoder to discover enhancements in multimodal understanding capabilities.. Within the following sections, the default Present-O employs discrete picture tokens as enter for each multimodal understanding and era. For simplicity, the methodology sections will elaborate solely on the default Present-O.
Structure
Present-O inherits the structure of present LLMs with none structure modifications, aside from prepending a QK-Norm operation to every consideration layer. Present-O is initialized with the weights of a pre-trained LLM and expands the scale of the embedding layer by incorporating 8,192 new learnable embeddings for discrete picture tokens. In contrast to state-of-the-art diffusion fashions that require a further textual content encoder, Present-O inherently encodes textual content conditional info for text-to-image era.
Unified Prompting
To carry out unified studying on multimodal understanding and era, Present-O makes use of a unified prompting technique to format varied sorts of enter information. Given an image-text pair (x, y), it’s first tokenized into M picture tokens and N textual content tokens by the picture and textual content tokenizers, respectively. The tokens are then shaped into an enter sequence in keeping with the duty kind, as illustrated within the following determine.
By using this immediate design, Present-O can successfully encode varied enter information for multimodal understanding, text-to-image era, and mixed-modality era as sequential information. This setup permits unified studying to function seamlessly throughout sequences for these varied duties. As soon as educated, Present-O could be prompted to deal with a variety of vision-language duties, together with visible query answering and text-to-image era.
Omni-Consideration Mechanism
In contrast to present works that mannequin sequences autoregressively solely, Present-O introduces an omni-attention mechanism, enabling it to mannequin varied kinds of alerts in distinct methods. This complete consideration mechanism adaptively switches between causal and full consideration based mostly on the format of the enter sequence. The next determine illustrates examples of omni-attention for various enter sequences.
Particularly, Present-O processes textual content tokens throughout the sequence by way of causal consideration, whereas picture tokens are dealt with utilizing full consideration, permitting every token to comprehensively work together with all others. In multimodal understanding, textual content tokens can attend to all earlier picture tokens, whereas in text-to-image era, picture tokens can work together with all previous textual content tokens. Omni-attention retains the textual content reasoning data from the pre-trained LLM and enhances the effectivity of picture era by lowering sampling steps. Moreover, it helps varied downstream purposes, equivalent to inpainting and extrapolation, with out requiring fine-tuning. When given solely textual content tokens, the mechanism defaults to causal consideration.
SHOW-O: Experiments and Outcomes
The next desk presents the multimodal understanding functionality of Present-O on public benchmarks, equivalent to picture captioning and visible question-answering duties.
The present model of Present-O is constructed upon Phi-1.5, and subsequently, Present-O’s understanding-only counterpart, LLaVA-v1.5-Phi-1.5, serves because the direct baseline. Present-O displays comparable efficiency in all analysis metrics to the baseline LLaVA-v1.5-Phi-1.5, which is devoted solely to multimodal understanding. This demonstrates the nice potential of the Present-O framework to unify multimodal understanding and era inside a single transformer. When in comparison with understanding-only fashions like InstructBLIP, Qwen-VL-Chat, and mPLUG-Owl2, Present-O, regardless of having a a lot smaller mannequin dimension, achieves aggressive efficiency on the POPE, MME, Flickr30k, and VQAv2 benchmarks, and performs higher on the GQA benchmark. When in comparison with unified fashions with considerably extra parameters, equivalent to NExT-GPT-13B and Chameleon-34B, Present-O additionally achieves sturdy efficiency on the Flickr30k benchmark and performs a lot better on the VQAv2 benchmark.
Given these promising outcomes, Present-O is envisioned as a possible next-generation basis mannequin for unifying understanding and era. These outcomes additionally show the potential of scaling Present-O to attain state-of-the-art efficiency.
Qualitative Comparisons
We current qualitative comparisons with diffusion-based fashions, equivalent to SDv1.5, SDXL, and the autoregressive-based mannequin LlamaGen, alongside unified fashions like LWM and SEED-X, as demonstrated within the following determine.
Present-O demonstrates the power to generate life like photographs with constant content material described in each brief and lengthy textual content prompts. In comparison with SDv1.5 and LlamaGen, Present-O displays higher visible high quality and stronger image-text alignment. As an example, within the second column, each SDv1.5 and LlamaGen fail to totally comprehend the textual content immediate and miss attributes equivalent to “sundown” and “blue domes” within the generated photographs. Compared to SDXL, Present-O supplies comparable visible high quality and alignment, as seen in examples like “a rally automobile race” and “beautiful distinction towards the colourful sundown.”
Textual content-Guided Inpainting and Extrapolation
Present-O naturally helps text-based inpainting and extrapolation with out requiring any fine-tuning. The next determine illustrates a number of examples.
On the prime of the determine, given an enter picture and an inpainting masks, Present-O can remodel a pink trolley automobile right into a blue sports activities automobile with modern curves and tinted home windows based mostly on a user-provided textual content immediate. Present-O can even extrapolate the unique picture horizontally or vertically based mostly on the given textual content immediate. As an example, within the second row, Present-O extrapolates a picture by including new objects, like “pink wildflowers.” The pixels in each the in-painted and extrapolated areas stay in step with the unique picture. These examples clearly show the inherent benefits of Present-O over autoregressive fashions for downstream purposes.
Ultimate Ideas
On this article now we have talked about Present-O, a unified transformer that integrates multimodal understanding and era. In contrast to absolutely autoregressive fashions, Present-O unifies autoregressive and discrete diffusion modeling to adaptively deal with inputs and outputs of assorted and combined modalities. The unified mannequin flexibly helps a variety of vision-language duties, together with visible query answering, text-to-image era, text-guided inpainting/extrapolation, and mixed-modality era. Throughout varied benchmarks, Present-O demonstrates comparable or superior efficiency to present particular person fashions with an equal or bigger variety of parameters, highlighting its potential as a next-generation basis mannequin. On this framework, the mannequin is tasked with predicting Gaussian noise added to the continual latent representations. In distinction, different fashions like D3PM, Masks-predict, ARDM, and MaskGIT use a discrete corruption course of as an alternative choice to Gaussian diffusion. Present-O is the primary to unify autoregressive and discrete diffusion modeling, enabling it to deal with completely different modalities in distinct methods. In depth experimental outcomes show that Present-O is akin to, and even higher than, particular person professional fashions throughout a variety of vision-language duties. This highlights its potential as a next-generation basis mannequin.
