Introduction
This work proposes an finish to finish 2D relighting diffusion mannequin. This mannequin learns bodily priors from artificial dataset that includes bodily based mostly supplies and HDR surroundings maps. It may be additional used to relight a number of views and be used to create a 3D illustration of the scene.
Technique
Given a picture and a goal HDR surroundings map, the objective is to be taught a mannequin that may synthesize a relit model of the picture which here’s a single object. That is achieved by adopting a pre-trained Zero-1-to-3 mannequin. Zero-1-to-3 is a diffusion mannequin that’s conditioned on view course to render novel views of an enter picture. They discard its novel view synthesis parts. To include lighting situations, they concatenate enter picture and surroundings map encodings with the denoising latent.
The enter HDR surroundings map E is cut up into two parts: E_l, a tone-mapped LDR illustration capturing lighting particulars in low-intensity areas, and E_h, a log-normalized map preserving data throughout the total spectrum. Collectively, these present the community with a balanced illustration of the vitality spectrum, guaranteeing correct relighting with out the generated output showing washed out resulting from excessive brightness.
Moreover the CLIP embedding of the enter picture can be handed as enter. Thus the enter to the mannequin is the Enter Picture, LDR Picture, Normalized HDR Picture and CLIP embedding of Picture all conditioning the denoising community. This community is then used as prior for additional 3D object relighting.
Implementation
The mannequin is educated on a customized Relit Objaverse Dataset that consists of 90K objects. For every object there are 204 pictures which are rendered below completely different lighting situations and viewpoints. In whole, the dataset consists of 18.4 M pictures at decision 512×512.
The mannequin is finetuned from Zero-1-to-3’s checkpoint and solely the denoising community is finetined. The enter surroundings map is downsampled to 256×256 decision. The mannequin is educated on 8 A6000 GPUs for five days. Additional downstream duties corresponding to text-based relighting and object insertion will be achieved.
Outcomes
They present comparisons with completely different backgrounds and comparisons with different works corresponding to DilightNet and IC-Gentle.
This determine compares the relighting outcomes of their technique with IC-Gentle, one other ControlNet based mostly technique. Their technique can produce constant lighting and shade with the rotating surroundings map.
This determine compares the relighting outcomes of their technique with DiLightnet, one other ControlNet based mostly technique. Their technique can produce specular highlights and correct colours.
Limitations
A serious limitation is that it solely produces low picture decision (256×256). Moreover it solely works on objects and performs poorly for portrait relighting.