Picture-to-Picture Translation with FLUX.1: Instinct and Tutorial | by Youness Mansar | Oct, 2024

This submit guides you thru producing new pictures based mostly on current ones and textual prompts. This method, offered in a paper referred to as SDEdit: Guided Picture Synthesis and Modifying with Stochastic Differential Equations is utilized right here to FLUX.1.

First, we’ll briefly clarify how latent diffusion fashions work. Then, we’ll see how SDEdit modifies the backward diffusion course of to edit pictures based mostly on textual content prompts. Lastly, we’ll present the code to run the complete pipeline.

Latent diffusion performs the diffusion course of in a lower-dimensional latent house. Let’s outline latent house:

A variational autoencoder (VAE) initiatives the picture from pixel house (the RGB-height-width illustration people perceive) to a smaller latent house. This compression retains sufficient info to reconstruct the picture later. The diffusion course of operates on this latent house as a result of it’s computationally cheaper and fewer delicate to irrelevant pixel-space particulars.

Now, lets clarify latent diffusion:

The diffusion course of has two elements:

• Ahead Diffusion: A scheduled, non-learned course of that transforms a pure picture into pure noise over a number of steps.
• Backward Diffusion: A realized course of that reconstructs a natural-looking picture from pure noise.

Notice that the noise is added to the latent house and follows a selected schedule, from weak to robust within the ahead course of.

Noise is added to the latent house following a selected schedule, progressing from weak to robust noise throughout ahead diffusion. This multi-step strategy simplifies the community’s activity in comparison with one-shot era strategies like GANs. The backward course of is realized by way of probability maximization, which is simpler to optimize than adversarial losses.

Textual content Conditioning

Technology can be conditioned on additional info like textual content, which is the immediate that you just may give to a Secure diffusion or a Flux.1 mannequin. This textual content is included as a “trace” to the diffusion mannequin when studying methods to do the backward course of. This textual content is encoded utilizing one thing like a CLIP or T5 mannequin and fed to the UNet or Transformer to information it in the direction of the appropriate authentic picture that was perturbed by noise.

The concept behind SDEdit is straightforward: Within the backward course of, as an alternative of ranging from full random noise just like the “Step 1” of the picture above, it begins with the enter picture + a scaled random noise, earlier than operating the common backward diffusion course of. So it goes as follows:

• Load the enter picture, preprocess it for the VAE
• Run it by way of the VAE and pattern one output (VAE returns a distribution, so we’d like the sampling to get one occasion of the distribution).
• Decide a beginning step t_i of the backward diffusion course of.
• Pattern some noise scaled to the extent of t_i and add it to the latent picture illustration.
• Begin the backward diffusion course of from t_i utilizing the noisy latent picture and the immediate.
• Challenge the end result again to the pixel house utilizing the VAE.
• Voila !

Right here is methods to run this workflow utilizing diffusers:

First, set up dependencies ▶️

pip set up git+https://github.com/huggingface/diffusers.git optimum-quanto

For now, you have to set up diffusers from supply as this characteristic is just not accessible but on pypi.

Subsequent, load the FluxImg2Img pipeline ▶️

import os
from diffusers import FluxImg2ImgPipeline
from optimum.quanto import qint8, qint4, quantize, freeze
import torch
from typing import Callable, Checklist, Elective, Union, Dict, Any
from PIL import Picture
import requests
import io
MODEL_PATH = os.getenv("MODEL_PATH", "black-forest-labs/FLUX.1-dev")
pipeline = FluxImg2ImgPipeline.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16)
quantize(pipeline.text_encoder, weights=qint4, exclude="proj_out")
freeze(pipeline.text_encoder)
quantize(pipeline.text_encoder_2, weights=qint4, exclude="proj_out")
freeze(pipeline.text_encoder_2)
quantize(pipeline.transformer, weights=qint8, exclude="proj_out")
freeze(pipeline.transformer)
pipeline = pipeline.to("cuda")
generator = torch.Generator(gadget="cuda").manual_seed(100)

This code masses the pipeline and quantizes some elements of it in order that it matches on an L4 GPU accessible on Colab.

Now, lets outline one utility perform to load pictures within the right measurement with out distortions ▶️

def resize_image_center_crop(image_path_or_url, target_width, target_height):
"""
Resizes a picture whereas sustaining facet ratio utilizing heart cropping.
Handles each native file paths and URLs.
Args:
image_path_or_url: Path to the picture file or URL.
target_width: Desired width of the output picture.
target_height: Desired top of the output picture.
Returns:
A PIL Picture object with the resized picture, or None if there's an error.
"""
strive:
if image_path_or_url.startswith(('http://', 'https://')):  # Examine if it is a URL
response = requests.get(image_path_or_url, stream=True)
response.raise_for_status()  # Increase HTTPError for unhealthy responses (4xx or 5xx)
img = Picture.open(io.BytesIO(response.content material))
else:  # Assume it is a native file path
img = Picture.open(image_path_or_url)
img_width, img_height = img.measurement
# Calculate facet ratios
aspect_ratio_img = img_width / img_height
aspect_ratio_target = target_width / target_height
# Decide cropping field
if aspect_ratio_img > aspect_ratio_target:  # Picture is wider than goal
new_width = int(img_height * aspect_ratio_target)
left = (img_width - new_width) // 2
proper = left + new_width
high = 0
backside = img_height
else:  # Picture is taller or equal to focus on
new_height = int(img_width / aspect_ratio_target)
left = 0
proper = img_width
high = (img_height - new_height) // 2
backside = high + new_height
# Crop the picture
cropped_img = img.crop((left, high, proper, backside))
# Resize to focus on dimensions
resized_img = cropped_img.resize((target_width, target_height), Picture.LANCZOS)
return resized_img
besides (FileNotFoundError, requests.exceptions.RequestException, IOError) as e:
print(f"Error: Couldn't open or course of picture from '{image_path_or_url}'.  Error: {e}")
return None
besides Exception as e: #Catch different potential exceptions throughout picture processing.
print(f"An sudden error occurred: {e}")
return None

Lastly, lets load the picture and run the pipeline ▶️

url = "https://pictures.unsplash.com/photo-1609665558965-8e4c789cd7c5?ixlib=rb-4.0.3&q=85&fm=jpg&crop=entropy&cs=srgb&dl=sven-mieke-G-8B32scqMc-unsplash.jpg"
picture = resize_image_center_crop(image_path_or_url=url, target_width=1024, target_height=1024)
immediate = "An image of a Tiger"
image2 = pipeline(immediate, picture=picture, guidance_scale=3.5, generator=generator, top=1024, width=1024, num_inference_steps=28, energy=0.9).pictures[0]

This transforms the next picture:

To this one:

You possibly can see that the cat has an analogous pose and form as the unique cat however with a unique coloration carpet. Which means that the mannequin adopted the identical sample as the unique picture whereas additionally taking some liberties to make it extra becoming to the textual content immediate.

There are two essential parameters right here:

• The num_inference_steps: It’s the variety of de-noising steps in the course of the backwards diffusion, a better quantity means higher high quality however longer era time
• The energy: It management how a lot noise or how far again within the diffusion course of you need to begin. A smaller quantity means little adjustments and better quantity means extra important adjustments.

Now you understand how Picture-to-Picture latent diffusion works and methods to run it in python. In my assessments, the outcomes can nonetheless be hit-and-miss with this strategy, I often want to vary the variety of steps, the energy and the immediate to get it to stick to the immediate higher. The following step would to look into an strategy that has higher immediate adherence whereas additionally preserving the important thing parts of the enter picture.

Full code: https://colab.analysis.google.com/drive/1GJ7gYjvp6LbmYwqcbu-ftsA6YHs8BnvO