View source for Latent diffusion model

{{Short description|Diffusion model over latent embedding space}}
{{Infobox software
| name = Latent Diffusion Model
| logo = 
| screenshot = 
| author = CompVis
| developer = 
| released = December 20, 2021
| latest release = 
| genre = {{ indented plainlist |
*[[Generative artificial intelligence|Generative model]]
*[[Diffusion model]]
}}
| license = [[MIT license|MIT]]
| repo = https://github.com/CompVis/latent-diffusion
| programming language = [[Python (programming language)|Python]]
| operating system = 
| website = 
}}

The '''Latent Diffusion Model''' ('''LDM''')<ref>{{Cite conference |last1=Rombach |first1=Robin |last2=Blattmann |first2=Andreas |last3=Lorenz |first3=Dominik |last4=Esser |first4=Patrick |last5=Ommer |first5=Björn |date=2022 |title=High-Resolution Image Synthesis With Latent Diffusion Models |conference=The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022|url=https://openaccess.thecvf.com/content/CVPR2022/html/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.html |language=en |pages=10684–10695}}</ref> is a [[diffusion model]] architecture developed by the CompVis (Computer Vision & Learning)<ref>{{Cite web |title=Home |url=https://ommer-lab.com/ |access-date=2024-09-05 |website=Computer Vision & Learning Group |language=en-US}}</ref> group at [[LMU Munich]].<ref name="stable-diffusion-github">{{cite web |date=17 September 2022 |title=Stable Diffusion Repository on GitHub |url=https://github.com/CompVis/stable-diffusion |url-status=live |archive-url=https://web.archive.org/web/20230118183342/https://github.com/CompVis/stable-diffusion |archive-date=January 18, 2023 |access-date=17 September 2022 |publisher=CompVis - Machine Vision and Learning Research Group, LMU Munich}}</ref>

Introduced in 2015, diffusion models (DMs) are trained with the objective of removing successive applications of noise (commonly [[Gaussian noise|Gaussian]]) on training images. The LDM is an improvement on standard DM by performing diffusion modeling in a [[latent space]], and by allowing self-attention and cross-attention conditioning.

LDMs are widely used in practical diffusion models. For instance, [[Stable Diffusion]] versions 1.1 to 2.1 were based on the LDM architecture.<ref name=":02">{{Cite web |last=Alammar |first=Jay |title=The Illustrated Stable Diffusion |url=https://jalammar.github.io/illustrated-stable-diffusion/ |url-status=live |archive-url=https://web.archive.org/web/20221101104342/https://jalammar.github.io/illustrated-stable-diffusion/ |archive-date=November 1, 2022 |access-date=2022-10-31 |website=jalammar.github.io}}</ref>

== Version history ==
Diffusion models were introduced in 2015 as a method to learn a model that can sample from a highly complex probability distribution. They used techniques from [[non-equilibrium thermodynamics]], especially [[diffusion]].<ref>{{Cite journal |last1=Sohl-Dickstein |first1=Jascha |last2=Weiss |first2=Eric |last3=Maheswaranathan |first3=Niru |last4=Ganguli |first4=Surya |date=2015-06-01 |title=Deep Unsupervised Learning using Nonequilibrium Thermodynamics |url=http://proceedings.mlr.press/v37/sohl-dickstein15.pdf |journal=Proceedings of the 32nd International Conference on Machine Learning |language=en |publisher=PMLR |volume=37 |pages=2256–2265|arxiv=1503.03585 }}</ref> It was accompanied by a software implementation in [[Theano (software)|Theano]].<ref>{{cite web |last=Sohl-Dickstein |first=Jascha |title=Sohl-Dickstein/Diffusion-Probabilistic-Models |website=[[GitHub]] |date=2024-09-01 |url=https://github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models |access-date=2024-09-07}}</ref>

A 2019 paper proposed the '''noise conditional score network (NCSN)''' or score-matching with Langevin dynamics (SMLD).<ref>{{cite web |title=ermongroup/ncsn |date=2019 |url=https://github.com/ermongroup/ncsn |access-date=2024-09-07 |publisher=ermongroup}}</ref> The paper was accompanied by a software package written in [[PyTorch]] release on GitHub.<ref>{{Cite journal |last1=Song |first1=Yang |last2=Ermon |first2=Stefano |date=2019 |title=Generative Modeling by Estimating Gradients of the Data Distribution |url=https://proceedings.neurips.cc/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=32|arxiv=1907.05600 }}</ref>

A 2020 paper<ref name="ho">{{Cite journal |last1=Ho |first1=Jonathan |last2=Jain |first2=Ajay |last3=Abbeel |first3=Pieter |date=2020 |title=Denoising Diffusion Probabilistic Models |url=https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=33 |pages=6840–6851}}</ref> proposed the '''Denoising Diffusion Probabilistic Model (DDPM)''', which improves upon the previous method  by [[Variational Bayesian methods|variational inference]]. The paper was accompanied by a software package written in [[TensorFlow]] release on GitHub.<ref>{{cite web |last=Ho |first=Jonathan |title=hojonathanho/diffusion |website=[[GitHub]] |date=Jun 20, 2020 |url=https://github.com/hojonathanho/diffusion |access-date=2024-09-07}}</ref> It was reimplemented in [[PyTorch]] by lucidrains.<ref>{{cite web |last=Wang |first=Phil |title=lucidrains/denoising-diffusion-pytorch |website=[[GitHub]] |date=2024-09-07 |url=https://github.com/lucidrains/denoising-diffusion-pytorch |access-date=2024-09-07}}</ref><ref>{{Cite web |title=The Annotated Diffusion Model |url=https://huggingface.co/blog/annotated-diffusion |access-date=2024-09-07 |website=huggingface.co}}</ref>

On December 20, 2021, the LDM paper was published on arXiv,<ref>{{cite arXiv |last1=Rombach |first1=Robin |title=High-Resolution Image Synthesis with Latent Diffusion Models |date=2021-12-20 |eprint=2112.10752 |last2=Blattmann |first2=Andreas |last3=Lorenz |first3=Dominik |last4=Esser |first4=Patrick |last5=Ommer |first5=Björn|class=cs.CV }}</ref> and both '''Stable Diffusion'''<ref>{{Cite web |title=Update README.md · CompVis/stable-diffusion@17e64e3 |url=https://github.com/CompVis/stable-diffusion/commit/17e64e320556d49e429f41a7093b291c49255470 |access-date=2024-09-07 |website=GitHub |language=en}}</ref> and LDM<ref>{{Cite web |title=Update README.md · CompVis/latent-diffusion@17e64e3 |url=https://github.com/CompVis/latent-diffusion/commit/17e64e320556d49e429f41a7093b291c49255470 |access-date=2024-09-07 |website=GitHub |language=en}}</ref> repositories were published on GitHub. However, they remained roughly the same. Substantial information concerning Stable Diffusion v1 was only added to GitHub on August 10, 2022.<ref>{{Cite web |title=stable diffusion · CompVis/stable-diffusion@2ff270f |url=https://github.com/CompVis/stable-diffusion/commit/2ff270f4e0c884d9684fa038f6d84d8600a94b39 |access-date=2024-09-07 |website=GitHub |language=en}}</ref>

All of Stable Diffusion (SD) versions 1.1 to XL were particular instantiations of the LDM architecture. 

SD 1.1 to 1.4 were released by CompVis in August 2022. There is no "version 1.0". SD 1.1 was a LDM trained on the laion2B-en dataset. SD 1.1 was finetuned to 1.2 on more aesthetic images. SD 1.2 was finetuned to 1.3, 1.4 and 1.5, with 10% of text-conditioning dropped, to improve classifier-free guidance.<ref>{{Cite web |date=2023-08-23 |title=CompVis (CompVis) |url=https://huggingface.co/CompVis |access-date=2024-03-06 |website=huggingface.co}}</ref><ref name=":0" /> SD 1.5 was released by [[Runway (company)|RunwayML]] in October 2022.<ref name=":0">{{Cite web |title=runwayml/stable-diffusion-v1-5 · Hugging Face |url=https://huggingface.co/runwayml/stable-diffusion-v1-5 |url-status=live |archive-url=https://web.archive.org/web/20230921025150/https://huggingface.co/runwayml/stable-diffusion-v1-5 |archive-date=September 21, 2023 |access-date=2023-08-17 |website=huggingface.co}}</ref>

== Architecture ==

While the LDM can work for generating arbitrary data conditional on arbitrary data, for concreteness, we describe its operation in conditional text-to-image generation.

LDM consists of a [[variational autoencoder]] (VAE), a modified [[U-Net]], and a text encoder.

The VAE encoder compresses the image from pixel space to a smaller dimensional [[latent space]], capturing a more fundamental semantic meaning of the image. Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion. The U-Net block, composed of a [[Residual neural network|ResNet]] backbone, [[Noise reduction|denoises]] the output from forward diffusion backwards to obtain a latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space.<ref name=":02" />

The denoising step can be conditioned on a string of text, an image, or another modality. The encoded conditioning data is exposed to denoising U-Nets via a [[Attention (machine learning)|cross-attention mechanism]].<ref name=":02" /> For conditioning on text, the fixed, a pretrained [[Contrastive Language-Image Pre-training|CLIP]] ViT-L/14 text encoder is used to transform text prompts to an embedding space.<ref name="stable-diffusion-github" />

=== Variational Autoencoder ===

To compress the image data, a variational autoencoder (VAE) is first trained on a dataset of images. The encoder part of the VAE takes an image as input and outputs a lower-dimensional latent representation of the image. This latent representation is then used as input to the U-Net. Once the model is trained, the encoder is used to encode images into latent representations, and the decoder is used to decode latent representations back into images.

Let the encoder and the decoder of the VAE be <math>E, D</math>.

To encode an RGB image, its three channels are divided by the maximum value, resulting in a tensor <math>x</math> of shape <math>(3, 512, 512)</math> with all entries within range <math>[0, 1]</math>. The encoded vector is <math>0.18215 \times E(2x - 1)</math>, with shape <math>(4, 64, 64)</math>, where 0.18215 is a hyperparameter, which the original authors picked to roughly [[Whitening transformation|whiten]] the encoded vector to roughly unit variance. Conversely, given a latent tensor <math>y</math>, the decoded image is <math>(D(y / 0.18125) + 1)/2</math>, then [[Clipping (signal processing)|clipped]] to the range <math>[0, 1]</math>.<ref>{{Cite web |title=Explanation of the 0.18215 factor in textual_inversion? · Issue #437 · huggingface/diffusers |url=https://github.com/huggingface/diffusers/issues/437 |access-date=2024-09-19 |website=GitHub |language=en}}</ref><ref>{{Cite web |title=diffusion-nbs/Stable Diffusion Deep Dive.ipynb at master · fastai/diffusion-nbs |url=https://github.com/fastai/diffusion-nbs/blob/master/Stable%20Diffusion%20Deep%20Dive.ipynb |access-date=2024-09-19 |website=GitHub |language=en}}</ref>

In the implemented version,<ref name="stable-diffusion-github" />{{Pg|location=ldm/models/autoencoder.py}} the encoder is a [[convolutional neural network]] (CNN) with a single self-attention mechanism near the end. It takes a tensor of shape <math>(3, H, W)</math> and outputs a tensor of shape <math>(8, H/8, W/8)</math>, being the concatenation of the predicted mean and variance of the latent vector, each of shape <math>(4, H/8, W/8)</math>. The variance is used in training, but after training, usually only the mean is taken, with the variance discarded.

The decoder is also a CNN with a single self-attention mechanism near the end. It takes a tensor of shape <math>(4, H/8, W/8)</math> and outputs a tensor of shape <math>(3, H, W)</math>.

=== U-Net ===

The U-Net backbone takes the following kinds of inputs:

* A '''latent image array''', produced by the VAE encoder. It has dimensions <math>(\text{channel}, \text{width}, \text{height})</math>. Typically, <math>(\text{channel}, \text{width}, \text{height}) = (4, 64, 64)</math>.
* A '''timestep-embedding vector''', which tells the backbone how much noise there is in the image. For example, an embedding of timestep <math>t = 0</math> would indicate that the input image is already noiseless, while <math>t = 100</math> would mean there is much noise.
* A '''modality-embedding vector sequence''', which indicates to the backbone about additional conditions for denoising. For example, in text-to-image generation, the text is divided into a sequence of tokens, then encoded by a text encoder, such as a [[DALL-E#Contrastive Language-Image Pre-training (CLIP)|CLIP encoder]], before feeding into the backbone. As another example, an input image can be processed by a [[Vision transformer|Vision Transformer]] into a sequence of vectors, which can then be used to condition the backbone for tasks such as generating an image in the same style.

Each run through the U-Net backbone produces a predicted noise vector. This noise vector is scaled down and subtracted away from the latent image array, resulting in a slightly less noisy latent image. The denoising is repeated according to a denoising schedule ("noise schedule"), and the output of the last step is processed by the VAE decoder into a finished image.
[[File:Encoder cross-attention.png|thumb|A single cross-attention mechanism as it appears in a standard Transformer language model.]]
[[File:Transformer,_full_architecture.png|thumb|Block diagram for the full Transformer architecture. The stack on the right is a standard pre-LN Transformer decoder, which is essentially the same as the <code>SpatialTransformer</code>.]]
Similar to the standard [[U-Net]], the U-Net backbone used in the SD 1.5 is essentially composed of down-scaling layers followed by up-scaling layers. However, the U-Net backbone has additional modules to allow for it to handle the embedding. As an illustration, we describe a single down-scaling layer in the backbone:

* The latent array and the time-embedding are processed by a <code>ResBlock</code>:
** The latent array is processed by a [[Convolutional neural network|convolutional layer]].
** The time-embedding vector is processed by a one-layered [[Feedforward neural network|feedforward network]], then added to the previous array (broadcast over all pixels).
** This is processed by another convolutional layer, then another time-embedding.
* The latent array and the embedding vector sequence are processed by a <code>SpatialTransformer</code>, which is essentially a standard pre-LN [[Transformer (deep learning architecture)#Decoder|Transformer decoder]] without causal masking.
** In the cross-attentional blocks, the latent array itself serves as the query sequence, one query-vector per pixel. For example, if, at this layer in the U-Net, the latent array has dimensions <math>(128, 32, 32)</math>, then the query sequence has <math>1024</math> vectors, each of which has <math>128</math> dimensions. The embedding vector sequence serves as both the key sequence and as the value sequence.
** When no embedding vector sequence is input, a cross-attentional block defaults to self-attention, with the latent array serving as the query, key, and value.<ref>{{Cite web |title=latent-diffusion/ldm/modules/attention.py at main · CompVis/latent-diffusion |url=https://github.com/CompVis/latent-diffusion/blob/main/ldm/modules/attention.py#L251 |access-date=2024-09-09 |website=GitHub |language=en}}</ref>{{Pg|location=line 251}}

In pseudocode,
<syntaxhighlight lang="python3">
def ResBlock(x, time, residual_channels):
    x_in = x
    time_embedding = feedforward_network(time)
    x = concatenate(x, residual_channels)
    x = conv_layer_1(activate(normalize_1(x))) + time_embedding
    x = conv_layer_2(dropout(activate(normalize_2(x))))
    return x_in + x

def SpatialTransformer(x, cond):
    x_in = x
    x = normalize(x)
    x = proj_in(x)
    x = cross_attention(x, cond)
    x = proj_out(x)
    return x_in + x
    
def unet(x, time, cond):
    residual_channels = []
    for resblock, spatialtransformer in downscaling_layers:
        x = resblock(x, time)
        residual_channels.append(x)
        x = spatialtransformer(x, cond)
        
    x = middle_layer.resblock_1(x, time)
    x = middle_layer.spatialtransformer(x, time)
    x = middle_layer.resblock_2(x, time)
    
    for resblock, spatialtransformer in upscaling_layers:
        residual = residual_channels.pop()
        x = resblock(concatenate(x, residual), time)
        x = spatialtransformer(x, cond)
    
    return x 
</syntaxhighlight>The detailed architecture may be found in.<ref>{{Cite web |title=U-Net for Stable Diffusion |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet.html |access-date=2024-08-31 |website=U-Net for Stable Diffusion |language=en}}</ref><ref>{{Cite web |title=Transformer for Stable Diffusion U-Net |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet_attention.html |access-date=2024-09-07 |website=Transformer for Stable Diffusion U-Net |language=en}}</ref>

== Training and inference ==
The LDM is trained by using a [[Markov chain]] to gradually add noise to the training images. The model is then trained to reverse this process, starting with a noisy image and gradually removing the noise until it recovers the original image.
More specifically, the training process can be described as follows:

* Forward diffusion process: Given a real image <math>x_0</math>, a sequence of latent variables <math>x_{1:T}</math> are generated by gradually adding Gaussian noise to the image, according to a pre-determined "noise schedule". 
* Reverse diffusion process: Starting from a Gaussian noise sample <math>x_T</math>, the model learns to predict the noise added at each step, in order to reverse the diffusion process and obtain a reconstruction of the original image <math>x_0</math>.

The model is trained to minimize the difference between the predicted noise and the actual noise added at each step. This is typically done using a [[mean squared error]] (MSE) loss function.

Once the model is trained, it can be used to generate new images by simply running the reverse diffusion process starting from a random noise sample. The model gradually removes the noise from the sample, guided by the learned noise distribution, until it generates a final image.

See the [[diffusion model]] page for details.

== See also ==
* [[Diffusion model]]
* [[Generative adversarial network]]
* [[Variational autoencoder]]
* [[Stable Diffusion]]

== References ==
<references/>

== Further reading ==

*{{cite web |last=Wang |first=Phil |title=lucidrains/denoising-diffusion-pytorch |website=[[GitHub]] |date=2024-09-07 |url=https://github.com/lucidrains/denoising-diffusion-pytorch?tab=readme-ov-file |access-date=2024-09-07}}
*{{Cite web |title=The Annotated Diffusion Model |url=https://huggingface.co/blog/annotated-diffusion |access-date=2024-09-07 |website=huggingface.co}}
*{{Cite web |title=U-Net for Stable Diffusion |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet.html |access-date=2024-08-31 |website=U-Net for Stable Diffusion |language=en}}
*{{Cite web |title=Transformer for Stable Diffusion U-Net |url=https://nn.labml.ai/diffusion/stable_diffusion/model/unet_attention.html |access-date=2024-09-07 |website=Transformer for Stable Diffusion U-Net |language=en}}

{{Artificial intelligence navbox}}

[[Category:Deep learning]]
[[Category:Generative artificial intelligence]]
[[Category:Image processing]]
[[Category:Artificial intelligence art]]
[[Category:Text-to-image generation]]
[[Category:Unsupervised learning]]
[[Category:2021 software]]