Author: Arun NandaLast Updated: Fri, Nov 11, 2022
Stable Diffusion is an open-source deep learning model that generates new images with input text prompts. It can also be used to modify images; the model can generate a new image from an input image and a text prompt.
This guide uses pre-trained Stable Diffusion models to set up a system to generate new images based on an input image and a text prompt. It doesn't cover how to train models from data.
The steps described here have been tested on a Vultr Cloud GPU instance with a 10 GB Video-RAM (V-RAM) share on an NVIDIA A100 GPU.
While it is possible to run Stable Diffusion on CPU-only machines, the processing times can be orders of magnitude higher in the absence of a GPU.
The system described here was set up on a VM running Debian 11. Operating System based commands are compatible with recent versions of Ubuntu. Python 3 commands are independent of the Operating System.
Given the rapid changes in deep learning, software updates can sometimes break compatibility. Wherever necessary, the installation commands specify the version numbers of the software used. Using newer or older versions of the same software may lead to conflicts between dependencies.
After generating images (in Section 4), you will need a way to view or copy them to the local machine.
sshfs can mount a remote directory on the local filesystem. You can then use the command line or the GUI to browse the remote directory mounted locally. You can also copy remote files to the local filesystem.
Check on the local machine if
sshfs is installed:
$ sshfs --version
If it isn't installed, install it:
# apt install sshfs
Create a mount point on the local machine:
$ mkdir ~/sshfs_mountpoint
Mount the server's home directory on this mount point:
$ sshfs -o follow_symlinks email@example.com:/home/username ~/sshfs_mountpoint
Before using pre-trained Stable Diffusion models, you need to be able to access them. These models are available from Hugging Face - a data science community and platform (hub) for sharing machine learning tools and models. To use any of the pre-trained models, you need an access token. As of November 2022, joining the hub and creating access tokens is free.
Sign up for an account on the Hugging Face website using your email address. Follow the on-screen instructions and create your account. You need to confirm your email address.
After signing in with a confirmed account, navigate to the Settings page (click on the profile picture icon at the top right of the screen and click on Settings from the drop-down options). On the Settings page, navigate to Access Tokens settings (click on Access Tokens in the vertical menu on the left). On the Access Tokens page, use the "New token" button to create a new token. Give it a relevant name and choose "read" for the role - you only need to read the pre-trained model.
You need this access code in Section 4.
By default, on a new Vultr instance, you log in as root. It is not advisable to install Python packages as root. Before continuing further, set up a new unprivileged user and configure SSH access for that user.
Further steps assume that you are logged in as a regular, unprivileged user.
Environments help to isolate software used by different projects by installing them into separate directory trees. This guide uses Conda to manage Python environments.
Miniconda is a minimal Conda installer. Visit the Conda website and copy the download link for the latest Miniconda release. Download the installation file:
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
Run the installation file to install Miniconda:
$ bash Miniconda3-latest-Linux-x86_64.sh
Follow all on-screen instructions during the installation process.
Check that Conda is installed:
$ conda --version
conda to the latest version:
$ conda upgrade conda
Create a new Conda environment:
$ conda create --name env1 python=3.9
This creates a new environment,
env1, and installs Python 3.9 and
pip into it. Follow the on-screen instructions during the installation process.
conda env list at the command line to view the list of Conda environments. Activate the new Conda environment:
$ conda activate env1
To deactivate the currently active environment and get back to the regular command line, use
All further steps assume that the Conda environment
env1 is active.
Export an environment variable containing the Conda installation path:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
After exporting the path, you no longer need to type out the full paths of the applications installed into the environment.
Create an activation directory within the Conda directory:
$ mkdir -p $CONDA_PREFIX/etc/conda/activate.d
In a new file,
env_vars.sh within the activation directory, add a command to export the variable with the Conda installation path:
$ echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
This automatically loads the path at the start of every Conda session.
torch) is a Python library that enables tensor computations and includes tools for processing neural networks. PyTorch modules work on both GPU and CPU. On systems with GPU support, Torch takes advantage of the GPU for accelerating tensor computations.
CUDA is a "parallel computing platform and programming model" developed by NVIDIA for its own GPUs.
cudatoolkit contains the runtime support for CUDA. As of November 2022,
cudatoolkit is not available to install via PyPI. Installing PyTorch from PyPI and CUDA from Conda can lead to version and GPU-driver-architecture-related incompatibilities. To ensure compatibility of GPU libraries, install both Pytorch and CUDA from Conda:
$ conda install pytorch=1.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
Before continuing with installing other packages, check that PyTorch has been installed with CUDA support. Note that you might need to deactivate and re-activate the Conda environment
env1. Start a Python shell and import
>>> import torch
Check the CUDA version in
The output should be a version number resembling
Check the availability of CUDA support in the PyTorch installation:
The output should be
Exit the Python shell and continue to install the remaining packages.
To install other versions of PyTorch and CUDA, choose the appropriate versions based on the GPU drivers installed on your computer. You might need to do this if, for instance, the pre-installed NVIDIA drivers are updated. When this happens, you can get error messages that say the CUDA architecture is incompatible with the current PyTorch installation. You might also get errors if your computer uses a different GPU than the one this guide was tested on (NVIDIA A100).
Check the highest CUDA version that can be supported by the installed drivers:
The first line of the output should look like this:
NVIDIA-SMI 510.85.02 Driver Version: 510.85.02 CUDA Version: 11.6
The top right corner of the output shows the highest supported CUDA version.
Check the compatibility information on the PyTorch website to determine suitable versions of PyTorch and CUDA for your computer. If needed, use those versions instead of the ones specified in Section 3.4.1.
Upgrade to the latest version of
$ pip install --upgrade pip
The packages below can also be installed from Conda, but the installation process is faster with
Pillow is the actively maintained (as of November 2022) fork of the Python Imaging Library, PIL, which is no longer maintained. Install Pillow:
$ pip install --upgrade Pillow
Machine Learning pipelines are a convenient way of bundling the entire data processing workflow to construct a model.
Diffusers provide access to pre-trained diffusion models in the form of prepackaged pipelines. It provides the tools for building and training diffusion models. Diffusers also include a number of different core neural network models - these are used as building blocks to create new pipelines.
Since August 2022, Stable Diffusion pipelines have been available via Diffusers. Install
$ pip install --upgrade diffusers==0.7.2
Transformers is a library of many pre-trained machine-learning models built by the Hugging Face team. These models are based on TensorFlow and PyTorch. Transformers models can be used as the building blocks while constructing a new pipeline. Install Transformers:
$ pip install --upgrade transformers==4.24.0
Open a new Python command line. Copy your Hugging Face access code and assign it to a variable:
>>> hf_token = 'hf_copythetokenfromyourhuggingfaceaccount'
>>> import torch
Image module of Pillow is used for loading images from files and for saving new images. Import the
>>> from PIL import Image
Many deep learning pipelines, including different Stable Diffusion pipelines, are available via Diffusers. The Stable Diffusion Img2Img pipeline includes all the processing steps to generate images given an input image and an input text prompt. Import the Img2Img pipeline:
>>> from diffusers import StableDiffusionImg2ImgPipeline
from_pretrained() function loads all the components of a pre-trained pipeline, including model weights.
>>> pipe = StableDiffusionImg2ImgPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=hf_token )
The parameters passed to the
from_pretrained() function are:
the model id of a pipeline. The function call above loads the
stable-diffusion-v1-4 model hosted under the CompVis organization's repo. The model id can also be the path to a local directory containing model weights or a path (local or URL) to a checkpoint file.
revision denotes the specific version of the model whose repo id is used. This is a git-based branch name, tag name, or commit id.
torch_dtype is the Torch datatype of the tensors used for pipeline computations.
float16 is specified explicitly so that the model computations are done in 16-bit floating point numbers (instead of the default 32-bit). This helps systems with limited memory. It is possible to let the system choose the optimal data type using
torch_dtype = "auto". The default setting is
torch_dtype = "full-precision".
use_auth_token specifies an access token to download the pre-trained model from the Hugging Face repo.
By default, the
from_pretrained() function above loads the model into RAM (i.e., the memory space that the CPU has access to). The GPU performs tensor computations much faster than the CPU. But the GPU does not have access to the RAM, it can only access its own memory. So, the model needs to be "moved" to the GPU. The
Tensor.to() function is used to do this. Move the model to the GPU device (in this case, the GPU device is CUDA):
>>> device = "cuda" >>> pipe = pipe.to(device)
If the input image is already present on the server, load it in the Python terminal:
>>> input_image = Image.open('/path/to/image')
If the image is on the local machine, copy it onto the server using
scp (on the local machine):
$ scp /local/path/to/image firstname.lastname@example.org:/remote/path/to/image
After the image has been copied onto the server, load it in the Python terminal as in the previous subsection:
>>> input_image = Image.open('/path/to/image')
It is advisable to resize the input image to either a width or height of 512 pixels.
>>> height_orig = input_image.height >>> width_orig = input_image.width >>> aspect_ratio = width_orig / height_orig >>> width_new = 512 >>> height_new = int(width_new / aspect_ratio) >>> input_image = input_image.resize((width_new, height_new), 0)
Stable Diffusion models have been trained on images of 512x512 resolution. It is recommended that at least one of the dimensions of the input image be 512 pixels. Trying to generate an image of a resolution greater than 512 along both axes can lead to the repetition of regions of the image. Note that trying to process an image of resolution 1024x1024 on a system with a 10 GB share of V-RAM (on an A100 GPU) can lead to out-of-memory errors.
Create an input text prompt:
>>> prompt = "wearing a christmas hat"
Generate a single new image using the default settings on the loaded pipeline:
>>> pipe_output = pipe( prompt = prompt, init_image = input_image, )
In the above function call, the parameter
prompt is the input text prompt, and
init_image is the input image, as defined earlier.
Extract the images from the output object:
>>> output_images = pipe_output.images
output_images is an array of images.
Save the image:
View the images using
sshfs, as described in Section 3.1.
The image generation pipeline accepts a number of additional options. Only the parameters
init_image are mandatory.
>>> pipe_output = pipe( prompt = prompt, init_image = input_image, strength = 0.75, num_inference_steps = 100, guidance_scale = 7.5, negative_prompt = negative_prompt, num_images_per_prompt = 5 )
The above call to the pipeline function uses the following parameters:
prompt is the input text prompt. It is either a single text string or an array of strings. Note that using an array of strings leads to greater V-RAM requirements. In general, the more complex the prompt, the more V-RAM is needed to process it.
init_image is the input image to be used as the starting point for generating the new image.
strength is a value between 0 and 1. It indicates the amount of transformation that the input image can be subject to. The higher this value, the more the input image is changed. The default value is 0.8. When
strength is set to 1, the input image plays almost no role in generating the new image.
num_inference_steps: Stable Diffusion starts with random noise and repeatedly de-noises it to generate a meaningful picture.
num_inference_steps denotes the number of de-noising steps. The larger this value, the higher the quality of the output image(s). The default value is 50. The actual number of steps is modulated by the
strength parameter. When the
strength parameter has a value less than 1, the actual number of iterations is less than the number specified in
guidance_scale: Guidance refers to the extent to which the input text prompt influences image generation. It can be enabled by setting
guidance_scale to a value greater than 1. The higher this value, the more closely the generated image conforms to the input text prompt. However, the more the generated image matches the input text prompt, the lower the quality of the image. The default value is 7.5.
negative_prompt: If the prompt guides what the image should be, the negative prompt guides what it should not be. The negative prompt applies only if the guidance scale value is greater than 1.
num_images_per_prompt specifies the number of images to generate. Note that trying to generate a large number of images per function call can lead to out-of-memory errors. If you need to generate more images, call the function repeatedly.
Extract and view the images as in the previous section.
The default values mentioned above are true as of November 2022 and might change in the future as models are updated.
The extent to which images generated by the model are "accurate" depends on the extent to which the model was trained on related images and their associated text tags. It also depends on the configuration of the parameters described in this section. It can take some trial and error to figure out a good set of parameter values.
Stability AI has released the model to the general public under a Creative ML OpenRAIL-M license. It permits commercial and non-commercial usage. The license is focused on ensuring ethical and legal usage of the model. It also mandates that the license itself be included with any distribution of the model, including to end users.
This guide gives an overview of how to set up a Stable Diffusion system to generate new images from a given image and a text prompt. The observations made here are accurate as of November 2022. Future developments and software upgrades can render them partially inaccurate. It is common to come across bugs and inconsistencies in the model behavior across different versions. It is helpful to be able to read and understand error messages while configuring the system. The model card page on the Hugging Face website and the Stable Diffusion blog have complete, up-to-date details of the model.