Run Stable Diffusion Locally on Bare Metal

Stable Diffusion is a latent text-to-image diffusion model—it generates images from text prompts using a 860M UNet and CLIP ViT-L/14 text encoder. The model runs on consumer GPUs with at least 10GB VRAM and produces 512×512 images by default. Running it locally means your prompts stay on your hardware, no API keys, no cloud bills, and you control the entire pipeline.

Prerequisites

• Linux or macOS system with Python 3.8+
• NVIDIA or AMD GPU with ≥10GB VRAM (NVIDIA strongly recommended for stability)
• 20–30 GB free disk space for model weights and conda environment
• conda package manager installed
• Internet connection to download model weights from Hugging Face (one-time)

Step 1: Clone the Repository and Set Up Conda Environment

git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion
conda env create -f environment.yaml
conda activate ldm

This creates a conda environment named ldm with PyTorch and all dependencies pinned to known-good versions.

Step 2: Install Additional Dependencies

If you’re updating an existing environment or want to ensure fresh packages:

conda install pytorch torchvision -c pytorch
pip install transformers==4.19.2 diffusers invisible-watermark
pip install -e .

The -e flag installs the local repository in editable mode, so changes to the code take effect immediately.

Step 3: Obtain Model Weights

Download the Stable Diffusion v1 checkpoint from Hugging Face. You’ll need a Hugging Face account and must accept the model license at https://huggingface.co/CompVis/stable-diffusion-v1-4.

Log in via the CLI:

huggingface-cli login

Enter your Hugging Face token when prompted. The token is saved to ~/.huggingface/token.

Step 4: Prepare Model Directory

Create the model directory structure:

mkdir -p models/ldm/stable-diffusion-v1/

The diffusers library will automatically download and cache the model weights here on first run. Alternatively, you can download the checkpoint manually and symlink it:

ln -s <path/to/model.ckpt> models/ldm/stable-diffusion-v1/model.ckpt

Step 5: Generate Your First Image

Use the reference sampling script with a text prompt:

python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms

Key flags:

• --prompt: The text description of what you want to generate
• --plms: Use the PLMS sampler (faster, deterministic)
• --scale: Guidance scale (default 7.5; higher values follow the prompt more strictly)
• --ddim_steps: Number of sampling steps (default 50; fewer steps = faster but lower quality)
• --H and --W: Image height and width in pixels (default 512×512)
• --n_samples: Batch size—how many images to generate per prompt
• --seed: Set a seed for reproducible results
• --outdir: Directory to save generated images (default outputs/txt2img-samples)

Generated images are saved to the output directory with a grid preview and individual samples.

Step 6: Use the Diffusers Library for Simpler Integration

For Python scripts and integrations, the diffusers library is simpler than the reference script. Create a file generate.py:

from torch import autocast
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    use_auth_token=True
).to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
with autocast("cuda"):
    image = pipe(prompt)["sample"][0]

image.save("astronaut_rides_horse.png")

Run it:

python generate.py

The first run downloads the model (~4 GB); subsequent runs use the cached weights.

Step 7: Image-to-Image Modification (Optional)

Stable Diffusion can also modify existing images. Provide a sketch or photo and a text prompt:

python scripts/img2img.py --prompt "A fantasy landscape, trending on artstation" --init-img path/to/sketch.jpg --strength 0.8

The --strength parameter controls how much the model changes the input (0.0 = no change, 1.0 = complete regeneration).

Performance Tuning

• Memory: If you hit out-of-memory errors, reduce --n_samples or use --precision autocast for lower precision (faster, slightly lower quality).
• Speed: Reduce --ddim_steps from 50 to 20–30 for faster generation at the cost of quality. PLMS sampling is faster than DDIM.
• Quality: Increase --scale to 8.0–10.0 for stricter adherence to the prompt; use 5.0–6.0 for more creative variation.

Important Notes

Keep this service on your LAN or behind a VPN. Do not expose the generation endpoint to the public internet. If you build a web UI around this, use firewall rules or a reverse proxy with authentication.

Stable Diffusion v1 was trained on a large internet dataset and can produce biased, low-quality, or inappropriate outputs. The model includes a safety checker to reduce explicit content, but it is not perfect. Review the model card at https://huggingface.co/CompVis/stable-diffusion-v1-4 for detailed limitations and ethical considerations.

Generated images are watermarked with an invisible watermark to help identify them as machine-generated.

Is It Worth It?

Yes, if you value privacy, want to avoid API costs, or need deterministic local generation for automation. A single GPU generates images in 20–60 seconds depending on step count and hardware. You’ll spend a few hours setting up the environment and downloading weights, but after that it’s instant and free. The tradeoff: you own the infrastructure and troubleshooting. For casual use, cloud APIs are simpler; for serious local AI work, this is the foundation.

Sources

• https://github.com/CompVis/stable-diffusion
• https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion
• https://huggingface.co/CompVis/stable-diffusion-v1-4