ControlNet: controlling the composition of your images

#The problem it solves

Text is a poor language for geometry. You describe a subject, style, and mood in a prompt, but words can’t lock down the position of a hand, the angle of a building, or the exact silhouette of a product. Rerun with a different seed and the composition changes completely. ControlNet changes the input, not the prompt: you provide an image that encodes the desired structure, the model is conditioned on it at every denoising step, and the prompt governs everything else.

The method comes from a research paper: “Adding Conditional Control to Text-to-Image Diffusion Models,” by Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, published in the ICCV 2023 proceedings and awarded the conference’s best paper prize, the Marr Prize. Its abstract describes the principle: ControlNet locks the diffusion model, reuses its pretrained encoding layers as a foundation, and connects the trainable copy using “zero convolutions”—convolution layers initialized to zero that gradually grow the parameters and prevent harmful noise from disrupting training. The authors also show that training remains robust with small datasets of fewer than 50,000 images, as well as with large datasets of more than one million.

What this changes for you: you add a control to an existing model without retraining it, and each type of control is a separate file. That’s also why you need the right file for the right model family.

#Choose the control type

The official ControlNet 1.1 repository lists 14 models, including 11 production-ready and 3 experimental models, all named according to a single rule. For depth, it lists three supported preprocessors: Depth_Midas, Depth_Leres, and Depth_Zoe. A preprocessor is the tool that transforms your reference image into a control map, such as turning a photo into a depth map.

The practical rule: choose the loosest control that still captures what matters to you. Conditioning on outlines in a photograph reproduces the photograph: if that's the desired result, generation was unnecessary. Depth or pose generally provide a better balance between adherence and creativity.

#The settings that matter

Names vary by tool, but three settings appear everywhere. The table gives their names in the three most common environments, based on their documentation.

Strength is the most effective lever: near the maximum, the result is rigid, and lowering the strength fixes most frozen-looking images. Stopping the control before denoising finishes locks in the composition while leaving the model free to render materials and light. The AUTOMATIC1111 extension adds a three-position Control Mode: Balanced, My prompt is more important, and ControlNet is more important, to arbitrate between the prompt and the control.

#Starting values published by the model authors

Rather than making up numbers, start with the ones provided by the publishers. They vary significantly from one family to another, which is a reminder that a setting that works for SD 1.5 cannot simply be carried over.

#Which ControlNet for which family

The landscape has changed significantly since SD 1.5, and many guides are outdated. The table distinguishes what the model lab publishes from what the community publishes, because this affects maintenance and licensing.

Two points deserve mention. First, Black Forest Labs has marked FLUX.1 Depth and FLUX.1 Canny as deprecated for its API, including the LoRA versions: the open weights remain available on Hugging Face, but the publisher no longer maintains them. Second, “Union” models combine several modes in a single file, saving disk space at the cost of variable maturity: InstantX’s ControlNet Union listing for Flux describes it as a first beta version that may not be fully trained.

The license does not automatically follow the base model: each control file has its own, and it varies from one file to another.

#What it costs in files and memory

A ControlNet is an additional network that runs alongside the base model: it adds VRAM usage and time per image. File sizes, published on Hugging Face, provide the first-order estimate: they are added to the weights of the base model and its text encoder.

Two takeaways. First, a control module in LoRA form is a fraction of the size of a full model: Stability AI presents its Control-LoRA models as a way to bring control to a wider range of consumer graphics cards. Second, watch the format: ControlNet 1.1 .pth files are Python pickles, which Hugging Face flags with a pickle-import warning. Prefer .safetensors versions, which cannot execute code.

How much additional VRAM? Vendors do not publish a general figure, and the measurement depends on the model and resolution. If memory is saturated, the remedies should be applied in this order: lower the resolution, reduce the number of simultaneous controls, then switch to a lighter control, such as a Control-LoRA or a half-precision version. For baseline memory, see the VRAM guide.

• VRAM: what it is and how much you need for AI

#Where to use it

ControlNet is not a standalone application: it runs inside an image-generation interface. In ComfyUI, two nodes do the work: Load ControlNet, which reads files from the models/controlnet folder, and Apply ControlNet, which receives the control image and the settings from the table above. The ComfyUI documentation states that the software core does not include all preprocessors: you often need an extension such as ComfyUI ControlNet aux. Remember that an extension is code executed with your permissions: install only extensions you know. For Z-Image Turbo, place the control file in a different folder, models/model_patches.

In the classic web interface, AUTOMATIC1111's sd-webui-controlnet extension adds a panel below the prompt. Its latest release, version 1.1.454, dates from July 2024, as does its latest commit: it still works with SD 1.5 and SDXL, but will not keep up with newer models. Forge integrates ControlNet directly, according to its README, without using this extension.

#Combine multiple controls at once

Nothing prevents you from stacking two controls, such as a pose control for the character’s posture and depth for the background. The diffusers documentation gives the rule: for good results, mask the conditioning inputs so they do not overlap, and experiment with different strengths to adjust the weight of each control. Its SDXL example combines canny and depth at 0.5 each. In ComfyUI, chain Apply ControlNet nodes. The same principles apply to both: stack only controls that address different aspects of the image, and lower each control’s strength because the effects add up.

#When control is ignored

The control does not appear in the list

The file is in the wrong folder, or its configuration file doesn't have the same name as the file itself. In ComfyUI, the folder is models/controlnet.

The output ignores the reference

A mismatch between the base model family and the control family, a preprocessor that produced an empty map, or a strength that is too low. Display the control map itself before suspecting anything else.

The images are static and lifeless

Strength too high, or control applied throughout the denoising process. Lower the strength and stop the control before the end.

Disappointing result with a Union model

The authors of InstantX's Union write that even a fully trained Union model can be worse than specialized models, such as the pose model. Try a dedicated ControlNet, and write a detailed prompt, as Shakker Labs' model card recommends.

Memory saturated

Lower the resolution, remove one control, or use a lighter version of the control.

• ComfyUI: install the node-based interface
• AUTOMATIC1111: the classic web interface
• Overview of local image generation
• The AI image toolkit: ControlNet workflows and settings by graphics card
• Source: the official ControlNet repository
• Source: the original research paper, ICCV 2023
• Source: Stability AI’s Control-LoRA models for SDXL
• Source: FLUX.1 Tools and the deprecation of Depth and Canny

#FAQ