Oysiyl/controlnet-lora-brightness-flux

FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses LoRA (Low-Rank Adaptation) combined with ControlNet architecture for efficient control, providing an ultra-lightweight control mechanism for FLUX's powerful 12B parameter model.

Model Description

This Control LoRA enables brightness-based conditioning for FLUX image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

Key Features:

• 🎨 Excellent brightness and pattern control with FLUX's superior quality
• 🚀 Compact model size: ~685MB per checkpoint
• ⚡ Fast inference: Leverages FLUX's optimized architecture
• 💡 Fixed conditioning strength: Baked into LoRA weights during training
• 🔄 Compatible with Diffusers: Uses standard FluxControlPipeline
• 📦 Multiple checkpoints: Track training progression at 25%, 50%, 75%, 100%
• 🖼️ Native FLUX resolution: Trained at 1024×1024
• 🎯 XLabs-proven parameters: Validated hyperparameters for stability

Intended Uses:

• Artistic QR code generation
• Image recoloring and colorization
• Lighting control in text-to-image generation
• Brightness-based pattern integration
• Watermark and subtle pattern embedding
• Photo enhancement and stylization

Training Details

Training Data

Trained on 10,000 samples from latentcat/grayscale_image_aesthetic_3M:

• High-quality aesthetic images
• Paired with grayscale/brightness versions
• Native resolution: 1024×1024

Training Configuration

Parameter	Value
Base Model	black-forest-labs/FLUX.1-dev
Model Size	12B parameters
Architecture	FLUX Control LoRA
LoRA Rank	128
Training Resolution	1024×1024
Training Steps	1,250 (1 epoch)
Batch Size	1 per device
Gradient Accumulation	8 (effective batch: 8)
Learning Rate	2e-5 constant (XLabs proven)
LR Warmup	10 steps (XLabs proven)
Weight Decay	0.01 (XLabs proven)
Max Grad Norm	1.0 (XLabs proven)
Empty Prompts	20%
Init Method	Default
Mixed Precision	BF16
Hardware	NVIDIA H100 80GB
Training Time	~3h 14min
Memory Usage	~45GB VRAM
Final Loss	~0.08-0.12

XLabs Proven Parameters

This model uses hyperparameters validated by XLabs-AI for FLUX ControlNet training:

• Learning Rate 2e-5: 5x lower than typical SDXL rates, critical for FLUX's 12B parameters
• Warmup 10 steps: Gradual LR increase for training stability
• Weight Decay 0.01: L2 regularization to prevent overfitting
• Max Grad Norm 1.0: Gradient clipping to prevent explosion

These parameters provide stable training and prevent divergence in large models.

Model Size Comparison

Model	Parameters	Size	Training	Resolution
This Control LoRA	~12B	~685MB	10k @ 1024	1024×1024
ControlNet (SDXL)	~700M	4.7GB	100k @ 512	512×512
T2I Adapter (SDXL)	~77M	302MB	100k @ 1024	1024×1024
SDXL Control LoRA	~7M	24MB	100k @ 1024	1024×1024

Usage

Installation

pip install diffusers transformers accelerate torch

Basic Usage

import torch
import sys
sys.path.insert(0, '/path/to/diffusers/src')

from diffusers import FluxControlPipeline
from PIL import Image

# Load FLUX Control Pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((1024, 1024))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,  # FLUX default
    guidance_scale=3.5,       # FLUX default
    height=1024,
    width=1024,
).images[0]

image.save("output.png")

Adjusting Control Strength

You can control the LoRA conditioning strength using set_adapters() with adapter_weights:

# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Now generate with adjusted strength
image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]

Recommended Scale Ranges:

• 0.5-0.7: Subtle control with hints of pattern
• 0.75-1.0: Balanced control (recommended for artistic QR codes)
• 1.0-1.25: Strong control with clear patterns
• 1.25-1.5: Very strong control

Note: Unlike SDXL ControlNet which uses controlnet_conditioning_scale, FLUX Control LoRA uses set_adapters() with adapter_weights to control conditioning strength.

Artistic QR Code Generation

import qrcode
from PIL import Image
import torch
from diffusers import FluxControlPipeline

# Generate QR code
qr = qrcode.QRCode(
    version=None,
    error_correction=qrcode.constants.ERROR_CORRECT_H,  # HIGH (30% tolerance)
    box_size=16,   # 16 pixels per module
    border=6,      # 6 modules border
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")

# Load pipeline with Control LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 2,500 samples, step 312)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-312")

# Mid checkpoint (50% - 5,000 samples, step 624)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-624")

# Late checkpoint (75% - 7,500 samples, step 936)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-936")

# Final model (10,000 samples, main branch - recommended)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")

Scale Comparison Grids

Visual comparison of different adapter weights (scales) for each checkpoint. Each grid shows the same prompt and QR code at scales 0.5, 0.75, 1.0, 1.25, and 1.5.

Checkpoint-312 (25% trained, 2,500 samples)

Scale progression: From subtle (0.5) to very strong (1.5) control. Early checkpoint shows dense pattern integration with visible QR structure across all scales.

Checkpoint-624 (50% trained, 5,000 samples)

Scale progression: Improved artistic balance at mid-training. Natural garden scenes emerge with adjustable pattern strength.

Checkpoint-936 (75% trained, 7,500 samples)

Scale progression: Refined artistic quality. Excellent detail preservation with smooth scale transitions.

Final Model (100% trained, 10,000 samples)

Scale progression: Best overall quality. Recommended scales: 0.75-1.0 for artistic QR codes, 1.0-1.25 for pattern integration.

Scale Selection Guide

Based on the comparison grids:

Scale	Control Strength	Best For	Visual Effect
0.5	Very subtle	Natural images, minimal pattern hints	Barely visible patterns, maximum artistic freedom
0.75	Light	Artistic images with soft pattern integration	Balanced artistic quality with gentle patterns
1.0	Balanced	Artistic QR codes (recommended)	Clear patterns with strong artistic elements
1.25	Strong	Pattern-guided art, structured designs	Dominant patterns with artistic overlay
1.5	Very strong	Technical applications, geometric patterns	Maximum pattern visibility, reduced artistic freedom

Checkpoint Progression

All checkpoints are included on the main branch to show training progression:

Checkpoint-312 (25% trained, 2,500 samples)

Early training stage - dense pattern integration with strong QR visibility.

Checkpoint-624 (50% trained, 5,000 samples)

Mid-training - improved artistic balance with natural perspective and composition.

Checkpoint-936 (75% trained, 7,500 samples)

Advanced training - refined floral composition with excellent detail and artistic quality.

Final Model (100% trained, 10,000 samples)

Final model - unique artistic interpretation with frame effect and natural scene integration.

Original QR Code

Validation QR code used during training (https://google.com, HIGH error correction).

Performance Comparison

vs SDXL Control LoRA

Metric	SDXL Control LoRA	This FLUX Control LoRA	Notes
Base Model	SDXL (2.6B)	FLUX.1-dev (12B)	4.6x larger model
Parameters	~7M	~12B	Full model parameters
Model Size	24MB	685MB	Includes full weights
LoRA Rank	16	128	8x higher rank
Training Samples	100k	10k	Fewer samples
Training Time	~21 min	~3h 14min	Larger model overhead
Variable Scale	✅ Yes (extra_condition_scale)	✅ Yes (set_adapters)	Different methods
Quality	Excellent	Superior	FLUX advantage

vs Full ControlNet

Metric	ControlNet (SDXL)	This FLUX Control LoRA	Advantage
Model Size	4.7GB	685MB	7x smaller
Storage (w/ checkpoints)	~18.8GB	~2.7GB	7x less storage
Training Time	~3 hours	3h 14min	Comparable
Flexibility	Variable scale	Fixed strength	Trade-off

When to Use This Model

✅ Use This FLUX Control LoRA When:

• Creating artistic images with FLUX's superior quality
• Need brightness-based control for FLUX generation
• Want production-ready artistic QR codes
• Working with FLUX.1-dev as base model
• Require checkpoint progression for analysis
• Need efficient storage vs full ControlNet

⚠️ Consider Alternatives When:

• Working with smaller datasets (SDXL trains faster with 10k samples)
• Need minimal model size (SDXL Control LoRA is 24MB vs 685MB)
• Require extremely precise control (use full ControlNet)
• Need multiple control types - train separate LoRAs

Limitations

Current Limitations

• FLUX-specific: Only works with FLUX.1-dev base model
• Grayscale conditioning only: Trained specifically for brightness/grayscale control
• Single control type: Only brightness, not other conditioning types (pose, depth, etc.)
• Larger model size: 685MB vs SDXL's 24MB (includes more parameters)
• Requires diffusers: Uses standard FluxControlPipeline from diffusers library

Recommendations

• Use set_adapters() with adapter_weights=[0.75-1.0] for best artistic QR results
• For multiple control types, train separate LoRAs
• Use final model for best results (recommended)
• Use checkpoint-936 (75%) for alternative artistic style
• Experiment with different scales (0.5-1.5) to find the right balance

Training Script

#!/bin/bash
set -e

# Configuration
LOCAL_DATASET_10K="$HOME/.cache/huggingface/datasets_partial/grayscale_10k/combined"
OUTPUT_DIR="$HOME/controlnet-lora-brightness-flux-10k"
VALIDATION_QR="$HOME/validation_qr.png"
CHECKPOINT_STEPS=78  # Checkpoint every 2,500 samples

# Change to training directory
cd diffusers/examples/flux-control

# Training with XLabs proven parameters
accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="$LOCAL_DATASET_10K" \
  --output_dir="$OUTPUT_DIR" \
  --mixed_precision="bf16" \
  --resolution=1024 \
  --learning_rate=2e-5 \
  --lr_scheduler=constant \
  --lr_warmup_steps=10 \
  --adam_weight_decay=0.01 \
  --max_grad_norm=1.0 \
  --proportion_empty_prompts=0.2 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=8 \
  --num_train_epochs=1 \
  --gradient_checkpointing \
  --checkpointing_steps=$CHECKPOINT_STEPS \
  --validation_steps=$CHECKPOINT_STEPS \
  --validation_image="$VALIDATION_QR" \
  --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \
  --num_validation_images=4 \
  --guidance_scale=3.5 \
  --seed=42 \
  --dataloader_num_workers=4 \
  --report_to="wandb" \
  --tracker_project_name="controlnet-lora-brightness-flux-10k"

Available Checkpoints

All checkpoints are available in the main branch:

• Root directory: Final model (10,000 samples, 1,250 steps, recommended)
• checkpoint-312/: Early checkpoint (2,500 samples, 25% trained)
• checkpoint-624/: Mid checkpoint (5,000 samples, 50% trained)
• checkpoint-936/: Late checkpoint (7,500 samples, 75% trained)

Key Differences: FLUX vs SDXL

Aspect	SDXL Control LoRA	FLUX Control LoRA
Model Size	2.6B params (UNet)	12B params
LoRA Rank	16	128 (8x higher)
Model File Size	~24MB	~685MB
Learning Rate	1e-4	2e-5 (XLabs proven)
Warmup Steps	0	10 (XLabs)
Weight Decay	Not used	0.01 (XLabs)
Max Grad Norm	Not used	1.0 (XLabs)
Variable Scale	✅ Yes (extra_condition_scale)	✅ Yes (set_adapters)
Inference Steps	30	28
Guidance Scale	7.5	3.5
Training Time (10k)	~21 min	~3h 14min
Memory (1024)	~40GB	~45GB

Troubleshooting

OOM (Out of Memory) Errors

If you encounter OOM during inference:

# Enable CPU offload
pipe.enable_model_cpu_offload()

# Or use sequential CPU offload (slower but less memory)
pipe.enable_sequential_cpu_offload()

Loading Errors

Make sure you have the latest diffusers:

pip install --upgrade diffusers transformers accelerate

Quality Issues

• Use guidance_scale=3.5 (FLUX default)
• Use num_inference_steps=28 or higher
• Try different checkpoints (final model recommended)
• Ensure control image is 1024×1024 resolution

Citation

@misc{controlnet-lora-brightness-flux,
  author = {Oysiyl},
  title = {FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)},
  year = {2026},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-flux}}
}

Acknowledgments

• Built with 🤗 Diffusers
• Base model: FLUX.1-dev by Black Forest Labs
• XLabs proven parameters: x-flux by XLabs-AI
• Dataset: grayscale_image_aesthetic_3M by latentcat
• Training infrastructure: NVIDIA H100 80GB
• LoRA implementation: PEFT by Hugging Face

License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at black-forest-labs/FLUX.1-dev.