Image-to-Image
Diffusers
Safetensors
PyTorch
flux2
flux
image-upscaling
latent-upscaling
super-resolution
diffusion
flow-matching
rectified-flow
generative-models
image-generation
Instructions to use MinhNH232331M/FlowUpscaler-diffusers with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use MinhNH232331M/FlowUpscaler-diffusers with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("MinhNH232331M/FlowUpscaler-diffusers", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- .vscode/settings.json +0 -4
- README.md +39 -23
- __pycache__/flow_upscaler_pipeline.cpython-313.pyc +0 -0
- __pycache__/upscaler_unet.cpython-313.pyc +0 -0
- config.json +0 -3
- flow_upscaler_pipeline.py +198 -46
- model_index.json +7 -0
- scheduler/scheduler_config.json +18 -0
- unet/config.json +9 -0
- unet/diffusion_pytorch_model.safetensors +3 -0
- upscaler_unet.py → unet/upscaler_unet.py +14 -1
.vscode/settings.json
DELETED
|
@@ -1,4 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"python-envs.defaultEnvManager": "ms-python.python:conda",
|
| 3 |
-
"python-envs.defaultPackageManager": "ms-python.python:conda"
|
| 4 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -13,7 +13,7 @@ tags:
|
|
| 13 |
- generative-models
|
| 14 |
- image-generation
|
| 15 |
- pytorch
|
| 16 |
-
library_name:
|
| 17 |
base_model:
|
| 18 |
- TensorForger/FlowUpscaler
|
| 19 |
pipeline_tag: image-to-image
|
|
@@ -31,11 +31,16 @@ Under the hood, it is a lightweight **Rectified Flow** model with **59M** parame
|
|
| 31 |
|
| 32 |
## Inference pipeline (`flow_upscaler_pipeline.py`)
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
`
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
1. **Encode** (`encode_image`): the input image is resized to a multiple of
|
| 41 |
16, VAE-encoded, and the 32-channel latents are normalized with the VAE's
|
|
@@ -44,35 +49,44 @@ Flux.2 VAE and runs the full loop:
|
|
| 44 |
in a **single FlowMatchEuler step**, conditioned on the small latent.
|
| 45 |
Passes chain for 4x/8x because a pass's output lives in the same
|
| 46 |
normalized latent space as its conditioning input.
|
| 47 |
-
3. **Decode + color fix** (`
|
| 48 |
VAE-decoded. With the default `color_fix=True`, each pass ends with a
|
| 49 |
latent low-frequency transplant and the decoded image gets a low-band
|
| 50 |
L/a/b transplant from the input — the two-stage drift correction detailed
|
| 51 |
in the next section.
|
| 52 |
|
| 53 |
Memory is bounded for large outputs: the VAE switches to tiling above 2048px,
|
| 54 |
-
the UNet runs through a chunked inference executor (`upscaler_unet.py`,
|
| 55 |
detailed in the next section), and outputs are capped at
|
| 56 |
`MAX_OUTPUT_SIDE = 16384` px per side (excess passes are dropped with a
|
| 57 |
warning). This keeps even 16K outputs within a 24GB-class GPU in bf16.
|
| 58 |
|
| 59 |
```python
|
| 60 |
import torch
|
| 61 |
-
from diffusers import AutoencoderKLFlux2
|
| 62 |
from PIL import Image
|
| 63 |
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
vae = AutoencoderKLFlux2.from_pretrained(
|
| 67 |
"black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
).to("cuda")
|
| 69 |
-
pipeline = FlowUpscalerPipeline("flow_upscaler.safetensors", vae, dtype=torch.bfloat16)
|
| 70 |
|
| 71 |
image = Image.open("input.png")
|
| 72 |
-
result = pipeline
|
| 73 |
result.save("output.png")
|
| 74 |
```
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
For the VAE, consider
|
| 77 |
[MageFlow-VAE-diffusers](https://huggingface.co/MinhNH232331M/MageFlow-VAE-diffusers)
|
| 78 |
— an efficient drop-in replacement for the Flux.2 VAE that operates in the
|
|
@@ -81,22 +95,24 @@ peak VRAM, with a chunked decoder that keeps memory near-constant at large
|
|
| 81 |
resolutions). Since upscaling cost here is dominated by the encode/decode
|
| 82 |
ends, it speeds up the whole loop.
|
| 83 |
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
`
|
| 88 |
-
|
| 89 |
-
|
|
|
|
| 90 |
|
| 91 |
|
| 92 |
-
## UNet inference optimizations (`upscaler_unet.py`)
|
| 93 |
|
| 94 |
The network is attention-free, so compute scales linearly with area — but the
|
| 95 |
stock forward's *memory* did not scale gracefully: at large latents each
|
| 96 |
residual block allocated 4-8 full-resolution temporaries (GroupNorm outputs,
|
| 97 |
SiLU copies, FiLM scale/shift maps, conv workspaces), and the skip-concat
|
| 98 |
-
up-block briefly held several 768-channel full-res tensors.
|
| 99 |
-
now layers four inference-only optimizations on the
|
|
|
|
| 100 |
are gated on `torch.is_grad_enabled()`, so training behavior is bit-for-bit
|
| 101 |
untouched.
|
| 102 |
|
|
|
|
| 13 |
- generative-models
|
| 14 |
- image-generation
|
| 15 |
- pytorch
|
| 16 |
+
library_name: diffusers
|
| 17 |
base_model:
|
| 18 |
- TensorForger/FlowUpscaler
|
| 19 |
pipeline_tag: image-to-image
|
|
|
|
| 31 |
|
| 32 |
## Inference pipeline (`flow_upscaler_pipeline.py`)
|
| 33 |
|
| 34 |
+
This repo is in **diffusers format**: `FlowUpscalerPipeline` is a
|
| 35 |
+
`DiffusionPipeline` whose components are the UNet (`unet/upscaler_unet.py`,
|
| 36 |
+
a diffusers `ModelMixin`, weights in
|
| 37 |
+
`unet/diffusion_pytorch_model.safetensors`), a shared Flux.2 VAE, and a
|
| 38 |
+
`FlowMatchEulerDiscreteScheduler` — the plain-Python equivalent of the
|
| 39 |
+
ComfyUI node, loadable with `DiffusionPipeline.from_pretrained`. The VAE is
|
| 40 |
+
intentionally not bundled (it is shared with any Flux.2 checkpoint, and
|
| 41 |
+
FLUX.2-dev is gated), so it is passed in at load time;
|
| 42 |
+
`trust_remote_code=True` is required because the pipeline and UNet classes
|
| 43 |
+
are loaded from this repo. The pipeline runs the full loop:
|
| 44 |
|
| 45 |
1. **Encode** (`encode_image`): the input image is resized to a multiple of
|
| 46 |
16, VAE-encoded, and the 32-channel latents are normalized with the VAE's
|
|
|
|
| 49 |
in a **single FlowMatchEuler step**, conditioned on the small latent.
|
| 50 |
Passes chain for 4x/8x because a pass's output lives in the same
|
| 51 |
normalized latent space as its conditioning input.
|
| 52 |
+
3. **Decode + color fix** (`__call__`): latents are denormalized and
|
| 53 |
VAE-decoded. With the default `color_fix=True`, each pass ends with a
|
| 54 |
latent low-frequency transplant and the decoded image gets a low-band
|
| 55 |
L/a/b transplant from the input — the two-stage drift correction detailed
|
| 56 |
in the next section.
|
| 57 |
|
| 58 |
Memory is bounded for large outputs: the VAE switches to tiling above 2048px,
|
| 59 |
+
the UNet runs through a chunked inference executor (`unet/upscaler_unet.py`,
|
| 60 |
detailed in the next section), and outputs are capped at
|
| 61 |
`MAX_OUTPUT_SIDE = 16384` px per side (excess passes are dropped with a
|
| 62 |
warning). This keeps even 16K outputs within a 24GB-class GPU in bf16.
|
| 63 |
|
| 64 |
```python
|
| 65 |
import torch
|
| 66 |
+
from diffusers import AutoencoderKLFlux2, DiffusionPipeline
|
| 67 |
from PIL import Image
|
| 68 |
|
| 69 |
+
vae = AutoencoderKLFlux2.from_pretrained( # any Flux.2 checkpoint's VAE
|
|
|
|
|
|
|
| 70 |
"black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
|
| 71 |
+
)
|
| 72 |
+
pipeline = DiffusionPipeline.from_pretrained(
|
| 73 |
+
"MinhNH232331M/FlowUpscaler-diffusers",
|
| 74 |
+
vae=vae, torch_dtype=torch.bfloat16, trust_remote_code=True,
|
| 75 |
).to("cuda")
|
|
|
|
| 76 |
|
| 77 |
image = Image.open("input.png")
|
| 78 |
+
result = pipeline(image, num_passes=2).images[0] # 2 passes = 4x
|
| 79 |
result.save("output.png")
|
| 80 |
```
|
| 81 |
|
| 82 |
+
Two compatibility paths are kept. `flow_upscaler.safetensors` at the repo
|
| 83 |
+
root is the same flat checkpoint the ComfyUI node consumes, and
|
| 84 |
+
`FlowUpscalerPipeline.from_single_file("flow_upscaler.safetensors", vae,
|
| 85 |
+
dtype=...)` replicates the pre-diffusers constructor for local checkouts.
|
| 86 |
+
The seed-based `pipeline.upscale_image(image, num_passes, seed=42, ...)`
|
| 87 |
+
method also remains, and a given seed reproduces the pre-conversion outputs
|
| 88 |
+
bit-for-bit (`__call__` takes a `generator` instead, diffusers-style).
|
| 89 |
+
|
| 90 |
For the VAE, consider
|
| 91 |
[MageFlow-VAE-diffusers](https://huggingface.co/MinhNH232331M/MageFlow-VAE-diffusers)
|
| 92 |
— an efficient drop-in replacement for the Flux.2 VAE that operates in the
|
|
|
|
| 95 |
resolutions). Since upscaling cost here is dominated by the encode/decode
|
| 96 |
ends, it speeds up the whole loop.
|
| 97 |
|
| 98 |
+
Pipeline call knobs: `num_passes` (2x per pass), `generator` (RNG for the
|
| 99 |
+
per-pass noise; `upscale_image` takes a `seed` instead), `color_fix` (master
|
| 100 |
+
switch for both correction stages), the per-stage `transplant_strength` /
|
| 101 |
+
`color_fix_strength` sliders described below, and `output_type` (`"pil"`
|
| 102 |
+
default, `"np"`, `"pt"`). `transplant_strength` also acts as a
|
| 103 |
+
fidelity/realism control — lower values track the input more faithfully,
|
| 104 |
+
higher values give the model's crisper rendition.
|
| 105 |
|
| 106 |
|
| 107 |
+
## UNet inference optimizations (`unet/upscaler_unet.py`)
|
| 108 |
|
| 109 |
The network is attention-free, so compute scales linearly with area — but the
|
| 110 |
stock forward's *memory* did not scale gracefully: at large latents each
|
| 111 |
residual block allocated 4-8 full-resolution temporaries (GroupNorm outputs,
|
| 112 |
SiLU copies, FiLM scale/shift maps, conv workspaces), and the skip-concat
|
| 113 |
+
up-block briefly held several 768-channel full-res tensors.
|
| 114 |
+
`unet/upscaler_unet.py` now layers four inference-only optimizations on the
|
| 115 |
+
same weights. All of them
|
| 116 |
are gated on `torch.is_grad_enabled()`, so training behavior is bit-for-bit
|
| 117 |
untouched.
|
| 118 |
|
__pycache__/flow_upscaler_pipeline.cpython-313.pyc
DELETED
|
Binary file (23.6 kB)
|
|
|
__pycache__/upscaler_unet.cpython-313.pyc
DELETED
|
Binary file (28.5 kB)
|
|
|
config.json
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model_type": "custom"
|
| 3 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
flow_upscaler_pipeline.py
CHANGED
|
@@ -1,5 +1,20 @@
|
|
| 1 |
"""Latent-space 2x upscaling with FlowUpscaler (single-step rectified flow).
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Inference convention replicated from the reference ComfyUI node
|
| 4 |
(github.com/TensorForger/comfyui-flow-upscaler) and the training notebooks in
|
| 5 |
github.com/tensorforger/CTGMWorkshop (notebooks/flow_upscaler): the UNet
|
|
@@ -14,9 +29,15 @@ pass ends with a low-frequency transplant from the conditioning latent
|
|
| 14 |
gets a low-band L/a/b transplant from the input (`_lab_stat_match`, fixes
|
| 15 |
the ~8%/pass chroma amplification and the tone-curve stretch the latent
|
| 16 |
transplant cannot see).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
"""
|
| 18 |
|
| 19 |
-
import
|
|
|
|
| 20 |
|
| 21 |
import numpy as np
|
| 22 |
import torch
|
|
@@ -24,11 +45,17 @@ import torch.nn.functional as F
|
|
| 24 |
from PIL import Image
|
| 25 |
from safetensors.torch import load_file
|
| 26 |
|
| 27 |
-
from diffusers import
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
-
logger = logging.
|
| 32 |
|
| 33 |
|
| 34 |
def patchify_latents(latents: torch.Tensor) -> torch.Tensor:
|
|
@@ -81,48 +108,96 @@ def _lab_to_rgb(lightness: torch.Tensor, a: torch.Tensor, b: torch.Tensor) -> to
|
|
| 81 |
).clamp(0.0, 1.0)
|
| 82 |
|
| 83 |
|
| 84 |
-
class FlowUpscalerPipeline:
|
| 85 |
-
"""
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
"""
|
| 91 |
|
|
|
|
|
|
|
| 92 |
# Encode/decode images larger than this (pixels per side) with VAE tiling
|
| 93 |
# to keep peak activation memory bounded on 24GB-class GPUs.
|
| 94 |
TILED_VAE_THRESHOLD = 2048
|
| 95 |
|
| 96 |
# Hard ceiling on the output side. Peak memory scales with latent *area*;
|
| 97 |
-
# with the chunked inference executor (upscaler_unet) the bf16
|
| 98 |
-
# measured 2.5GiB at an 8K-UHD output and 7.6GiB at 16K-UHD
|
| 99 |
-
# 5.5GiB), so even 16K fits a 23GB L4 alongside
|
| 100 |
# pipelines. fp32 needs roughly double.
|
| 101 |
MAX_OUTPUT_SIDE = 16384
|
| 102 |
|
| 103 |
def __init__(
|
| 104 |
self,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
model_path: str,
|
| 106 |
vae,
|
| 107 |
device: str = "cuda",
|
| 108 |
dtype: torch.dtype = torch.float32,
|
| 109 |
-
):
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
def _bn_stats(self) -> tuple[torch.Tensor, torch.Tensor]:
|
| 118 |
mean = self.vae.bn.running_mean.view(1, -1, 1, 1)
|
| 119 |
std = torch.sqrt(self.vae.bn.running_var.view(1, -1, 1, 1) + self.vae.config.batch_norm_eps)
|
| 120 |
-
|
|
|
|
| 121 |
|
| 122 |
def normalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
|
| 123 |
"""Raw VAE latents (B, 32, H, W) -> bn-normalized, still unpatchified."""
|
| 124 |
mean, std = self._bn_stats()
|
| 125 |
-
latents = patchify_latents(latents.to(self.
|
| 126 |
return unpatchify_latents((latents - mean) / std)
|
| 127 |
|
| 128 |
def denormalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
|
|
@@ -177,23 +252,23 @@ class FlowUpscalerPipeline:
|
|
| 177 |
) -> torch.Tensor:
|
| 178 |
"""One 2x pass in normalized latent space: (B, 32, H, W) -> (B, 32, 2H, 2W)."""
|
| 179 |
batch_size, _, height, width = latents_small.shape
|
| 180 |
-
|
|
|
|
| 181 |
|
| 182 |
-
scheduler
|
| 183 |
-
|
|
|
|
| 184 |
|
| 185 |
-
latents =
|
| 186 |
-
|
| 187 |
-
std=1.0,
|
| 188 |
-
size=(batch_size, 32, height * 2, width * 2),
|
| 189 |
-
dtype=self.dtype,
|
| 190 |
-
device=self.device,
|
| 191 |
generator=generator,
|
|
|
|
|
|
|
| 192 |
)
|
| 193 |
-
for t in scheduler.timesteps:
|
| 194 |
-
t = t.
|
| 195 |
velocity = self.unet(sample=latents, timestep=t, latents_small=latents_small)
|
| 196 |
-
latents = scheduler.step(velocity, t, latents).prev_sample
|
| 197 |
if color_fix:
|
| 198 |
latents = self._lowfreq_transplant(latents, latents_small, strength=transplant_strength)
|
| 199 |
return latents
|
|
@@ -213,7 +288,7 @@ class FlowUpscalerPipeline:
|
|
| 213 |
image = image.resize((width, height), Image.LANCZOS)
|
| 214 |
|
| 215 |
pixels = torch.from_numpy(np.array(image)).float().div(127.5).sub(1.0)
|
| 216 |
-
pixels = pixels.permute(2, 0, 1).unsqueeze(0).to(self.
|
| 217 |
previous_tiling = self.vae.use_tiling
|
| 218 |
if max(image.size) > self.TILED_VAE_THRESHOLD:
|
| 219 |
self.vae.use_tiling = True
|
|
@@ -306,15 +381,51 @@ class FlowUpscalerPipeline:
|
|
| 306 |
return out
|
| 307 |
|
| 308 |
@torch.no_grad()
|
| 309 |
-
def
|
| 310 |
self,
|
| 311 |
image: Image.Image,
|
| 312 |
num_passes: int = 1,
|
| 313 |
-
|
| 314 |
color_fix: bool = True,
|
| 315 |
transplant_strength: float = 1.0,
|
| 316 |
color_fix_strength: float = 1.0,
|
| 317 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
# Reduce passes if the result would exceed MAX_OUTPUT_SIDE (e.g. a
|
| 319 |
# 2048px generation with 2 passes requested runs only one). Checked
|
| 320 |
# before encoding so oversized inputs are rejected without GPU work.
|
|
@@ -334,7 +445,6 @@ class FlowUpscalerPipeline:
|
|
| 334 |
requested, num_passes, self.MAX_OUTPUT_SIDE,
|
| 335 |
)
|
| 336 |
|
| 337 |
-
generator = torch.Generator(device=self.device).manual_seed(seed)
|
| 338 |
latents = self.encode_image(image)
|
| 339 |
for _ in range(num_passes):
|
| 340 |
latents = self.upscale_latents(
|
|
@@ -342,9 +452,51 @@ class FlowUpscalerPipeline:
|
|
| 342 |
transplant_strength=transplant_strength,
|
| 343 |
)
|
| 344 |
decoded = self._decode_to_tensor(latents)
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
|
| 350 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""Latent-space 2x upscaling with FlowUpscaler (single-step rectified flow).
|
| 2 |
|
| 3 |
+
Diffusers-format pipeline. The components are the FlowUpscaler UNet
|
| 4 |
+
(`unet/upscaler_unet.py`, weights in `unet/diffusion_pytorch_model.safetensors`),
|
| 5 |
+
a shared Flux.2 VAE, and a `FlowMatchEulerDiscreteScheduler`. The VAE is not
|
| 6 |
+
bundled with this repo (it is shared with any Flux.2 checkpoint), so pass it
|
| 7 |
+
explicitly:
|
| 8 |
+
|
| 9 |
+
vae = AutoencoderKLFlux2.from_pretrained(
|
| 10 |
+
"black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
|
| 11 |
+
)
|
| 12 |
+
pipeline = DiffusionPipeline.from_pretrained(
|
| 13 |
+
"MinhNH232331M/FlowUpscaler-diffusers",
|
| 14 |
+
vae=vae, torch_dtype=torch.bfloat16, trust_remote_code=True,
|
| 15 |
+
).to("cuda")
|
| 16 |
+
image = pipeline(image, num_passes=2).images[0]
|
| 17 |
+
|
| 18 |
Inference convention replicated from the reference ComfyUI node
|
| 19 |
(github.com/TensorForger/comfyui-flow-upscaler) and the training notebooks in
|
| 20 |
github.com/tensorforger/CTGMWorkshop (notebooks/flow_upscaler): the UNet
|
|
|
|
| 29 |
gets a low-band L/a/b transplant from the input (`_lab_stat_match`, fixes
|
| 30 |
the ~8%/pass chroma amplification and the tone-curve stretch the latent
|
| 31 |
transplant cannot see).
|
| 32 |
+
|
| 33 |
+
This file is executed standalone by diffusers' remote-code loader, so it must
|
| 34 |
+
stay at the repo root and must not import sibling modules at the top level
|
| 35 |
+
(the UNet class arrives as an instantiated component; `from_single_file` loads
|
| 36 |
+
its module by explicit path).
|
| 37 |
"""
|
| 38 |
|
| 39 |
+
import importlib.util
|
| 40 |
+
import os
|
| 41 |
|
| 42 |
import numpy as np
|
| 43 |
import torch
|
|
|
|
| 45 |
from PIL import Image
|
| 46 |
from safetensors.torch import load_file
|
| 47 |
|
| 48 |
+
from diffusers import (
|
| 49 |
+
AutoencoderKLFlux2,
|
| 50 |
+
DiffusionPipeline,
|
| 51 |
+
FlowMatchEulerDiscreteScheduler,
|
| 52 |
+
ImagePipelineOutput,
|
| 53 |
+
ModelMixin,
|
| 54 |
+
)
|
| 55 |
+
from diffusers.utils import logging
|
| 56 |
+
from diffusers.utils.torch_utils import randn_tensor
|
| 57 |
|
| 58 |
+
logger = logging.get_logger(__name__)
|
| 59 |
|
| 60 |
|
| 61 |
def patchify_latents(latents: torch.Tensor) -> torch.Tensor:
|
|
|
|
| 108 |
).clamp(0.0, 1.0)
|
| 109 |
|
| 110 |
|
| 111 |
+
class FlowUpscalerPipeline(DiffusionPipeline):
|
| 112 |
+
r"""Image-in / image-out 2x latent upscaling with the FlowUpscaler UNet.
|
| 113 |
+
|
| 114 |
+
Each pass denoises pure noise into the 2x latent in a single
|
| 115 |
+
FlowMatchEuler step, conditioned on the input latent; passes chain for
|
| 116 |
+
4x/8x. The UNet defaults to float32 (the training dtype); bf16 matches it
|
| 117 |
+
to ~54 dB PSNR while halving the forward peak and runtime (pass
|
| 118 |
+
`torch_dtype=torch.bfloat16` to `from_pretrained`). The shared VAE is
|
| 119 |
+
used in its own dtype.
|
| 120 |
+
|
| 121 |
+
Args:
|
| 122 |
+
unet (`UpscalerUNet`):
|
| 123 |
+
Attention-free flow-matching UNet predicting velocity in the
|
| 124 |
+
bn-normalized, unpatchified Flux.2 latent space
|
| 125 |
+
(`unet/upscaler_unet.py` in this repo).
|
| 126 |
+
vae (`AutoencoderKLFlux2`):
|
| 127 |
+
A Flux.2 VAE (or a drop-in replacement operating in the same
|
| 128 |
+
latent space, e.g. MageFlow). Its BatchNorm running stats define
|
| 129 |
+
the latent normalization.
|
| 130 |
+
scheduler (`FlowMatchEulerDiscreteScheduler`):
|
| 131 |
+
Scheduler for the single Euler step of each pass.
|
| 132 |
"""
|
| 133 |
|
| 134 |
+
model_cpu_offload_seq = "unet->vae"
|
| 135 |
+
|
| 136 |
# Encode/decode images larger than this (pixels per side) with VAE tiling
|
| 137 |
# to keep peak activation memory bounded on 24GB-class GPUs.
|
| 138 |
TILED_VAE_THRESHOLD = 2048
|
| 139 |
|
| 140 |
# Hard ceiling on the output side. Peak memory scales with latent *area*;
|
| 141 |
+
# with the chunked inference executor (unet/upscaler_unet.py) the bf16
|
| 142 |
+
# UNet pass measured 2.5GiB at an 8K-UHD output and 7.6GiB at 16K-UHD
|
| 143 |
+
# (decode 5.5GiB), so even 16K fits a 23GB L4 alongside resident
|
| 144 |
# pipelines. fp32 needs roughly double.
|
| 145 |
MAX_OUTPUT_SIDE = 16384
|
| 146 |
|
| 147 |
def __init__(
|
| 148 |
self,
|
| 149 |
+
unet: ModelMixin, # an UpscalerUNet, loaded dynamically from unet/upscaler_unet.py
|
| 150 |
+
vae: AutoencoderKLFlux2,
|
| 151 |
+
scheduler: FlowMatchEulerDiscreteScheduler,
|
| 152 |
+
):
|
| 153 |
+
super().__init__()
|
| 154 |
+
self.register_modules(unet=unet, vae=vae, scheduler=scheduler)
|
| 155 |
+
|
| 156 |
+
@classmethod
|
| 157 |
+
def from_single_file(
|
| 158 |
+
cls,
|
| 159 |
model_path: str,
|
| 160 |
vae,
|
| 161 |
device: str = "cuda",
|
| 162 |
dtype: torch.dtype = torch.float32,
|
| 163 |
+
) -> "FlowUpscalerPipeline":
|
| 164 |
+
"""Build the pipeline from the flat `flow_upscaler.safetensors` checkpoint.
|
| 165 |
+
|
| 166 |
+
Mirrors the pre-diffusers constructor
|
| 167 |
+
``FlowUpscalerPipeline(model_path, vae, device, dtype)``. Needs
|
| 168 |
+
`unet/upscaler_unet.py` (or `upscaler_unet.py`) next to this file —
|
| 169 |
+
i.e. a local checkout of the repo; from the Hub use `from_pretrained`.
|
| 170 |
+
"""
|
| 171 |
+
here = os.path.dirname(os.path.abspath(__file__))
|
| 172 |
+
candidates = [
|
| 173 |
+
os.path.join(here, "unet", "upscaler_unet.py"),
|
| 174 |
+
os.path.join(here, "upscaler_unet.py"),
|
| 175 |
+
]
|
| 176 |
+
module_path = next((p for p in candidates if os.path.isfile(p)), None)
|
| 177 |
+
if module_path is None:
|
| 178 |
+
raise FileNotFoundError(
|
| 179 |
+
"from_single_file requires unet/upscaler_unet.py next to "
|
| 180 |
+
f"{__file__}; use FlowUpscalerPipeline.from_pretrained(...) instead."
|
| 181 |
+
)
|
| 182 |
+
spec = importlib.util.spec_from_file_location("upscaler_unet", module_path)
|
| 183 |
+
module = importlib.util.module_from_spec(spec)
|
| 184 |
+
spec.loader.exec_module(module)
|
| 185 |
+
|
| 186 |
+
unet = module.UpscalerUNet()
|
| 187 |
+
unet.load_state_dict(load_file(model_path))
|
| 188 |
+
unet.to(device, dtype).eval()
|
| 189 |
+
return cls(unet=unet, vae=vae, scheduler=FlowMatchEulerDiscreteScheduler())
|
| 190 |
|
| 191 |
def _bn_stats(self) -> tuple[torch.Tensor, torch.Tensor]:
|
| 192 |
mean = self.vae.bn.running_mean.view(1, -1, 1, 1)
|
| 193 |
std = torch.sqrt(self.vae.bn.running_var.view(1, -1, 1, 1) + self.vae.config.batch_norm_eps)
|
| 194 |
+
device, dtype = self._execution_device, self.unet.dtype
|
| 195 |
+
return mean.to(device, dtype), std.to(device, dtype)
|
| 196 |
|
| 197 |
def normalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
|
| 198 |
"""Raw VAE latents (B, 32, H, W) -> bn-normalized, still unpatchified."""
|
| 199 |
mean, std = self._bn_stats()
|
| 200 |
+
latents = patchify_latents(latents.to(self._execution_device, self.unet.dtype))
|
| 201 |
return unpatchify_latents((latents - mean) / std)
|
| 202 |
|
| 203 |
def denormalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
|
|
|
|
| 252 |
) -> torch.Tensor:
|
| 253 |
"""One 2x pass in normalized latent space: (B, 32, H, W) -> (B, 32, 2H, 2W)."""
|
| 254 |
batch_size, _, height, width = latents_small.shape
|
| 255 |
+
device, dtype = self._execution_device, self.unet.dtype
|
| 256 |
+
latents_small = latents_small.to(device, dtype)
|
| 257 |
|
| 258 |
+
# set_timesteps resets the scheduler's step state, so chained passes
|
| 259 |
+
# each run a fresh single-step schedule.
|
| 260 |
+
self.scheduler.set_timesteps(1, device=device, mu=1.0)
|
| 261 |
|
| 262 |
+
latents = randn_tensor(
|
| 263 |
+
(batch_size, self.unet.config.sample_channels, height * 2, width * 2),
|
|
|
|
|
|
|
|
|
|
|
|
|
| 264 |
generator=generator,
|
| 265 |
+
device=device,
|
| 266 |
+
dtype=dtype,
|
| 267 |
)
|
| 268 |
+
for t in self.scheduler.timesteps:
|
| 269 |
+
t = t.view(1)
|
| 270 |
velocity = self.unet(sample=latents, timestep=t, latents_small=latents_small)
|
| 271 |
+
latents = self.scheduler.step(velocity, t, latents).prev_sample
|
| 272 |
if color_fix:
|
| 273 |
latents = self._lowfreq_transplant(latents, latents_small, strength=transplant_strength)
|
| 274 |
return latents
|
|
|
|
| 288 |
image = image.resize((width, height), Image.LANCZOS)
|
| 289 |
|
| 290 |
pixels = torch.from_numpy(np.array(image)).float().div(127.5).sub(1.0)
|
| 291 |
+
pixels = pixels.permute(2, 0, 1).unsqueeze(0).to(self._execution_device, self.vae.dtype)
|
| 292 |
previous_tiling = self.vae.use_tiling
|
| 293 |
if max(image.size) > self.TILED_VAE_THRESHOLD:
|
| 294 |
self.vae.use_tiling = True
|
|
|
|
| 381 |
return out
|
| 382 |
|
| 383 |
@torch.no_grad()
|
| 384 |
+
def __call__(
|
| 385 |
self,
|
| 386 |
image: Image.Image,
|
| 387 |
num_passes: int = 1,
|
| 388 |
+
generator: torch.Generator | None = None,
|
| 389 |
color_fix: bool = True,
|
| 390 |
transplant_strength: float = 1.0,
|
| 391 |
color_fix_strength: float = 1.0,
|
| 392 |
+
output_type: str = "pil",
|
| 393 |
+
return_dict: bool = True,
|
| 394 |
+
) -> ImagePipelineOutput | tuple:
|
| 395 |
+
r"""Upscale `image` by 2x per pass.
|
| 396 |
+
|
| 397 |
+
Args:
|
| 398 |
+
image (`PIL.Image.Image`):
|
| 399 |
+
Input image. Resized to a multiple of 16 before encoding.
|
| 400 |
+
num_passes (`int`, defaults to 1):
|
| 401 |
+
Number of chained 2x passes (2 -> 4x, 3 -> 8x). Reduced
|
| 402 |
+
automatically (with a warning) if the output would exceed
|
| 403 |
+
`MAX_OUTPUT_SIDE` pixels per side.
|
| 404 |
+
generator (`torch.Generator`, *optional*):
|
| 405 |
+
RNG for the per-pass noise. Use a generator on the pipeline's
|
| 406 |
+
device for reproducible results.
|
| 407 |
+
color_fix (`bool`, defaults to `True`):
|
| 408 |
+
Master switch for both drift corrections (latent low-band
|
| 409 |
+
transplant each pass + pixel-space L/a/b low-band transplant
|
| 410 |
+
from the input after decoding).
|
| 411 |
+
transplant_strength (`float`, defaults to 1.0):
|
| 412 |
+
Latent-transplant stage; also a fidelity/realism control
|
| 413 |
+
(lower = truer to input, higher = crisper).
|
| 414 |
+
color_fix_strength (`float`, defaults to 1.0):
|
| 415 |
+
Pixel-space stage; residual color drift scales with
|
| 416 |
+
`1 - strength`.
|
| 417 |
+
output_type (`str`, defaults to `"pil"`):
|
| 418 |
+
`"pil"`, `"np"` (float32 in [0, 1], NHWC) or `"pt"` (float32
|
| 419 |
+
in [0, 1], NCHW, on the pipeline's device).
|
| 420 |
+
return_dict (`bool`, defaults to `True`):
|
| 421 |
+
Return an `ImagePipelineOutput` instead of a plain tuple.
|
| 422 |
+
|
| 423 |
+
Returns:
|
| 424 |
+
[`~pipelines.ImagePipelineOutput`] or `tuple`: the upscaled image.
|
| 425 |
+
"""
|
| 426 |
+
if output_type not in ("pil", "np", "pt"):
|
| 427 |
+
raise ValueError(f"`output_type` must be 'pil', 'np' or 'pt', got {output_type!r}.")
|
| 428 |
+
|
| 429 |
# Reduce passes if the result would exceed MAX_OUTPUT_SIDE (e.g. a
|
| 430 |
# 2048px generation with 2 passes requested runs only one). Checked
|
| 431 |
# before encoding so oversized inputs are rejected without GPU work.
|
|
|
|
| 445 |
requested, num_passes, self.MAX_OUTPUT_SIDE,
|
| 446 |
)
|
| 447 |
|
|
|
|
| 448 |
latents = self.encode_image(image)
|
| 449 |
for _ in range(num_passes):
|
| 450 |
latents = self.upscale_latents(
|
|
|
|
| 452 |
transplant_strength=transplant_strength,
|
| 453 |
)
|
| 454 |
decoded = self._decode_to_tensor(latents)
|
| 455 |
+
|
| 456 |
+
if color_fix:
|
| 457 |
+
reference = torch.from_numpy(np.asarray(image.convert("RGB")).copy())
|
| 458 |
+
reference = reference.to(self._execution_device).permute(2, 0, 1)[None].float().div_(255.0)
|
| 459 |
+
matched = self._lab_stat_match(decoded, reference, strength=float(color_fix_strength))
|
| 460 |
+
if output_type == "pil":
|
| 461 |
+
images = [Image.fromarray(matched.cpu().numpy())]
|
| 462 |
+
elif output_type == "np":
|
| 463 |
+
images = matched.float().div_(255.0)[None].cpu().numpy()
|
| 464 |
+
else:
|
| 465 |
+
images = matched.permute(2, 0, 1)[None].float().div_(255.0)
|
| 466 |
+
else:
|
| 467 |
+
if output_type == "pil":
|
| 468 |
+
images = [self._tensor_to_pil(decoded)]
|
| 469 |
+
elif output_type == "np":
|
| 470 |
+
images = decoded.permute(0, 2, 3, 1).float().cpu().numpy()
|
| 471 |
+
else:
|
| 472 |
+
images = decoded.float()
|
| 473 |
+
|
| 474 |
+
if not return_dict:
|
| 475 |
+
return (images,)
|
| 476 |
+
return ImagePipelineOutput(images=images)
|
| 477 |
+
|
| 478 |
+
@torch.no_grad()
|
| 479 |
+
def upscale_image(
|
| 480 |
+
self,
|
| 481 |
+
image: Image.Image,
|
| 482 |
+
num_passes: int = 1,
|
| 483 |
+
seed: int = 42,
|
| 484 |
+
color_fix: bool = True,
|
| 485 |
+
transplant_strength: float = 1.0,
|
| 486 |
+
color_fix_strength: float = 1.0,
|
| 487 |
+
) -> Image.Image:
|
| 488 |
+
"""Seed-based convenience wrapper around `__call__` returning a PIL image.
|
| 489 |
+
|
| 490 |
+
Kept API-compatible with the pre-diffusers pipeline: the generator is
|
| 491 |
+
created on the execution device, so a given seed reproduces the same
|
| 492 |
+
output as before.
|
| 493 |
+
"""
|
| 494 |
+
generator = torch.Generator(device=self._execution_device).manual_seed(seed)
|
| 495 |
+
return self(
|
| 496 |
+
image,
|
| 497 |
+
num_passes=num_passes,
|
| 498 |
+
generator=generator,
|
| 499 |
+
color_fix=color_fix,
|
| 500 |
+
transplant_strength=transplant_strength,
|
| 501 |
+
color_fix_strength=color_fix_strength,
|
| 502 |
+
).images[0]
|
model_index.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": ["flow_upscaler_pipeline", "FlowUpscalerPipeline"],
|
| 3 |
+
"_diffusers_version": "0.37.1",
|
| 4 |
+
"scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
|
| 5 |
+
"unet": ["upscaler_unet", "UpscalerUNet"],
|
| 6 |
+
"vae": ["diffusers", "AutoencoderKLFlux2"]
|
| 7 |
+
}
|
scheduler/scheduler_config.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "FlowMatchEulerDiscreteScheduler",
|
| 3 |
+
"_diffusers_version": "0.37.1",
|
| 4 |
+
"base_image_seq_len": 256,
|
| 5 |
+
"base_shift": 0.5,
|
| 6 |
+
"invert_sigmas": false,
|
| 7 |
+
"max_image_seq_len": 4096,
|
| 8 |
+
"max_shift": 1.15,
|
| 9 |
+
"num_train_timesteps": 1000,
|
| 10 |
+
"shift": 1.0,
|
| 11 |
+
"shift_terminal": null,
|
| 12 |
+
"stochastic_sampling": false,
|
| 13 |
+
"time_shift_type": "exponential",
|
| 14 |
+
"use_beta_sigmas": false,
|
| 15 |
+
"use_dynamic_shifting": false,
|
| 16 |
+
"use_exponential_sigmas": false,
|
| 17 |
+
"use_karras_sigmas": false
|
| 18 |
+
}
|
unet/config.json
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "UpscalerUNet",
|
| 3 |
+
"_diffusers_version": "0.37.1",
|
| 4 |
+
"base_channels": 384,
|
| 5 |
+
"cond_dim": 1024,
|
| 6 |
+
"dropout": 0.01,
|
| 7 |
+
"sample_channels": 32,
|
| 8 |
+
"time_dim": 512
|
| 9 |
+
}
|
unet/diffusion_pytorch_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7930adcbf8545c23fe60d5d1157de2fedfd497a0bed48018f315c110a3e3e330
|
| 3 |
+
size 237086000
|
upscaler_unet.py → unet/upscaler_unet.py
RENAMED
|
@@ -2,6 +2,9 @@ import torch
|
|
| 2 |
import torch.nn as nn
|
| 3 |
import torch.nn.functional as F
|
| 4 |
|
|
|
|
|
|
|
|
|
|
| 5 |
|
| 6 |
def make_group_norm(
|
| 7 |
channels: int, max_groups: int = 32, eps: float = 1e-6
|
|
@@ -485,7 +488,17 @@ class FilmCond2D(nn.Module):
|
|
| 485 |
return x.mul_(1 + scale).add_(shift)
|
| 486 |
|
| 487 |
|
| 488 |
-
class UpscalerUNet(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 489 |
def __init__(
|
| 490 |
self,
|
| 491 |
sample_channels: int = 32,
|
|
|
|
| 2 |
import torch.nn as nn
|
| 3 |
import torch.nn.functional as F
|
| 4 |
|
| 5 |
+
from diffusers.configuration_utils import ConfigMixin, register_to_config
|
| 6 |
+
from diffusers.models.modeling_utils import ModelMixin
|
| 7 |
+
|
| 8 |
|
| 9 |
def make_group_norm(
|
| 10 |
channels: int, max_groups: int = 32, eps: float = 1e-6
|
|
|
|
| 488 |
return x.mul_(1 + scale).add_(shift)
|
| 489 |
|
| 490 |
|
| 491 |
+
class UpscalerUNet(ModelMixin, ConfigMixin):
|
| 492 |
+
"""FlowUpscaler velocity-prediction UNet (diffusers `ModelMixin`).
|
| 493 |
+
|
| 494 |
+
Attention-free U-Net with SDXL-style residual blocks conditioned on the
|
| 495 |
+
timestep (FiLM from a sinusoidal embedding) and on the low-resolution
|
| 496 |
+
latents (multi-scale FiLM from `LowResEncoder`). The module tree is
|
| 497 |
+
identical to the original standalone `nn.Module`, so the state dict is
|
| 498 |
+
interchangeable with the flat `flow_upscaler.safetensors` checkpoint.
|
| 499 |
+
"""
|
| 500 |
+
|
| 501 |
+
@register_to_config
|
| 502 |
def __init__(
|
| 503 |
self,
|
| 504 |
sample_channels: int = 32,
|