MinhNH232331M commited on
Commit
2043d66
·
verified ·
1 Parent(s): 2d15c10

Upload folder using huggingface_hub

Browse files
.vscode/settings.json DELETED
@@ -1,4 +0,0 @@
1
- {
2
- "python-envs.defaultEnvManager": "ms-python.python:conda",
3
- "python-envs.defaultPackageManager": "ms-python.python:conda"
4
- }
 
 
 
 
 
README.md CHANGED
@@ -13,7 +13,7 @@ tags:
13
  - generative-models
14
  - image-generation
15
  - pytorch
16
- library_name: pytorch
17
  base_model:
18
  - TensorForger/FlowUpscaler
19
  pipeline_tag: image-to-image
@@ -31,11 +31,16 @@ Under the hood, it is a lightweight **Rectified Flow** model with **59M** parame
31
 
32
  ## Inference pipeline (`flow_upscaler_pipeline.py`)
33
 
34
- `flow_upscaler_pipeline.py` is a standalone, diffusers-based image-in /
35
- image-out wrapper around the model — the plain-Python equivalent of the
36
- ComfyUI node. `FlowUpscalerPipeline` combines the UNet defined in
37
- `upscaler_unet.py` (weights: `flow_upscaler.safetensors`) with a shared
38
- Flux.2 VAE and runs the full loop:
 
 
 
 
 
39
 
40
  1. **Encode** (`encode_image`): the input image is resized to a multiple of
41
  16, VAE-encoded, and the 32-channel latents are normalized with the VAE's
@@ -44,35 +49,44 @@ Flux.2 VAE and runs the full loop:
44
  in a **single FlowMatchEuler step**, conditioned on the small latent.
45
  Passes chain for 4x/8x because a pass's output lives in the same
46
  normalized latent space as its conditioning input.
47
- 3. **Decode + color fix** (`upscale_image`): latents are denormalized and
48
  VAE-decoded. With the default `color_fix=True`, each pass ends with a
49
  latent low-frequency transplant and the decoded image gets a low-band
50
  L/a/b transplant from the input — the two-stage drift correction detailed
51
  in the next section.
52
 
53
  Memory is bounded for large outputs: the VAE switches to tiling above 2048px,
54
- the UNet runs through a chunked inference executor (`upscaler_unet.py`,
55
  detailed in the next section), and outputs are capped at
56
  `MAX_OUTPUT_SIDE = 16384` px per side (excess passes are dropped with a
57
  warning). This keeps even 16K outputs within a 24GB-class GPU in bf16.
58
 
59
  ```python
60
  import torch
61
- from diffusers import AutoencoderKLFlux2 # any Flux.2 checkpoint's VAE
62
  from PIL import Image
63
 
64
- from flow_upscaler_pipeline import FlowUpscalerPipeline
65
-
66
- vae = AutoencoderKLFlux2.from_pretrained(
67
  "black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
 
 
 
 
68
  ).to("cuda")
69
- pipeline = FlowUpscalerPipeline("flow_upscaler.safetensors", vae, dtype=torch.bfloat16)
70
 
71
  image = Image.open("input.png")
72
- result = pipeline.upscale_image(image, num_passes=2) # 2 passes = 4x
73
  result.save("output.png")
74
  ```
75
 
 
 
 
 
 
 
 
 
76
  For the VAE, consider
77
  [MageFlow-VAE-diffusers](https://huggingface.co/MinhNH232331M/MageFlow-VAE-diffusers)
78
  — an efficient drop-in replacement for the Flux.2 VAE that operates in the
@@ -81,22 +95,24 @@ peak VRAM, with a chunked decoder that keeps memory near-constant at large
81
  resolutions). Since upscaling cost here is dominated by the encode/decode
82
  ends, it speeds up the whole loop.
83
 
84
- `upscale_image` knobs: `num_passes` (2x per pass), `seed`, `color_fix`
85
- (master switch for both correction stages), and the per-stage
86
- `transplant_strength` / `color_fix_strength` sliders described below.
87
- `transplant_strength` also acts as a fidelity/realism control — lower values
88
- track the input more faithfully, higher values give the model's crisper
89
- rendition.
 
90
 
91
 
92
- ## UNet inference optimizations (`upscaler_unet.py`)
93
 
94
  The network is attention-free, so compute scales linearly with area — but the
95
  stock forward's *memory* did not scale gracefully: at large latents each
96
  residual block allocated 4-8 full-resolution temporaries (GroupNorm outputs,
97
  SiLU copies, FiLM scale/shift maps, conv workspaces), and the skip-concat
98
- up-block briefly held several 768-channel full-res tensors. `upscaler_unet.py`
99
- now layers four inference-only optimizations on the same weights. All of them
 
100
  are gated on `torch.is_grad_enabled()`, so training behavior is bit-for-bit
101
  untouched.
102
 
 
13
  - generative-models
14
  - image-generation
15
  - pytorch
16
+ library_name: diffusers
17
  base_model:
18
  - TensorForger/FlowUpscaler
19
  pipeline_tag: image-to-image
 
31
 
32
  ## Inference pipeline (`flow_upscaler_pipeline.py`)
33
 
34
+ This repo is in **diffusers format**: `FlowUpscalerPipeline` is a
35
+ `DiffusionPipeline` whose components are the UNet (`unet/upscaler_unet.py`,
36
+ a diffusers `ModelMixin`, weights in
37
+ `unet/diffusion_pytorch_model.safetensors`), a shared Flux.2 VAE, and a
38
+ `FlowMatchEulerDiscreteScheduler` — the plain-Python equivalent of the
39
+ ComfyUI node, loadable with `DiffusionPipeline.from_pretrained`. The VAE is
40
+ intentionally not bundled (it is shared with any Flux.2 checkpoint, and
41
+ FLUX.2-dev is gated), so it is passed in at load time;
42
+ `trust_remote_code=True` is required because the pipeline and UNet classes
43
+ are loaded from this repo. The pipeline runs the full loop:
44
 
45
  1. **Encode** (`encode_image`): the input image is resized to a multiple of
46
  16, VAE-encoded, and the 32-channel latents are normalized with the VAE's
 
49
  in a **single FlowMatchEuler step**, conditioned on the small latent.
50
  Passes chain for 4x/8x because a pass's output lives in the same
51
  normalized latent space as its conditioning input.
52
+ 3. **Decode + color fix** (`__call__`): latents are denormalized and
53
  VAE-decoded. With the default `color_fix=True`, each pass ends with a
54
  latent low-frequency transplant and the decoded image gets a low-band
55
  L/a/b transplant from the input — the two-stage drift correction detailed
56
  in the next section.
57
 
58
  Memory is bounded for large outputs: the VAE switches to tiling above 2048px,
59
+ the UNet runs through a chunked inference executor (`unet/upscaler_unet.py`,
60
  detailed in the next section), and outputs are capped at
61
  `MAX_OUTPUT_SIDE = 16384` px per side (excess passes are dropped with a
62
  warning). This keeps even 16K outputs within a 24GB-class GPU in bf16.
63
 
64
  ```python
65
  import torch
66
+ from diffusers import AutoencoderKLFlux2, DiffusionPipeline
67
  from PIL import Image
68
 
69
+ vae = AutoencoderKLFlux2.from_pretrained( # any Flux.2 checkpoint's VAE
 
 
70
  "black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
71
+ )
72
+ pipeline = DiffusionPipeline.from_pretrained(
73
+ "MinhNH232331M/FlowUpscaler-diffusers",
74
+ vae=vae, torch_dtype=torch.bfloat16, trust_remote_code=True,
75
  ).to("cuda")
 
76
 
77
  image = Image.open("input.png")
78
+ result = pipeline(image, num_passes=2).images[0] # 2 passes = 4x
79
  result.save("output.png")
80
  ```
81
 
82
+ Two compatibility paths are kept. `flow_upscaler.safetensors` at the repo
83
+ root is the same flat checkpoint the ComfyUI node consumes, and
84
+ `FlowUpscalerPipeline.from_single_file("flow_upscaler.safetensors", vae,
85
+ dtype=...)` replicates the pre-diffusers constructor for local checkouts.
86
+ The seed-based `pipeline.upscale_image(image, num_passes, seed=42, ...)`
87
+ method also remains, and a given seed reproduces the pre-conversion outputs
88
+ bit-for-bit (`__call__` takes a `generator` instead, diffusers-style).
89
+
90
  For the VAE, consider
91
  [MageFlow-VAE-diffusers](https://huggingface.co/MinhNH232331M/MageFlow-VAE-diffusers)
92
  — an efficient drop-in replacement for the Flux.2 VAE that operates in the
 
95
  resolutions). Since upscaling cost here is dominated by the encode/decode
96
  ends, it speeds up the whole loop.
97
 
98
+ Pipeline call knobs: `num_passes` (2x per pass), `generator` (RNG for the
99
+ per-pass noise; `upscale_image` takes a `seed` instead), `color_fix` (master
100
+ switch for both correction stages), the per-stage `transplant_strength` /
101
+ `color_fix_strength` sliders described below, and `output_type` (`"pil"`
102
+ default, `"np"`, `"pt"`). `transplant_strength` also acts as a
103
+ fidelity/realism control — lower values track the input more faithfully,
104
+ higher values give the model's crisper rendition.
105
 
106
 
107
+ ## UNet inference optimizations (`unet/upscaler_unet.py`)
108
 
109
  The network is attention-free, so compute scales linearly with area — but the
110
  stock forward's *memory* did not scale gracefully: at large latents each
111
  residual block allocated 4-8 full-resolution temporaries (GroupNorm outputs,
112
  SiLU copies, FiLM scale/shift maps, conv workspaces), and the skip-concat
113
+ up-block briefly held several 768-channel full-res tensors.
114
+ `unet/upscaler_unet.py` now layers four inference-only optimizations on the
115
+ same weights. All of them
116
  are gated on `torch.is_grad_enabled()`, so training behavior is bit-for-bit
117
  untouched.
118
 
__pycache__/flow_upscaler_pipeline.cpython-313.pyc DELETED
Binary file (23.6 kB)
 
__pycache__/upscaler_unet.cpython-313.pyc DELETED
Binary file (28.5 kB)
 
config.json DELETED
@@ -1,3 +0,0 @@
1
- {
2
- "model_type": "custom"
3
- }
 
 
 
 
flow_upscaler_pipeline.py CHANGED
@@ -1,5 +1,20 @@
1
  """Latent-space 2x upscaling with FlowUpscaler (single-step rectified flow).
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  Inference convention replicated from the reference ComfyUI node
4
  (github.com/TensorForger/comfyui-flow-upscaler) and the training notebooks in
5
  github.com/tensorforger/CTGMWorkshop (notebooks/flow_upscaler): the UNet
@@ -14,9 +29,15 @@ pass ends with a low-frequency transplant from the conditioning latent
14
  gets a low-band L/a/b transplant from the input (`_lab_stat_match`, fixes
15
  the ~8%/pass chroma amplification and the tone-curve stretch the latent
16
  transplant cannot see).
 
 
 
 
 
17
  """
18
 
19
- import logging
 
20
 
21
  import numpy as np
22
  import torch
@@ -24,11 +45,17 @@ import torch.nn.functional as F
24
  from PIL import Image
25
  from safetensors.torch import load_file
26
 
27
- from diffusers import FlowMatchEulerDiscreteScheduler
28
-
29
- from upscaler_unet import UpscalerUNet
 
 
 
 
 
 
30
 
31
- logger = logging.getLogger(__name__)
32
 
33
 
34
  def patchify_latents(latents: torch.Tensor) -> torch.Tensor:
@@ -81,48 +108,96 @@ def _lab_to_rgb(lightness: torch.Tensor, a: torch.Tensor, b: torch.Tensor) -> to
81
  ).clamp(0.0, 1.0)
82
 
83
 
84
- class FlowUpscalerPipeline:
85
- """Wraps UpscalerUNet + the Flux.2 VAE for image-in / image-out 2x upscaling.
86
-
87
- The UNet defaults to float32 (the training dtype); bf16 matches it to
88
- ~54 dB PSNR while halving the forward peak and runtime (app_local passes
89
- bf16). The shared VAE is used as-is (bf16 in this project).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
  """
91
 
 
 
92
  # Encode/decode images larger than this (pixels per side) with VAE tiling
93
  # to keep peak activation memory bounded on 24GB-class GPUs.
94
  TILED_VAE_THRESHOLD = 2048
95
 
96
  # Hard ceiling on the output side. Peak memory scales with latent *area*;
97
- # with the chunked inference executor (upscaler_unet) the bf16 UNet pass
98
- # measured 2.5GiB at an 8K-UHD output and 7.6GiB at 16K-UHD (decode
99
- # 5.5GiB), so even 16K fits a 23GB L4 alongside the app's resident
100
  # pipelines. fp32 needs roughly double.
101
  MAX_OUTPUT_SIDE = 16384
102
 
103
  def __init__(
104
  self,
 
 
 
 
 
 
 
 
 
 
105
  model_path: str,
106
  vae,
107
  device: str = "cuda",
108
  dtype: torch.dtype = torch.float32,
109
- ):
110
- self.unet = UpscalerUNet()
111
- self.unet.load_state_dict(load_file(model_path))
112
- self.unet.to(device, dtype).eval()
113
- self.vae = vae
114
- self.device = device
115
- self.dtype = dtype
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
 
117
  def _bn_stats(self) -> tuple[torch.Tensor, torch.Tensor]:
118
  mean = self.vae.bn.running_mean.view(1, -1, 1, 1)
119
  std = torch.sqrt(self.vae.bn.running_var.view(1, -1, 1, 1) + self.vae.config.batch_norm_eps)
120
- return mean.to(self.device, self.dtype), std.to(self.device, self.dtype)
 
121
 
122
  def normalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
123
  """Raw VAE latents (B, 32, H, W) -> bn-normalized, still unpatchified."""
124
  mean, std = self._bn_stats()
125
- latents = patchify_latents(latents.to(self.device, self.dtype))
126
  return unpatchify_latents((latents - mean) / std)
127
 
128
  def denormalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
@@ -177,23 +252,23 @@ class FlowUpscalerPipeline:
177
  ) -> torch.Tensor:
178
  """One 2x pass in normalized latent space: (B, 32, H, W) -> (B, 32, 2H, 2W)."""
179
  batch_size, _, height, width = latents_small.shape
180
- latents_small = latents_small.to(self.device, self.dtype)
 
181
 
182
- scheduler = FlowMatchEulerDiscreteScheduler()
183
- scheduler.set_timesteps(1, mu=1.0)
 
184
 
185
- latents = torch.normal(
186
- mean=0.0,
187
- std=1.0,
188
- size=(batch_size, 32, height * 2, width * 2),
189
- dtype=self.dtype,
190
- device=self.device,
191
  generator=generator,
 
 
192
  )
193
- for t in scheduler.timesteps:
194
- t = t.to(self.device).view(1)
195
  velocity = self.unet(sample=latents, timestep=t, latents_small=latents_small)
196
- latents = scheduler.step(velocity, t, latents).prev_sample
197
  if color_fix:
198
  latents = self._lowfreq_transplant(latents, latents_small, strength=transplant_strength)
199
  return latents
@@ -213,7 +288,7 @@ class FlowUpscalerPipeline:
213
  image = image.resize((width, height), Image.LANCZOS)
214
 
215
  pixels = torch.from_numpy(np.array(image)).float().div(127.5).sub(1.0)
216
- pixels = pixels.permute(2, 0, 1).unsqueeze(0).to(self.device, self.vae.dtype)
217
  previous_tiling = self.vae.use_tiling
218
  if max(image.size) > self.TILED_VAE_THRESHOLD:
219
  self.vae.use_tiling = True
@@ -306,15 +381,51 @@ class FlowUpscalerPipeline:
306
  return out
307
 
308
  @torch.no_grad()
309
- def upscale_image(
310
  self,
311
  image: Image.Image,
312
  num_passes: int = 1,
313
- seed: int = 42,
314
  color_fix: bool = True,
315
  transplant_strength: float = 1.0,
316
  color_fix_strength: float = 1.0,
317
- ) -> Image.Image:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
318
  # Reduce passes if the result would exceed MAX_OUTPUT_SIDE (e.g. a
319
  # 2048px generation with 2 passes requested runs only one). Checked
320
  # before encoding so oversized inputs are rejected without GPU work.
@@ -334,7 +445,6 @@ class FlowUpscalerPipeline:
334
  requested, num_passes, self.MAX_OUTPUT_SIDE,
335
  )
336
 
337
- generator = torch.Generator(device=self.device).manual_seed(seed)
338
  latents = self.encode_image(image)
339
  for _ in range(num_passes):
340
  latents = self.upscale_latents(
@@ -342,9 +452,51 @@ class FlowUpscalerPipeline:
342
  transplant_strength=transplant_strength,
343
  )
344
  decoded = self._decode_to_tensor(latents)
345
- if not color_fix:
346
- return self._tensor_to_pil(decoded)
347
- reference = torch.from_numpy(np.asarray(image.convert("RGB")).copy())
348
- reference = reference.to(self.device).permute(2, 0, 1)[None].float().div_(255.0)
349
- matched = self._lab_stat_match(decoded, reference, strength=float(color_fix_strength))
350
- return Image.fromarray(matched.cpu().numpy())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  """Latent-space 2x upscaling with FlowUpscaler (single-step rectified flow).
2
 
3
+ Diffusers-format pipeline. The components are the FlowUpscaler UNet
4
+ (`unet/upscaler_unet.py`, weights in `unet/diffusion_pytorch_model.safetensors`),
5
+ a shared Flux.2 VAE, and a `FlowMatchEulerDiscreteScheduler`. The VAE is not
6
+ bundled with this repo (it is shared with any Flux.2 checkpoint), so pass it
7
+ explicitly:
8
+
9
+ vae = AutoencoderKLFlux2.from_pretrained(
10
+ "black-forest-labs/FLUX.2-dev", subfolder="vae", torch_dtype=torch.bfloat16
11
+ )
12
+ pipeline = DiffusionPipeline.from_pretrained(
13
+ "MinhNH232331M/FlowUpscaler-diffusers",
14
+ vae=vae, torch_dtype=torch.bfloat16, trust_remote_code=True,
15
+ ).to("cuda")
16
+ image = pipeline(image, num_passes=2).images[0]
17
+
18
  Inference convention replicated from the reference ComfyUI node
19
  (github.com/TensorForger/comfyui-flow-upscaler) and the training notebooks in
20
  github.com/tensorforger/CTGMWorkshop (notebooks/flow_upscaler): the UNet
 
29
  gets a low-band L/a/b transplant from the input (`_lab_stat_match`, fixes
30
  the ~8%/pass chroma amplification and the tone-curve stretch the latent
31
  transplant cannot see).
32
+
33
+ This file is executed standalone by diffusers' remote-code loader, so it must
34
+ stay at the repo root and must not import sibling modules at the top level
35
+ (the UNet class arrives as an instantiated component; `from_single_file` loads
36
+ its module by explicit path).
37
  """
38
 
39
+ import importlib.util
40
+ import os
41
 
42
  import numpy as np
43
  import torch
 
45
  from PIL import Image
46
  from safetensors.torch import load_file
47
 
48
+ from diffusers import (
49
+ AutoencoderKLFlux2,
50
+ DiffusionPipeline,
51
+ FlowMatchEulerDiscreteScheduler,
52
+ ImagePipelineOutput,
53
+ ModelMixin,
54
+ )
55
+ from diffusers.utils import logging
56
+ from diffusers.utils.torch_utils import randn_tensor
57
 
58
+ logger = logging.get_logger(__name__)
59
 
60
 
61
  def patchify_latents(latents: torch.Tensor) -> torch.Tensor:
 
108
  ).clamp(0.0, 1.0)
109
 
110
 
111
+ class FlowUpscalerPipeline(DiffusionPipeline):
112
+ r"""Image-in / image-out 2x latent upscaling with the FlowUpscaler UNet.
113
+
114
+ Each pass denoises pure noise into the 2x latent in a single
115
+ FlowMatchEuler step, conditioned on the input latent; passes chain for
116
+ 4x/8x. The UNet defaults to float32 (the training dtype); bf16 matches it
117
+ to ~54 dB PSNR while halving the forward peak and runtime (pass
118
+ `torch_dtype=torch.bfloat16` to `from_pretrained`). The shared VAE is
119
+ used in its own dtype.
120
+
121
+ Args:
122
+ unet (`UpscalerUNet`):
123
+ Attention-free flow-matching UNet predicting velocity in the
124
+ bn-normalized, unpatchified Flux.2 latent space
125
+ (`unet/upscaler_unet.py` in this repo).
126
+ vae (`AutoencoderKLFlux2`):
127
+ A Flux.2 VAE (or a drop-in replacement operating in the same
128
+ latent space, e.g. MageFlow). Its BatchNorm running stats define
129
+ the latent normalization.
130
+ scheduler (`FlowMatchEulerDiscreteScheduler`):
131
+ Scheduler for the single Euler step of each pass.
132
  """
133
 
134
+ model_cpu_offload_seq = "unet->vae"
135
+
136
  # Encode/decode images larger than this (pixels per side) with VAE tiling
137
  # to keep peak activation memory bounded on 24GB-class GPUs.
138
  TILED_VAE_THRESHOLD = 2048
139
 
140
  # Hard ceiling on the output side. Peak memory scales with latent *area*;
141
+ # with the chunked inference executor (unet/upscaler_unet.py) the bf16
142
+ # UNet pass measured 2.5GiB at an 8K-UHD output and 7.6GiB at 16K-UHD
143
+ # (decode 5.5GiB), so even 16K fits a 23GB L4 alongside resident
144
  # pipelines. fp32 needs roughly double.
145
  MAX_OUTPUT_SIDE = 16384
146
 
147
  def __init__(
148
  self,
149
+ unet: ModelMixin, # an UpscalerUNet, loaded dynamically from unet/upscaler_unet.py
150
+ vae: AutoencoderKLFlux2,
151
+ scheduler: FlowMatchEulerDiscreteScheduler,
152
+ ):
153
+ super().__init__()
154
+ self.register_modules(unet=unet, vae=vae, scheduler=scheduler)
155
+
156
+ @classmethod
157
+ def from_single_file(
158
+ cls,
159
  model_path: str,
160
  vae,
161
  device: str = "cuda",
162
  dtype: torch.dtype = torch.float32,
163
+ ) -> "FlowUpscalerPipeline":
164
+ """Build the pipeline from the flat `flow_upscaler.safetensors` checkpoint.
165
+
166
+ Mirrors the pre-diffusers constructor
167
+ ``FlowUpscalerPipeline(model_path, vae, device, dtype)``. Needs
168
+ `unet/upscaler_unet.py` (or `upscaler_unet.py`) next to this file —
169
+ i.e. a local checkout of the repo; from the Hub use `from_pretrained`.
170
+ """
171
+ here = os.path.dirname(os.path.abspath(__file__))
172
+ candidates = [
173
+ os.path.join(here, "unet", "upscaler_unet.py"),
174
+ os.path.join(here, "upscaler_unet.py"),
175
+ ]
176
+ module_path = next((p for p in candidates if os.path.isfile(p)), None)
177
+ if module_path is None:
178
+ raise FileNotFoundError(
179
+ "from_single_file requires unet/upscaler_unet.py next to "
180
+ f"{__file__}; use FlowUpscalerPipeline.from_pretrained(...) instead."
181
+ )
182
+ spec = importlib.util.spec_from_file_location("upscaler_unet", module_path)
183
+ module = importlib.util.module_from_spec(spec)
184
+ spec.loader.exec_module(module)
185
+
186
+ unet = module.UpscalerUNet()
187
+ unet.load_state_dict(load_file(model_path))
188
+ unet.to(device, dtype).eval()
189
+ return cls(unet=unet, vae=vae, scheduler=FlowMatchEulerDiscreteScheduler())
190
 
191
  def _bn_stats(self) -> tuple[torch.Tensor, torch.Tensor]:
192
  mean = self.vae.bn.running_mean.view(1, -1, 1, 1)
193
  std = torch.sqrt(self.vae.bn.running_var.view(1, -1, 1, 1) + self.vae.config.batch_norm_eps)
194
+ device, dtype = self._execution_device, self.unet.dtype
195
+ return mean.to(device, dtype), std.to(device, dtype)
196
 
197
  def normalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
198
  """Raw VAE latents (B, 32, H, W) -> bn-normalized, still unpatchified."""
199
  mean, std = self._bn_stats()
200
+ latents = patchify_latents(latents.to(self._execution_device, self.unet.dtype))
201
  return unpatchify_latents((latents - mean) / std)
202
 
203
  def denormalize_latents(self, latents: torch.Tensor) -> torch.Tensor:
 
252
  ) -> torch.Tensor:
253
  """One 2x pass in normalized latent space: (B, 32, H, W) -> (B, 32, 2H, 2W)."""
254
  batch_size, _, height, width = latents_small.shape
255
+ device, dtype = self._execution_device, self.unet.dtype
256
+ latents_small = latents_small.to(device, dtype)
257
 
258
+ # set_timesteps resets the scheduler's step state, so chained passes
259
+ # each run a fresh single-step schedule.
260
+ self.scheduler.set_timesteps(1, device=device, mu=1.0)
261
 
262
+ latents = randn_tensor(
263
+ (batch_size, self.unet.config.sample_channels, height * 2, width * 2),
 
 
 
 
264
  generator=generator,
265
+ device=device,
266
+ dtype=dtype,
267
  )
268
+ for t in self.scheduler.timesteps:
269
+ t = t.view(1)
270
  velocity = self.unet(sample=latents, timestep=t, latents_small=latents_small)
271
+ latents = self.scheduler.step(velocity, t, latents).prev_sample
272
  if color_fix:
273
  latents = self._lowfreq_transplant(latents, latents_small, strength=transplant_strength)
274
  return latents
 
288
  image = image.resize((width, height), Image.LANCZOS)
289
 
290
  pixels = torch.from_numpy(np.array(image)).float().div(127.5).sub(1.0)
291
+ pixels = pixels.permute(2, 0, 1).unsqueeze(0).to(self._execution_device, self.vae.dtype)
292
  previous_tiling = self.vae.use_tiling
293
  if max(image.size) > self.TILED_VAE_THRESHOLD:
294
  self.vae.use_tiling = True
 
381
  return out
382
 
383
  @torch.no_grad()
384
+ def __call__(
385
  self,
386
  image: Image.Image,
387
  num_passes: int = 1,
388
+ generator: torch.Generator | None = None,
389
  color_fix: bool = True,
390
  transplant_strength: float = 1.0,
391
  color_fix_strength: float = 1.0,
392
+ output_type: str = "pil",
393
+ return_dict: bool = True,
394
+ ) -> ImagePipelineOutput | tuple:
395
+ r"""Upscale `image` by 2x per pass.
396
+
397
+ Args:
398
+ image (`PIL.Image.Image`):
399
+ Input image. Resized to a multiple of 16 before encoding.
400
+ num_passes (`int`, defaults to 1):
401
+ Number of chained 2x passes (2 -> 4x, 3 -> 8x). Reduced
402
+ automatically (with a warning) if the output would exceed
403
+ `MAX_OUTPUT_SIDE` pixels per side.
404
+ generator (`torch.Generator`, *optional*):
405
+ RNG for the per-pass noise. Use a generator on the pipeline's
406
+ device for reproducible results.
407
+ color_fix (`bool`, defaults to `True`):
408
+ Master switch for both drift corrections (latent low-band
409
+ transplant each pass + pixel-space L/a/b low-band transplant
410
+ from the input after decoding).
411
+ transplant_strength (`float`, defaults to 1.0):
412
+ Latent-transplant stage; also a fidelity/realism control
413
+ (lower = truer to input, higher = crisper).
414
+ color_fix_strength (`float`, defaults to 1.0):
415
+ Pixel-space stage; residual color drift scales with
416
+ `1 - strength`.
417
+ output_type (`str`, defaults to `"pil"`):
418
+ `"pil"`, `"np"` (float32 in [0, 1], NHWC) or `"pt"` (float32
419
+ in [0, 1], NCHW, on the pipeline's device).
420
+ return_dict (`bool`, defaults to `True`):
421
+ Return an `ImagePipelineOutput` instead of a plain tuple.
422
+
423
+ Returns:
424
+ [`~pipelines.ImagePipelineOutput`] or `tuple`: the upscaled image.
425
+ """
426
+ if output_type not in ("pil", "np", "pt"):
427
+ raise ValueError(f"`output_type` must be 'pil', 'np' or 'pt', got {output_type!r}.")
428
+
429
  # Reduce passes if the result would exceed MAX_OUTPUT_SIDE (e.g. a
430
  # 2048px generation with 2 passes requested runs only one). Checked
431
  # before encoding so oversized inputs are rejected without GPU work.
 
445
  requested, num_passes, self.MAX_OUTPUT_SIDE,
446
  )
447
 
 
448
  latents = self.encode_image(image)
449
  for _ in range(num_passes):
450
  latents = self.upscale_latents(
 
452
  transplant_strength=transplant_strength,
453
  )
454
  decoded = self._decode_to_tensor(latents)
455
+
456
+ if color_fix:
457
+ reference = torch.from_numpy(np.asarray(image.convert("RGB")).copy())
458
+ reference = reference.to(self._execution_device).permute(2, 0, 1)[None].float().div_(255.0)
459
+ matched = self._lab_stat_match(decoded, reference, strength=float(color_fix_strength))
460
+ if output_type == "pil":
461
+ images = [Image.fromarray(matched.cpu().numpy())]
462
+ elif output_type == "np":
463
+ images = matched.float().div_(255.0)[None].cpu().numpy()
464
+ else:
465
+ images = matched.permute(2, 0, 1)[None].float().div_(255.0)
466
+ else:
467
+ if output_type == "pil":
468
+ images = [self._tensor_to_pil(decoded)]
469
+ elif output_type == "np":
470
+ images = decoded.permute(0, 2, 3, 1).float().cpu().numpy()
471
+ else:
472
+ images = decoded.float()
473
+
474
+ if not return_dict:
475
+ return (images,)
476
+ return ImagePipelineOutput(images=images)
477
+
478
+ @torch.no_grad()
479
+ def upscale_image(
480
+ self,
481
+ image: Image.Image,
482
+ num_passes: int = 1,
483
+ seed: int = 42,
484
+ color_fix: bool = True,
485
+ transplant_strength: float = 1.0,
486
+ color_fix_strength: float = 1.0,
487
+ ) -> Image.Image:
488
+ """Seed-based convenience wrapper around `__call__` returning a PIL image.
489
+
490
+ Kept API-compatible with the pre-diffusers pipeline: the generator is
491
+ created on the execution device, so a given seed reproduces the same
492
+ output as before.
493
+ """
494
+ generator = torch.Generator(device=self._execution_device).manual_seed(seed)
495
+ return self(
496
+ image,
497
+ num_passes=num_passes,
498
+ generator=generator,
499
+ color_fix=color_fix,
500
+ transplant_strength=transplant_strength,
501
+ color_fix_strength=color_fix_strength,
502
+ ).images[0]
model_index.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": ["flow_upscaler_pipeline", "FlowUpscalerPipeline"],
3
+ "_diffusers_version": "0.37.1",
4
+ "scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
5
+ "unet": ["upscaler_unet", "UpscalerUNet"],
6
+ "vae": ["diffusers", "AutoencoderKLFlux2"]
7
+ }
scheduler/scheduler_config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "FlowMatchEulerDiscreteScheduler",
3
+ "_diffusers_version": "0.37.1",
4
+ "base_image_seq_len": 256,
5
+ "base_shift": 0.5,
6
+ "invert_sigmas": false,
7
+ "max_image_seq_len": 4096,
8
+ "max_shift": 1.15,
9
+ "num_train_timesteps": 1000,
10
+ "shift": 1.0,
11
+ "shift_terminal": null,
12
+ "stochastic_sampling": false,
13
+ "time_shift_type": "exponential",
14
+ "use_beta_sigmas": false,
15
+ "use_dynamic_shifting": false,
16
+ "use_exponential_sigmas": false,
17
+ "use_karras_sigmas": false
18
+ }
unet/config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "UpscalerUNet",
3
+ "_diffusers_version": "0.37.1",
4
+ "base_channels": 384,
5
+ "cond_dim": 1024,
6
+ "dropout": 0.01,
7
+ "sample_channels": 32,
8
+ "time_dim": 512
9
+ }
unet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7930adcbf8545c23fe60d5d1157de2fedfd497a0bed48018f315c110a3e3e330
3
+ size 237086000
upscaler_unet.py → unet/upscaler_unet.py RENAMED
@@ -2,6 +2,9 @@ import torch
2
  import torch.nn as nn
3
  import torch.nn.functional as F
4
 
 
 
 
5
 
6
  def make_group_norm(
7
  channels: int, max_groups: int = 32, eps: float = 1e-6
@@ -485,7 +488,17 @@ class FilmCond2D(nn.Module):
485
  return x.mul_(1 + scale).add_(shift)
486
 
487
 
488
- class UpscalerUNet(nn.Module):
 
 
 
 
 
 
 
 
 
 
489
  def __init__(
490
  self,
491
  sample_channels: int = 32,
 
2
  import torch.nn as nn
3
  import torch.nn.functional as F
4
 
5
+ from diffusers.configuration_utils import ConfigMixin, register_to_config
6
+ from diffusers.models.modeling_utils import ModelMixin
7
+
8
 
9
  def make_group_norm(
10
  channels: int, max_groups: int = 32, eps: float = 1e-6
 
488
  return x.mul_(1 + scale).add_(shift)
489
 
490
 
491
+ class UpscalerUNet(ModelMixin, ConfigMixin):
492
+ """FlowUpscaler velocity-prediction UNet (diffusers `ModelMixin`).
493
+
494
+ Attention-free U-Net with SDXL-style residual blocks conditioned on the
495
+ timestep (FiLM from a sinusoidal embedding) and on the low-resolution
496
+ latents (multi-scale FiLM from `LowResEncoder`). The module tree is
497
+ identical to the original standalone `nn.Module`, so the state dict is
498
+ interchangeable with the flat `flow_upscaler.safetensors` checkpoint.
499
+ """
500
+
501
+ @register_to_config
502
  def __init__(
503
  self,
504
  sample_channels: int = 32,