BiliSakura commited on
Commit
ba1e0da
·
verified ·
1 Parent(s): 5aa0066

Sync updated code and configs (no weight re-upload)

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +160 -0
  2. decur-mit-b2-hha/config.json +30 -0
  3. decur-mit-b2-hha/dat_blocks.py +679 -0
  4. decur-mit-b2-hha/image_processing_decur.py +160 -0
  5. decur-mit-b2-hha/modeling_decur.py +285 -0
  6. decur-mit-b2-hha/pipeline_decur.py +58 -0
  7. decur-mit-b2-hha/preprocessor_config.json +15 -0
  8. decur-mit-b2-rgb/__pycache__/modeling_decur.cpython-311.pyc +0 -0
  9. decur-mit-b2-rgb/config.json +30 -0
  10. decur-mit-b2-rgb/dat_blocks.py +679 -0
  11. decur-mit-b2-rgb/image_processing_decur.py +160 -0
  12. decur-mit-b2-rgb/modeling_decur.py +285 -0
  13. decur-mit-b2-rgb/pipeline_decur.py +58 -0
  14. decur-mit-b2-rgb/preprocessor_config.json +15 -0
  15. decur-mit-b5-hha/config.json +30 -0
  16. decur-mit-b5-hha/dat_blocks.py +679 -0
  17. decur-mit-b5-hha/image_processing_decur.py +160 -0
  18. decur-mit-b5-hha/modeling_decur.py +285 -0
  19. decur-mit-b5-hha/pipeline_decur.py +58 -0
  20. decur-mit-b5-hha/preprocessor_config.json +15 -0
  21. decur-mit-b5-rgb/config.json +30 -0
  22. decur-mit-b5-rgb/dat_blocks.py +679 -0
  23. decur-mit-b5-rgb/image_processing_decur.py +160 -0
  24. decur-mit-b5-rgb/modeling_decur.py +285 -0
  25. decur-mit-b5-rgb/pipeline_decur.py +58 -0
  26. decur-mit-b5-rgb/preprocessor_config.json +15 -0
  27. decur-resnet50-dem/config.json +30 -0
  28. decur-resnet50-dem/dat_blocks.py +679 -0
  29. decur-resnet50-dem/image_processing_decur.py +160 -0
  30. decur-resnet50-dem/modeling_decur.py +285 -0
  31. decur-resnet50-dem/pipeline_decur.py +58 -0
  32. decur-resnet50-dem/preprocessor_config.json +15 -0
  33. decur-resnet50-rda-dem/config.json +30 -0
  34. decur-resnet50-rda-dem/dat_blocks.py +679 -0
  35. decur-resnet50-rda-dem/image_processing_decur.py +160 -0
  36. decur-resnet50-rda-dem/modeling_decur.py +285 -0
  37. decur-resnet50-rda-dem/pipeline_decur.py +58 -0
  38. decur-resnet50-rda-dem/preprocessor_config.json +15 -0
  39. decur-resnet50-rda-rgb/config.json +30 -0
  40. decur-resnet50-rda-rgb/dat_blocks.py +679 -0
  41. decur-resnet50-rda-rgb/image_processing_decur.py +160 -0
  42. decur-resnet50-rda-rgb/modeling_decur.py +285 -0
  43. decur-resnet50-rda-rgb/pipeline_decur.py +58 -0
  44. decur-resnet50-rda-rgb/preprocessor_config.json +15 -0
  45. decur-resnet50-rda-s1/config.json +30 -0
  46. decur-resnet50-rda-s1/dat_blocks.py +679 -0
  47. decur-resnet50-rda-s1/image_processing_decur.py +160 -0
  48. decur-resnet50-rda-s1/modeling_decur.py +285 -0
  49. decur-resnet50-rda-s1/pipeline_decur.py +58 -0
  50. decur-resnet50-rda-s1/preprocessor_config.json +15 -0
README.md ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - remote-sensing
7
+ - earth-observation
8
+ - self-supervised-learning
9
+ - multispectral
10
+ - sar
11
+ - rgb
12
+ - depth
13
+ - feature-extraction
14
+ - decur
15
+ - resnet
16
+ - vit
17
+ - segformer
18
+ - transformers
19
+ library_name: transformers
20
+ pipeline_tag: feature-extraction
21
+ ---
22
+
23
+ # DeCUR Transformers Models
24
+
25
+ Self-contained Hugging Face checkpoints converted from the official [DeCUR](https://arxiv.org/abs/2312.10132) pretrained weights. Each subfolder ships remote code (`modeling_decur.py`, processor, pipeline) and loads with `trust_remote_code=True`.
26
+
27
+ **Original paper:** [DeCUR: Decoupling Common and Unique Representations for Multimodal Self-Supervision](https://arxiv.org/abs/2312.10132)
28
+ **Converted for Hugging Face by:** BiliSakura
29
+
30
+ ## Available checkpoints (16 models)
31
+
32
+ | Folder | Backbone | Modality | Channels | RDA |
33
+ |--------|----------|----------|----------|-----|
34
+ | `decur-resnet50-s1` | ResNet-50 | Sentinel-1 SAR | 2 | no |
35
+ | `decur-resnet50-s2c` | ResNet-50 | Sentinel-2 L1C | 13 | no |
36
+ | `decur-resnet50-rda-s1` | ResNet-50 + RDA | Sentinel-1 SAR | 2 | yes |
37
+ | `decur-resnet50-rda-s2c` | ResNet-50 + RDA | Sentinel-2 L1C | 13 | yes |
38
+ | `decur-vit-small-patch16-s1` | ViT-S/16 | Sentinel-1 SAR | 2 | no |
39
+ | `decur-vit-small-patch16-s2c` | ViT-S/16 | Sentinel-2 L1C | 13 | no |
40
+ | `decur-resnet50-rgb` | ResNet-50 | GeoNRW RGB | 3 | no |
41
+ | `decur-resnet50-dem` | ResNet-50 | GeoNRW DEM | 3 | no |
42
+ | `decur-resnet50-rda-rgb` | ResNet-50 + RDA | GeoNRW RGB | 3 | yes |
43
+ | `decur-resnet50-rda-dem` | ResNet-50 + RDA | GeoNRW DEM | 3 | yes |
44
+ | `decur-vit-small-patch16-rgb` | ViT-S/16 | GeoNRW RGB | 3 | no |
45
+ | `decur-vit-small-patch16-dem` | ViT-S/16 | GeoNRW DEM | 3 | no |
46
+ | `decur-mit-b2-rgb` | SegFormer MiT-B2 | SUN RGB-D RGB | 3 | no |
47
+ | `decur-mit-b2-hha` | SegFormer MiT-B2 | SUN RGB-D HHA | 3 | no |
48
+ | `decur-mit-b5-rgb` | SegFormer MiT-B5 | SUN RGB-D RGB | 3 | no |
49
+ | `decur-mit-b5-hha` | SegFormer MiT-B5 | SUN RGB-D HHA | 3 | no |
50
+
51
+ Legacy `.pth` filename mapping is in [`conversion_manifest.json`](conversion_manifest.json).
52
+
53
+ ## Usage
54
+
55
+ Processors default to **`do_resize: false`** — pass images at native `(H, W, C)`; only value rescaling (`/255`) is applied unless you enable normalization.
56
+
57
+ ```python
58
+ from transformers import pipeline
59
+ import numpy as np
60
+
61
+ REPO = "/path/to/DECUR-transformers"
62
+ SUBFOLDER = "decur-resnet50-s2c"
63
+
64
+ pipe = pipeline(
65
+ task="decur-feature-extraction",
66
+ model=REPO,
67
+ trust_remote_code=True,
68
+ model_kwargs={"subfolder": SUBFOLDER},
69
+ )
70
+
71
+ # Native Sentinel-2 patch (e.g. 512×512, 13 bands)
72
+ image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
73
+ features = pipe(image, pool=True, return_tensors=True)
74
+ print(features.shape) # torch.Size([1, 2048])
75
+
76
+ # Dense token map (spatial grid scales with input size)
77
+ tokens = pipe(image, pool=False, return_tensors=True)
78
+ print(tokens.shape) # ResNet: [1, 256, 2048] for 512×512
79
+ ```
80
+
81
+ Sentinel-1 ViT (2 channels):
82
+
83
+ ```python
84
+ SUBFOLDER = "decur-vit-small-patch16-s1"
85
+ pipe = pipeline(
86
+ task="decur-feature-extraction",
87
+ model=REPO,
88
+ trust_remote_code=True,
89
+ model_kwargs={"subfolder": SUBFOLDER},
90
+ )
91
+ image = np.random.randint(0, 255, (448, 448, 2), dtype=np.uint8)
92
+ features = pipe(image, pool=True, return_tensors=True)
93
+ print(features.shape) # torch.Size([1, 384])
94
+ ```
95
+
96
+ Load components directly:
97
+
98
+ ```python
99
+ from transformers import AutoModel, AutoImageProcessor
100
+
101
+ model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
102
+ processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
103
+ inputs = processor(image, return_tensors="pt")
104
+ out = model(**inputs)
105
+ ```
106
+
107
+ Each checkpoint folder can also be used as a standalone local path (omit `subfolder`):
108
+
109
+ ```python
110
+ pipe = pipeline(
111
+ task="decur-feature-extraction",
112
+ model="/path/to/DECUR-transformers/decur-mit-b2-rgb",
113
+ trust_remote_code=True,
114
+ )
115
+ ```
116
+
117
+ ## Native resolution notes
118
+
119
+ | Backbone | Native larger input | Pooled output | Sequence output |
120
+ |----------|---------------------|---------------|-----------------|
121
+ | ResNet-50 | Yes (fully conv) | `[B, 2048]` | `[B, H'×W', 2048]` |
122
+ | ViT-S/16 | Yes (`interpolate_pos_encoding=True` by default) | `[B, 384]` (CLS) | `[B, N, 384]` |
123
+ | MiT-B2/B5 | Yes | `[B, 512]` (spatial mean) | `[B, H'×W', 512]` |
124
+
125
+ - **`size` in `preprocessor_config.json`** (224×224) is the pretraining reference, not a forced input size.
126
+ - **ViT** uses interpolated positional embeddings by default for non-224 inputs. Disable with `model(..., interpolate_pos_encoding=False)`.
127
+ - **Opt in to resize** (e.g. match 224×224 pretraining):
128
+
129
+ ```python
130
+ features = pipe(
131
+ image,
132
+ pool=True,
133
+ return_tensors=True,
134
+ image_processor_kwargs={"do_resize": True},
135
+ )
136
+ ```
137
+
138
+ The custom pipeline is registered in each checkpoint's `config.json` under `custom_pipelines.decur-feature-extraction` (see [Adding a new pipeline](https://huggingface.co/docs/transformers/add_new_pipeline#upload-to-the-hub)).
139
+
140
+ ## Test CLI
141
+
142
+ ```bash
143
+ conda activate rsgen
144
+ python test_decur.py --model decur-mit-b2-rgb
145
+ python test_decur.py --all
146
+ ```
147
+
148
+ ## Dependencies
149
+
150
+ - `transformers`
151
+ - `torch`
152
+ - `torchvision` (ResNet backbones)
153
+ - `einops` (RDA modules)
154
+ - `opencv-python` (only when resizing multispectral inputs with >4 channels)
155
+
156
+ ## Notes
157
+
158
+ - Pooled features: ResNet `[B, 2048]`, ViT `[B, 384]` (CLS token), MiT `[B, 512]` (spatial mean).
159
+ - GeoNRW DEM checkpoints use **3 input channels** in the released weights (not 1-channel DSM).
160
+ - RDA ResNet models include deformable attention modules (`da_l3`, `da_l4`).
decur-mit-b2-hha/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "mit_b2",
6
+ "dtype": "float32",
7
+ "hidden_size": 512,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "hha",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 4,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": false,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-mit-b2-hha/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-mit-b2-hha/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-mit-b2-hha/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-mit-b2-hha/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-mit-b2-hha/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-mit-b2-rgb/__pycache__/modeling_decur.cpython-311.pyc ADDED
Binary file (13.6 kB). View file
 
decur-mit-b2-rgb/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "mit_b2",
6
+ "dtype": "float32",
7
+ "hidden_size": 512,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "rgb",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 4,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": false,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-mit-b2-rgb/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-mit-b2-rgb/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-mit-b2-rgb/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-mit-b2-rgb/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-mit-b2-rgb/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-mit-b5-hha/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "mit_b5",
6
+ "dtype": "float32",
7
+ "hidden_size": 512,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "hha",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 4,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": false,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-mit-b5-hha/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-mit-b5-hha/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-mit-b5-hha/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-mit-b5-hha/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-mit-b5-hha/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-mit-b5-rgb/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "mit_b5",
6
+ "dtype": "float32",
7
+ "hidden_size": 512,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "rgb",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 4,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": false,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-mit-b5-rgb/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-mit-b5-rgb/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-mit-b5-rgb/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-mit-b5-rgb/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-mit-b5-rgb/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-resnet50-dem/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "resnet50",
6
+ "dtype": "float32",
7
+ "hidden_size": 2048,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "dem",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 16,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": false,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-resnet50-dem/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-resnet50-dem/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-resnet50-dem/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-resnet50-dem/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-resnet50-dem/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-resnet50-rda-dem/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "resnet50",
6
+ "dtype": "float32",
7
+ "hidden_size": 2048,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "dem",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 16,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": true,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-resnet50-rda-dem/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-resnet50-rda-dem/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-resnet50-rda-dem/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-resnet50-rda-dem/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-resnet50-rda-dem/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-resnet50-rda-rgb/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "resnet50",
6
+ "dtype": "float32",
7
+ "hidden_size": 2048,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "rgb",
12
+ "model_type": "decur",
13
+ "num_channels": 3,
14
+ "patch_size": 16,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": true,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-resnet50-rda-rgb/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-resnet50-rda-rgb/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-resnet50-rda-rgb/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-resnet50-rda-rgb/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-resnet50-rda-rgb/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }
decur-resnet50-rda-s1/config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "DeCURModel"
4
+ ],
5
+ "backbone": "resnet50",
6
+ "dtype": "float32",
7
+ "hidden_size": 2048,
8
+ "id2label": {},
9
+ "image_size": 224,
10
+ "label2id": {},
11
+ "modality": "s1",
12
+ "model_type": "decur",
13
+ "num_channels": 2,
14
+ "patch_size": 16,
15
+ "transformers_version": "5.0.0",
16
+ "use_rda": true,
17
+ "auto_map": {
18
+ "AutoConfig": "modeling_decur.DeCURConfig",
19
+ "AutoModel": "modeling_decur.DeCURModel",
20
+ "AutoModelForImageClassification": "modeling_decur.DeCURForImageClassification"
21
+ },
22
+ "custom_pipelines": {
23
+ "decur-feature-extraction": {
24
+ "impl": "pipeline_decur.DeCURImageFeatureExtractionPipeline",
25
+ "pt": [
26
+ "AutoModel"
27
+ ]
28
+ }
29
+ }
30
+ }
decur-resnet50-rda-s1/dat_blocks.py ADDED
@@ -0,0 +1,679 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # --------------------------------------------------------
2
+ # Swin Transformer
3
+ # Copyright (c) 2021 Microsoft
4
+ # Licensed under The MIT License [see LICENSE for details]
5
+ # Written by Ze Liu
6
+ # --------------------------------------------------------
7
+ # Vision Transformer with Deformable Attention
8
+ # Modified by Zhuofan Xia
9
+ # --------------------------------------------------------
10
+
11
+ import math
12
+ import numpy as np
13
+ import torch
14
+ import torch.nn as nn
15
+ import torch.nn.functional as F
16
+ import einops
17
+ from collections.abc import Sequence
18
+
19
+
20
+ def to_2tuple(x: int | Sequence[int]) -> tuple[int, int]:
21
+ if isinstance(x, Sequence) and not isinstance(x, (str, bytes)):
22
+ if len(x) == 2:
23
+ return int(x[0]), int(x[1])
24
+ if len(x) == 1:
25
+ return int(x[0]), int(x[0])
26
+ return int(x), int(x)
27
+
28
+
29
+ def trunc_normal_(tensor: torch.Tensor, mean: float = 0.0, std: float = 1.0) -> torch.Tensor:
30
+ return nn.init.trunc_normal_(tensor, mean=mean, std=std)
31
+
32
+
33
+ class LocalAttention(nn.Module):
34
+
35
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop):
36
+
37
+ super().__init__()
38
+
39
+ window_size = to_2tuple(window_size)
40
+
41
+ self.proj_qkv = nn.Linear(dim, 3 * dim)
42
+ self.heads = heads
43
+ assert dim % heads == 0
44
+ head_dim = dim // heads
45
+ self.scale = head_dim ** -0.5
46
+ self.proj_out = nn.Linear(dim, dim)
47
+ self.window_size = window_size
48
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
49
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
50
+
51
+ Wh, Ww = self.window_size
52
+ self.relative_position_bias_table = nn.Parameter(
53
+ torch.zeros((2 * Wh - 1) * (2 * Ww - 1), heads)
54
+ )
55
+ trunc_normal_(self.relative_position_bias_table, std=0.01)
56
+
57
+ coords_h = torch.arange(self.window_size[0])
58
+ coords_w = torch.arange(self.window_size[1])
59
+ coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing='ij')) # 2, Wh, Ww
60
+ coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
61
+ relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
62
+ relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
63
+ relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0
64
+ relative_coords[:, :, 1] += self.window_size[1] - 1
65
+ relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
66
+ relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
67
+ self.register_buffer("relative_position_index", relative_position_index)
68
+
69
+ def forward(self, x, mask=None):
70
+
71
+ B, C, H, W = x.size()
72
+ r1, r2 = H // self.window_size[0], W // self.window_size[1]
73
+
74
+ x_total = einops.rearrange(x, 'b c (r1 h1) (r2 w1) -> b (r1 r2) (h1 w1) c', h1=self.window_size[0], w1=self.window_size[1]) # B x Nr x Ws x C
75
+
76
+ x_total = einops.rearrange(x_total, 'b m n c -> (b m) n c')
77
+
78
+ qkv = self.proj_qkv(x_total) # B' x N x 3C
79
+ q, k, v = torch.chunk(qkv, 3, dim=2)
80
+
81
+ q = q * self.scale
82
+ q, k, v = [einops.rearrange(t, 'b n (h c1) -> b h n c1', h=self.heads) for t in [q, k, v]]
83
+ attn = torch.einsum('b h m c, b h n c -> b h m n', q, k)
84
+
85
+ relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
86
+ self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1) # Wh*Ww,Wh*Ww,nH
87
+ relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
88
+ attn_bias = relative_position_bias
89
+ attn = attn + attn_bias.unsqueeze(0)
90
+
91
+ if mask is not None:
92
+ # attn : (b * nW) h w w
93
+ # mask : nW ww ww
94
+ nW, ww, _ = mask.size()
95
+ attn = einops.rearrange(attn, '(b n) h w1 w2 -> b n h w1 w2', n=nW, h=self.heads, w1=ww, w2=ww) + mask.reshape(1, nW, 1, ww, ww)
96
+ attn = einops.rearrange(attn, 'b n h w1 w2 -> (b n) h w1 w2')
97
+ attn = self.attn_drop(attn.softmax(dim=3))
98
+
99
+ x = torch.einsum('b h m n, b h n c -> b h m c', attn, v)
100
+ x = einops.rearrange(x, 'b h n c1 -> b n (h c1)')
101
+ x = self.proj_drop(self.proj_out(x)) # B' x N x C
102
+ x = einops.rearrange(x, '(b r1 r2) (h1 w1) c -> b c (r1 h1) (r2 w1)', r1=r1, r2=r2, h1=self.window_size[0], w1=self.window_size[1]) # B x C x H x W
103
+
104
+ return x, None, None
105
+
106
+
107
+ class ShiftWindowAttention(LocalAttention):
108
+
109
+ def __init__(self, dim, heads, window_size, attn_drop, proj_drop, shift_size, fmap_size):
110
+
111
+ super().__init__(dim, heads, window_size, attn_drop, proj_drop)
112
+
113
+ self.fmap_size = to_2tuple(fmap_size)
114
+ self.shift_size = shift_size
115
+
116
+ assert 0 < self.shift_size < min(self.window_size), "wrong shift size."
117
+
118
+ img_mask = torch.zeros(*self.fmap_size) # H W
119
+ h_slices = (slice(0, -self.window_size[0]),
120
+ slice(-self.window_size[0], -self.shift_size),
121
+ slice(-self.shift_size, None))
122
+ w_slices = (slice(0, -self.window_size[1]),
123
+ slice(-self.window_size[1], -self.shift_size),
124
+ slice(-self.shift_size, None))
125
+ cnt = 0
126
+ for h in h_slices:
127
+ for w in w_slices:
128
+ img_mask[h, w] = cnt
129
+ cnt += 1
130
+ mask_windows = einops.rearrange(img_mask, '(r1 h1) (r2 w1) -> (r1 r2) (h1 w1)', h1=self.window_size[0],w1=self.window_size[1])
131
+ attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) # nW ww ww
132
+ attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
133
+ self.register_buffer("attn_mask", attn_mask)
134
+
135
+ def forward(self, x):
136
+
137
+ shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(2, 3))
138
+ sw_x, _, _ = super().forward(shifted_x, self.attn_mask)
139
+ x = torch.roll(sw_x, shifts=(self.shift_size, self.shift_size), dims=(2, 3))
140
+
141
+ return x, None, None
142
+
143
+
144
+ class DAttentionBaseline(nn.Module):
145
+
146
+ def __init__(
147
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
148
+ attn_drop, proj_drop, stride,
149
+ offset_range_factor, use_pe, dwc_pe,
150
+ no_off, fixed_pe, ksize, log_cpb
151
+ ):
152
+
153
+ super().__init__()
154
+ self.dwc_pe = dwc_pe
155
+ self.n_head_channels = n_head_channels
156
+ self.scale = self.n_head_channels ** -0.5
157
+ self.n_heads = n_heads
158
+ self.q_h, self.q_w = q_size
159
+ # self.kv_h, self.kv_w = kv_size
160
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
161
+ self.nc = n_head_channels * n_heads
162
+ self.n_groups = n_groups
163
+ self.n_group_channels = self.nc // self.n_groups
164
+ self.n_group_heads = self.n_heads // self.n_groups
165
+ self.use_pe = use_pe
166
+ self.fixed_pe = fixed_pe
167
+ self.no_off = no_off
168
+ self.offset_range_factor = offset_range_factor
169
+ self.ksize = ksize
170
+ self.log_cpb = log_cpb
171
+ self.stride = stride
172
+ kk = self.ksize
173
+ pad_size = kk // 2 if kk != stride else 0
174
+
175
+ self.conv_offset = nn.Sequential(
176
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
177
+ LayerNormProxy(self.n_group_channels),
178
+ nn.GELU(),
179
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
180
+ )
181
+ if self.no_off:
182
+ for m in self.conv_offset.parameters():
183
+ m.requires_grad_(False)
184
+
185
+ self.proj_q = nn.Conv2d(
186
+ self.nc, self.nc,
187
+ kernel_size=1, stride=1, padding=0
188
+ )
189
+
190
+ self.proj_k = nn.Conv2d(
191
+ self.nc, self.nc,
192
+ kernel_size=1, stride=1, padding=0
193
+ )
194
+
195
+ self.proj_v = nn.Conv2d(
196
+ self.nc, self.nc,
197
+ kernel_size=1, stride=1, padding=0
198
+ )
199
+
200
+ self.proj_out = nn.Conv2d(
201
+ self.nc, self.nc,
202
+ kernel_size=1, stride=1, padding=0
203
+ )
204
+
205
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
206
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
207
+
208
+ if self.use_pe and not self.no_off:
209
+ if self.dwc_pe:
210
+ self.rpe_table = nn.Conv2d(
211
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
212
+ elif self.fixed_pe:
213
+ self.rpe_table = nn.Parameter(
214
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
215
+ )
216
+ trunc_normal_(self.rpe_table, std=0.01)
217
+ elif self.log_cpb:
218
+ # Borrowed from Swin-V2
219
+ self.rpe_table = nn.Sequential(
220
+ nn.Linear(2, 32, bias=True),
221
+ nn.ReLU(inplace=True),
222
+ nn.Linear(32, self.n_group_heads, bias=False)
223
+ )
224
+ else:
225
+ self.rpe_table = nn.Parameter(
226
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
227
+ )
228
+ trunc_normal_(self.rpe_table, std=0.01)
229
+ else:
230
+ self.rpe_table = None
231
+
232
+ @torch.no_grad()
233
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
234
+
235
+ ref_y, ref_x = torch.meshgrid(
236
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
237
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
238
+ indexing='ij'
239
+ )
240
+ ref = torch.stack((ref_y, ref_x), -1)
241
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
242
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
243
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
244
+
245
+ return ref
246
+
247
+ @torch.no_grad()
248
+ def _get_q_grid(self, H, W, B, dtype, device):
249
+
250
+ ref_y, ref_x = torch.meshgrid(
251
+ torch.arange(0, H, dtype=dtype, device=device),
252
+ torch.arange(0, W, dtype=dtype, device=device),
253
+ indexing='ij'
254
+ )
255
+ ref = torch.stack((ref_y, ref_x), -1)
256
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
257
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
258
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
259
+
260
+ return ref
261
+
262
+ def forward(self, x):
263
+
264
+ B, C, H, W = x.size()
265
+ dtype, device = x.dtype, x.device
266
+
267
+ q = self.proj_q(x)
268
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
269
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
270
+ Hk, Wk = offset.size(2), offset.size(3)
271
+ n_sample = Hk * Wk
272
+
273
+ if self.offset_range_factor >= 0 and not self.no_off:
274
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
275
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
276
+
277
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
278
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
279
+
280
+ if self.no_off:
281
+ offset = offset.fill_(0.0)
282
+
283
+ if self.offset_range_factor >= 0:
284
+ pos = offset + reference
285
+ else:
286
+ pos = (offset + reference).clamp(-1., +1.)
287
+
288
+ if self.no_off:
289
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
290
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
291
+ else:
292
+ x_sampled = F.grid_sample(
293
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
294
+ grid=pos[..., (1, 0)], # y, x -> x, y
295
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
296
+
297
+
298
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
299
+
300
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
301
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
302
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
303
+
304
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
305
+ attn = attn.mul(self.scale)
306
+
307
+ if self.use_pe and (not self.no_off):
308
+
309
+ if self.dwc_pe:
310
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
311
+ elif self.fixed_pe:
312
+ rpe_table = self.rpe_table
313
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
314
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
315
+ elif self.log_cpb:
316
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
317
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
318
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
319
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
320
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
321
+ else:
322
+ rpe_table = self.rpe_table
323
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
324
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
325
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
326
+ attn_bias = F.grid_sample(
327
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
328
+ grid=displacement[..., (1, 0)],
329
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
330
+
331
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
332
+ attn = attn + attn_bias
333
+
334
+ attn = F.softmax(attn, dim=2)
335
+ attn = self.attn_drop(attn)
336
+
337
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
338
+
339
+ if self.use_pe and self.dwc_pe:
340
+ out = out + residual_lepe
341
+ out = out.reshape(B, C, H, W)
342
+
343
+ y = self.proj_drop(self.proj_out(out))
344
+
345
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
346
+
347
+
348
+ class DAttentionCross(nn.Module):
349
+
350
+ def __init__(
351
+ self, q_size, kv_size, n_heads, n_head_channels, n_groups,
352
+ attn_drop, proj_drop, stride,
353
+ offset_range_factor, use_pe, dwc_pe,
354
+ no_off, fixed_pe, ksize, log_cpb
355
+ ):
356
+
357
+ super().__init__()
358
+ self.dwc_pe = dwc_pe
359
+ self.n_head_channels = n_head_channels
360
+ self.scale = self.n_head_channels ** -0.5
361
+ self.n_heads = n_heads
362
+ self.q_h, self.q_w = q_size
363
+ # self.kv_h, self.kv_w = kv_size
364
+ self.kv_h, self.kv_w = self.q_h // stride, self.q_w // stride
365
+ self.nc = n_head_channels * n_heads
366
+ self.n_groups = n_groups
367
+ self.n_group_channels = self.nc // self.n_groups
368
+ self.n_group_heads = self.n_heads // self.n_groups
369
+ self.use_pe = use_pe
370
+ self.fixed_pe = fixed_pe
371
+ self.no_off = no_off
372
+ self.offset_range_factor = offset_range_factor
373
+ self.ksize = ksize
374
+ self.log_cpb = log_cpb
375
+ self.stride = stride
376
+ kk = self.ksize
377
+ pad_size = kk // 2 if kk != stride else 0
378
+
379
+ self.conv_offset = nn.Sequential(
380
+ nn.Conv2d(self.n_group_channels, self.n_group_channels, kk, stride, pad_size, groups=self.n_group_channels),
381
+ LayerNormProxy(self.n_group_channels),
382
+ nn.GELU(),
383
+ nn.Conv2d(self.n_group_channels, 2, 1, 1, 0, bias=False)
384
+ )
385
+ if self.no_off:
386
+ for m in self.conv_offset.parameters():
387
+ m.requires_grad_(False)
388
+
389
+ self.proj_q = nn.Conv2d(
390
+ self.nc, self.nc,
391
+ kernel_size=1, stride=1, padding=0
392
+ )
393
+
394
+ self.proj_k = nn.Conv2d(
395
+ self.nc, self.nc,
396
+ kernel_size=1, stride=1, padding=0
397
+ )
398
+
399
+ self.proj_v = nn.Conv2d(
400
+ self.nc, self.nc,
401
+ kernel_size=1, stride=1, padding=0
402
+ )
403
+
404
+ self.proj_out = nn.Conv2d(
405
+ self.nc, self.nc,
406
+ kernel_size=1, stride=1, padding=0
407
+ )
408
+
409
+ self.proj_drop = nn.Dropout(proj_drop, inplace=True)
410
+ self.attn_drop = nn.Dropout(attn_drop, inplace=True)
411
+
412
+ if self.use_pe and not self.no_off:
413
+ if self.dwc_pe:
414
+ self.rpe_table = nn.Conv2d(
415
+ self.nc, self.nc, kernel_size=3, stride=1, padding=1, groups=self.nc)
416
+ elif self.fixed_pe:
417
+ self.rpe_table = nn.Parameter(
418
+ torch.zeros(self.n_heads, self.q_h * self.q_w, self.kv_h * self.kv_w)
419
+ )
420
+ trunc_normal_(self.rpe_table, std=0.01)
421
+ elif self.log_cpb:
422
+ # Borrowed from Swin-V2
423
+ self.rpe_table = nn.Sequential(
424
+ nn.Linear(2, 32, bias=True),
425
+ nn.ReLU(inplace=True),
426
+ nn.Linear(32, self.n_group_heads, bias=False)
427
+ )
428
+ else:
429
+ self.rpe_table = nn.Parameter(
430
+ torch.zeros(self.n_heads, self.q_h * 2 - 1, self.q_w * 2 - 1)
431
+ )
432
+ trunc_normal_(self.rpe_table, std=0.01)
433
+ else:
434
+ self.rpe_table = None
435
+
436
+ @torch.no_grad()
437
+ def _get_ref_points(self, H_key, W_key, B, dtype, device):
438
+
439
+ ref_y, ref_x = torch.meshgrid(
440
+ torch.linspace(0.5, H_key - 0.5, H_key, dtype=dtype, device=device),
441
+ torch.linspace(0.5, W_key - 0.5, W_key, dtype=dtype, device=device),
442
+ indexing='ij'
443
+ )
444
+ ref = torch.stack((ref_y, ref_x), -1)
445
+ ref[..., 1].div_(W_key - 1.0).mul_(2.0).sub_(1.0)
446
+ ref[..., 0].div_(H_key - 1.0).mul_(2.0).sub_(1.0)
447
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
448
+
449
+ return ref
450
+
451
+ @torch.no_grad()
452
+ def _get_q_grid(self, H, W, B, dtype, device):
453
+
454
+ ref_y, ref_x = torch.meshgrid(
455
+ torch.arange(0, H, dtype=dtype, device=device),
456
+ torch.arange(0, W, dtype=dtype, device=device),
457
+ indexing='ij'
458
+ )
459
+ ref = torch.stack((ref_y, ref_x), -1)
460
+ ref[..., 1].div_(W - 1.0).mul_(2.0).sub_(1.0)
461
+ ref[..., 0].div_(H - 1.0).mul_(2.0).sub_(1.0)
462
+ ref = ref[None, ...].expand(B * self.n_groups, -1, -1, -1) # B * g H W 2
463
+
464
+ return ref
465
+
466
+ def forward(self, x, x1):
467
+
468
+ B, C, H, W = x.size()
469
+ dtype, device = x.dtype, x.device
470
+
471
+ #q = self.proj_q(x)
472
+ q = self.proj_q(x1)
473
+ q_off = einops.rearrange(q, 'b (g c) h w -> (b g) c h w', g=self.n_groups, c=self.n_group_channels)
474
+ offset = self.conv_offset(q_off).contiguous() # B * g 2 Hg Wg
475
+ Hk, Wk = offset.size(2), offset.size(3)
476
+ n_sample = Hk * Wk
477
+
478
+ if self.offset_range_factor >= 0 and not self.no_off:
479
+ offset_range = torch.tensor([1.0 / (Hk - 1.0), 1.0 / (Wk - 1.0)], device=device).reshape(1, 2, 1, 1)
480
+ offset = offset.tanh().mul(offset_range).mul(self.offset_range_factor)
481
+
482
+ offset = einops.rearrange(offset, 'b p h w -> b h w p')
483
+ reference = self._get_ref_points(Hk, Wk, B, dtype, device)
484
+
485
+ if self.no_off:
486
+ offset = offset.fill_(0.0)
487
+
488
+ if self.offset_range_factor >= 0:
489
+ pos = offset + reference
490
+ else:
491
+ pos = (offset + reference).clamp(-1., +1.)
492
+
493
+ if self.no_off:
494
+ x_sampled = F.avg_pool2d(x, kernel_size=self.stride, stride=self.stride)
495
+ assert x_sampled.size(2) == Hk and x_sampled.size(3) == Wk, f"Size is {x_sampled.size()}"
496
+ else:
497
+ x_sampled = F.grid_sample(
498
+ input=x.reshape(B * self.n_groups, self.n_group_channels, H, W),
499
+ grid=pos[..., (1, 0)], # y, x -> x, y
500
+ mode='bilinear', align_corners=True) # B * g, Cg, Hg, Wg
501
+
502
+
503
+ x_sampled = x_sampled.reshape(B, C, 1, n_sample)
504
+
505
+ q = q.reshape(B * self.n_heads, self.n_head_channels, H * W)
506
+ k = self.proj_k(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
507
+ v = self.proj_v(x_sampled).reshape(B * self.n_heads, self.n_head_channels, n_sample)
508
+
509
+ attn = torch.einsum('b c m, b c n -> b m n', q, k) # B * h, HW, Ns
510
+ attn = attn.mul(self.scale)
511
+
512
+ if self.use_pe and (not self.no_off):
513
+
514
+ if self.dwc_pe:
515
+ residual_lepe = self.rpe_table(q.reshape(B, C, H, W)).reshape(B * self.n_heads, self.n_head_channels, H * W)
516
+ elif self.fixed_pe:
517
+ rpe_table = self.rpe_table
518
+ attn_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
519
+ attn = attn + attn_bias.reshape(B * self.n_heads, H * W, n_sample)
520
+ elif self.log_cpb:
521
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
522
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(4.0) # d_y, d_x [-8, +8]
523
+ displacement = torch.sign(displacement) * torch.log2(torch.abs(displacement) + 1.0) / np.log2(8.0)
524
+ attn_bias = self.rpe_table(displacement) # B * g, H * W, n_sample, h_g
525
+ attn = attn + einops.rearrange(attn_bias, 'b m n h -> (b h) m n', h=self.n_group_heads)
526
+ else:
527
+ rpe_table = self.rpe_table
528
+ rpe_bias = rpe_table[None, ...].expand(B, -1, -1, -1)
529
+ q_grid = self._get_q_grid(H, W, B, dtype, device)
530
+ displacement = (q_grid.reshape(B * self.n_groups, H * W, 2).unsqueeze(2) - pos.reshape(B * self.n_groups, n_sample, 2).unsqueeze(1)).mul(0.5)
531
+ attn_bias = F.grid_sample(
532
+ input=einops.rearrange(rpe_bias, 'b (g c) h w -> (b g) c h w', c=self.n_group_heads, g=self.n_groups),
533
+ grid=displacement[..., (1, 0)],
534
+ mode='bilinear', align_corners=True) # B * g, h_g, HW, Ns
535
+
536
+ attn_bias = attn_bias.reshape(B * self.n_heads, H * W, n_sample)
537
+ attn = attn + attn_bias
538
+
539
+ attn = F.softmax(attn, dim=2)
540
+ attn = self.attn_drop(attn)
541
+
542
+ out = torch.einsum('b m n, b c n -> b c m', attn, v)
543
+
544
+ if self.use_pe and self.dwc_pe:
545
+ out = out + residual_lepe
546
+ out = out.reshape(B, C, H, W)
547
+
548
+ y = self.proj_drop(self.proj_out(out))
549
+
550
+ return y, pos.reshape(B, self.n_groups, Hk, Wk, 2), reference.reshape(B, self.n_groups, Hk, Wk, 2)
551
+
552
+
553
+ class PyramidAttention(nn.Module):
554
+
555
+ def __init__(self, dim, num_heads=8, attn_drop=0., proj_drop=0., sr_ratio=1):
556
+
557
+ super().__init__()
558
+
559
+ assert dim % num_heads == 0, f"dim {dim} should be divided by num_heads {num_heads}."
560
+
561
+ self.dim = dim
562
+ self.num_heads = num_heads
563
+ self.head_dim = dim // num_heads
564
+ self.scale = self.head_dim ** -0.5
565
+
566
+ self.q = nn.Conv2d(dim, dim, 1, 1, 0)
567
+ self.kv = nn.Conv2d(dim, dim * 2, 1, 1, 0)
568
+ self.attn_drop = nn.Dropout(attn_drop)
569
+ self.proj = nn.Conv2d(dim, dim, 1, 1, 0)
570
+ self.proj_drop = nn.Dropout(proj_drop)
571
+
572
+ self.sr_ratio = sr_ratio
573
+ if sr_ratio > 1:
574
+ self.proj_ds = nn.Sequential(
575
+ nn.Conv2d(dim, dim, kernel_size=sr_ratio, stride=sr_ratio),
576
+ LayerNormProxy(dim)
577
+ )
578
+
579
+
580
+ def forward(self, x):
581
+
582
+ B, C, H, W = x.size()
583
+ Nq = H * W
584
+ q = self.q(x)
585
+
586
+ if self.sr_ratio > 1:
587
+ x_ds = self.proj_ds(x)
588
+ kv = self.kv(x_ds)
589
+ else:
590
+ kv = self.kv(x)
591
+
592
+ k, v = torch.chunk(kv, 2, dim=1)
593
+ Nk = (H // self.sr_ratio) * (W // self.sr_ratio)
594
+ q = q.reshape(B * self.num_heads, self.head_dim, Nq).mul(self.scale)
595
+ k = k.reshape(B * self.num_heads, self.head_dim, Nk)
596
+ v = v.reshape(B * self.num_heads, self.head_dim, Nk)
597
+ attn = torch.einsum('b c m, b c n -> b m n', q, k)
598
+ attn = F.softmax(attn, dim=2)
599
+ attn = self.attn_drop(attn)
600
+
601
+ x = torch.einsum('b m n, b c n -> b c m', attn, v)
602
+ x = x.reshape(B, C, H, W)
603
+ x = self.proj(x)
604
+ x = self.proj_drop(x)
605
+
606
+ return x, None, None
607
+
608
+
609
+ class TransformerMLP(nn.Module):
610
+
611
+ def __init__(self, channels, expansion, drop):
612
+
613
+ super().__init__()
614
+
615
+ self.dim1 = channels
616
+ self.dim2 = channels * expansion
617
+ self.chunk = nn.Sequential()
618
+ self.chunk.add_module('linear1', nn.Linear(self.dim1, self.dim2))
619
+ self.chunk.add_module('act', nn.GELU())
620
+ self.chunk.add_module('drop1', nn.Dropout(drop, inplace=True))
621
+ self.chunk.add_module('linear2', nn.Linear(self.dim2, self.dim1))
622
+ self.chunk.add_module('drop2', nn.Dropout(drop, inplace=True))
623
+
624
+ def forward(self, x):
625
+
626
+ _, _, H, W = x.size()
627
+ x = einops.rearrange(x, 'b c h w -> b (h w) c')
628
+ x = self.chunk(x)
629
+ x = einops.rearrange(x, 'b (h w) c -> b c h w', h=H, w=W)
630
+ return x
631
+
632
+ class LayerNormProxy(nn.Module):
633
+
634
+ def __init__(self, dim):
635
+
636
+ super().__init__()
637
+ self.norm = nn.LayerNorm(dim)
638
+
639
+ def forward(self, x):
640
+
641
+ x = einops.rearrange(x, 'b c h w -> b h w c')
642
+ x = self.norm(x)
643
+ return einops.rearrange(x, 'b h w c -> b c h w')
644
+
645
+
646
+ class TransformerMLPWithConv(nn.Module):
647
+
648
+ def __init__(self, channels, expansion, drop):
649
+
650
+ super().__init__()
651
+
652
+ self.dim1 = channels
653
+ self.dim2 = channels * expansion
654
+ self.linear1 = nn.Sequential(
655
+ nn.Conv2d(self.dim1, self.dim2, 1, 1, 0),
656
+ # nn.GELU(),
657
+ # nn.BatchNorm2d(self.dim2, eps=1e-5)
658
+ )
659
+ self.drop1 = nn.Dropout(drop, inplace=True)
660
+ self.act = nn.GELU()
661
+ # self.bn = nn.BatchNorm2d(self.dim2, eps=1e-5)
662
+ self.linear2 = nn.Sequential(
663
+ nn.Conv2d(self.dim2, self.dim1, 1, 1, 0),
664
+ # nn.BatchNorm2d(self.dim1, eps=1e-5)
665
+ )
666
+ self.drop2 = nn.Dropout(drop, inplace=True)
667
+ self.dwc = nn.Conv2d(self.dim2, self.dim2, 3, 1, 1, groups=self.dim2)
668
+
669
+ def forward(self, x):
670
+
671
+ x = self.linear1(x)
672
+ x = self.drop1(x)
673
+ x = x + self.dwc(x)
674
+ x = self.act(x)
675
+ # x = self.bn(x)
676
+ x = self.linear2(x)
677
+ x = self.drop2(x)
678
+
679
+ return x
decur-resnet50-rda-s1/image_processing_decur.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Image processor for DeCUR models."""
3
+
4
+ from typing import Optional, Union
5
+
6
+ import numpy as np
7
+
8
+ from transformers.image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict
9
+ from transformers.image_transforms import to_channel_dimension_format
10
+ from transformers.image_utils import (
11
+ ChannelDimension,
12
+ ImageInput,
13
+ PILImageResampling,
14
+ infer_channel_dimension_format,
15
+ make_flat_list_of_images,
16
+ to_numpy_array,
17
+ valid_images,
18
+ validate_preprocess_arguments,
19
+ )
20
+ from transformers.utils import TensorType, filter_out_non_signature_kwargs, logging
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ def _resize_multispectral(image: np.ndarray, size: dict[str, int], input_data_format: ChannelDimension) -> np.ndarray:
27
+ target_height, target_width = size["height"], size["width"]
28
+
29
+ if input_data_format == ChannelDimension.FIRST:
30
+ image = np.transpose(image, (1, 2, 0))
31
+
32
+ height, width, _ = image.shape
33
+ if height == target_height and width == target_width:
34
+ resized = image
35
+ else:
36
+ try:
37
+ import cv2
38
+ except ImportError as exc:
39
+ raise ImportError(
40
+ "Multispectral resize requires OpenCV (`opencv-python`) when input has more than 4 channels."
41
+ ) from exc
42
+ resized = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_LINEAR)
43
+
44
+ if input_data_format == ChannelDimension.FIRST:
45
+ return np.transpose(resized, (2, 0, 1))
46
+ return resized
47
+
48
+
49
+ class DeCURImageProcessor(BaseImageProcessor):
50
+ """
51
+ Image processor for DeCUR Earth observation and RGB-D encoders.
52
+
53
+ By default, inputs are rescaled to `[0, 1]` by dividing by 255. DeCUR recommends mapping each channel to
54
+ uint8 using per-channel mean/std before inference; enable `do_normalize` with dataset statistics when needed.
55
+ """
56
+
57
+ model_input_names = ["pixel_values"]
58
+
59
+ def __init__(
60
+ self,
61
+ do_resize: bool = False,
62
+ size: Optional[dict[str, int]] = None,
63
+ resample: PILImageResampling = PILImageResampling.BILINEAR,
64
+ do_rescale: bool = True,
65
+ rescale_factor: float = 1 / 255.0,
66
+ do_normalize: bool = False,
67
+ image_mean: Optional[Union[float, list[float]]] = None,
68
+ image_std: Optional[Union[float, list[float]]] = None,
69
+ do_convert_rgb: bool = False,
70
+ **kwargs,
71
+ ):
72
+ super().__init__(**kwargs)
73
+ size = size if size is not None else {"height": 224, "width": 224}
74
+ self.do_resize = do_resize
75
+ self.size = size
76
+ self.resample = resample
77
+ self.do_rescale = do_rescale
78
+ self.rescale_factor = rescale_factor
79
+ self.do_normalize = do_normalize
80
+ self.image_mean = image_mean
81
+ self.image_std = image_std
82
+ self.do_convert_rgb = do_convert_rgb
83
+
84
+ @filter_out_non_signature_kwargs()
85
+ def preprocess(
86
+ self,
87
+ images: ImageInput,
88
+ do_resize: Optional[bool] = None,
89
+ size: Optional[dict[str, int]] = None,
90
+ resample: Optional[PILImageResampling] = None,
91
+ do_rescale: Optional[bool] = None,
92
+ rescale_factor: Optional[float] = None,
93
+ do_normalize: Optional[bool] = None,
94
+ image_mean: Optional[Union[float, list[float]]] = None,
95
+ image_std: Optional[Union[float, list[float]]] = None,
96
+ return_tensors: Optional[Union[str, TensorType]] = None,
97
+ data_format: Union[str, ChannelDimension] = ChannelDimension.FIRST,
98
+ input_data_format: Optional[Union[str, ChannelDimension]] = None,
99
+ do_convert_rgb: Optional[bool] = None,
100
+ ):
101
+ do_resize = do_resize if do_resize is not None else self.do_resize
102
+ size = size if size is not None else self.size
103
+ size = get_size_dict(size, default_to_square=True)
104
+ resample = resample if resample is not None else self.resample
105
+ do_rescale = do_rescale if do_rescale is not None else self.do_rescale
106
+ rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
107
+ do_normalize = do_normalize if do_normalize is not None else self.do_normalize
108
+ image_mean = image_mean if image_mean is not None else self.image_mean
109
+ image_std = image_std if image_std is not None else self.image_std
110
+ do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
111
+
112
+ if do_normalize and (image_mean is None or image_std is None):
113
+ raise ValueError(
114
+ "Normalization requires `image_mean` and `image_std` with one value per channel."
115
+ )
116
+
117
+ images = make_flat_list_of_images(images)
118
+ if not valid_images(images):
119
+ raise ValueError("Invalid image type. Must be PIL, numpy, or torch tensor.")
120
+
121
+ validate_preprocess_arguments(
122
+ do_rescale=do_rescale,
123
+ rescale_factor=rescale_factor,
124
+ do_normalize=do_normalize,
125
+ image_mean=image_mean,
126
+ image_std=image_std,
127
+ do_resize=do_resize,
128
+ size=size,
129
+ resample=resample,
130
+ )
131
+
132
+ processed_images = []
133
+ for image in images:
134
+ image = to_numpy_array(image)
135
+ if do_convert_rgb:
136
+ image = self._convert_image_to_rgb(image)
137
+
138
+ if input_data_format is None:
139
+ try:
140
+ input_data_format = infer_channel_dimension_format(image)
141
+ except ValueError:
142
+ input_data_format = ChannelDimension.LAST
143
+
144
+ if do_resize:
145
+ image = _resize_multispectral(image, size=size, input_data_format=input_data_format)
146
+
147
+ if do_rescale:
148
+ image = image * rescale_factor
149
+
150
+ if do_normalize:
151
+ image = self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
152
+
153
+ image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
154
+ processed_images.append(image)
155
+
156
+ data = {"pixel_values": processed_images}
157
+ return BatchFeature(data=data, tensor_type=return_tensors)
158
+
159
+
160
+ __all__ = ["DeCURImageProcessor"]
decur-resnet50-rda-s1/modeling_decur.py ADDED
@@ -0,0 +1,285 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """Self-contained DeCUR model and config for trust_remote_code loading."""
3
+
4
+ from typing import Optional
5
+
6
+ import torch
7
+ import torch.nn as nn
8
+ from torchvision import models
9
+
10
+ from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
11
+ from transformers.modeling_outputs import BaseModelOutputWithPooling, ImageClassifierOutput
12
+ from transformers.modeling_utils import PreTrainedModel
13
+ from transformers.models.segformer.configuration_segformer import SegformerConfig
14
+ from transformers.models.segformer.modeling_segformer import SegformerModel
15
+ from transformers.models.vit.configuration_vit import ViTConfig
16
+ from transformers.models.vit.modeling_vit import ViTModel
17
+ from transformers.processing_utils import Unpack
18
+ from transformers.utils import TransformersKwargs, logging
19
+
20
+ from .dat_blocks import DAttentionBaseline
21
+
22
+
23
+ logger = logging.get_logger(__name__)
24
+
25
+
26
+ class DeCURConfig(PreTrainedConfig):
27
+ model_type = "decur"
28
+
29
+ def __init__(
30
+ self,
31
+ backbone="resnet50",
32
+ num_channels=13,
33
+ modality="s2c",
34
+ hidden_size=2048,
35
+ image_size=224,
36
+ patch_size=16,
37
+ use_rda=False,
38
+ num_hidden_layers=12,
39
+ num_attention_heads=6,
40
+ num_labels=0,
41
+ **kwargs,
42
+ ):
43
+ super().__init__(**kwargs)
44
+
45
+ self.backbone = backbone
46
+ self.num_channels = num_channels
47
+ self.modality = modality
48
+ self.hidden_size = hidden_size
49
+ self.image_size = image_size
50
+ self.patch_size = patch_size
51
+ self.use_rda = use_rda
52
+ self.num_hidden_layers = num_hidden_layers
53
+ self.num_attention_heads = num_attention_heads
54
+ self.num_labels = num_labels
55
+
56
+ def get_vit_config(self) -> ViTConfig:
57
+ if self.backbone != "vits16":
58
+ raise ValueError(f"Backbone '{self.backbone}' is not a ViT encoder.")
59
+
60
+ return ViTConfig(
61
+ hidden_size=self.hidden_size,
62
+ num_hidden_layers=self.num_hidden_layers,
63
+ num_attention_heads=self.num_attention_heads,
64
+ intermediate_size=self.hidden_size * 4,
65
+ image_size=self.image_size,
66
+ patch_size=self.patch_size,
67
+ num_channels=self.num_channels,
68
+ qkv_bias=True,
69
+ encoder_stride=self.patch_size,
70
+ )
71
+
72
+
73
+ def get_segformer_config(self) -> SegformerConfig:
74
+ if self.backbone not in {"mit_b2", "mit_b5"}:
75
+ raise ValueError(f"Backbone '{self.backbone}' is not a SegFormer encoder.")
76
+
77
+ depths = {"mit_b2": [3, 4, 6, 3], "mit_b5": [3, 6, 40, 3]}[self.backbone]
78
+ return SegformerConfig(
79
+ num_channels=self.num_channels,
80
+ hidden_sizes=[64, 128, 320, 512],
81
+ depths=depths,
82
+ num_attention_heads=[1, 2, 5, 8],
83
+ sr_ratios=[8, 4, 2, 1],
84
+ patch_sizes=[7, 3, 3, 3],
85
+ strides=[4, 2, 2, 2],
86
+ mlp_ratios=[4, 4, 4, 4],
87
+ hidden_act="gelu",
88
+ drop_path_rate=0.1,
89
+ reshape_last_stage=True,
90
+ )
91
+
92
+
93
+ class DeCURPreTrainedModel(PreTrainedModel):
94
+ config: DeCURConfig
95
+ base_model_prefix = "decur"
96
+ main_input_name = "pixel_values"
97
+ input_modalities = ("image",)
98
+ supports_gradient_checkpointing = False
99
+
100
+
101
+ def _build_resnet_encoder(config: DeCURConfig) -> nn.Module:
102
+ backbone = models.resnet50(weights=None)
103
+ if config.num_channels != 3:
104
+ backbone.conv1 = nn.Conv2d(config.num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
105
+ backbone.fc = nn.Identity()
106
+ return backbone
107
+
108
+
109
+ def _build_vit_encoder(config: DeCURConfig) -> ViTModel:
110
+ return ViTModel(config.get_vit_config(), add_pooling_layer=True)
111
+
112
+
113
+ def _build_segformer_encoder(config: DeCURConfig) -> SegformerModel:
114
+ return SegformerModel(config.get_segformer_config())
115
+
116
+
117
+ def _build_rda_modules() -> tuple[DAttentionBaseline, DAttentionBaseline]:
118
+ da_l3 = DAttentionBaseline(
119
+ q_size=(14, 14),
120
+ kv_size=(14, 14),
121
+ n_heads=8,
122
+ n_head_channels=128,
123
+ n_groups=4,
124
+ attn_drop=0,
125
+ proj_drop=0,
126
+ stride=2,
127
+ offset_range_factor=-1,
128
+ use_pe=True,
129
+ dwc_pe=False,
130
+ no_off=False,
131
+ fixed_pe=False,
132
+ ksize=5,
133
+ log_cpb=False,
134
+ )
135
+ da_l4 = DAttentionBaseline(
136
+ q_size=(7, 7),
137
+ kv_size=(7, 7),
138
+ n_heads=16,
139
+ n_head_channels=128,
140
+ n_groups=8,
141
+ attn_drop=0,
142
+ proj_drop=0,
143
+ stride=1,
144
+ offset_range_factor=-1,
145
+ use_pe=True,
146
+ dwc_pe=False,
147
+ no_off=False,
148
+ fixed_pe=False,
149
+ ksize=3,
150
+ log_cpb=False,
151
+ )
152
+ return da_l3, da_l4
153
+
154
+
155
+ def _build_encoder(config: DeCURConfig) -> nn.Module:
156
+ if config.backbone == "resnet50":
157
+ return _build_resnet_encoder(config)
158
+ if config.backbone == "vits16":
159
+ return _build_vit_encoder(config)
160
+ if config.backbone in {"mit_b2", "mit_b5"}:
161
+ return _build_segformer_encoder(config)
162
+ raise ValueError(f"Unsupported backbone '{config.backbone}'")
163
+
164
+
165
+ class DeCURModel(DeCURPreTrainedModel):
166
+ def __init__(self, config: DeCURConfig):
167
+ super().__init__(config)
168
+ self.encoder = _build_encoder(config)
169
+ if config.use_rda and config.backbone == "resnet50":
170
+ self.da_l3, self.da_l4 = _build_rda_modules()
171
+ else:
172
+ self.da_l3 = None
173
+ self.da_l4 = None
174
+ self.post_init()
175
+
176
+ def _forward_resnet(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
177
+ x = self.encoder.conv1(pixel_values)
178
+ x = self.encoder.bn1(x)
179
+ x = self.encoder.relu(x)
180
+ x = self.encoder.maxpool(x)
181
+ x = self.encoder.layer1(x)
182
+ x = self.encoder.layer2(x)
183
+ x = self.encoder.layer3(x)
184
+ if self.da_l3 is not None:
185
+ x1, _, _ = self.da_l3(x)
186
+ x = x + x1
187
+ x = self.encoder.layer4(x)
188
+ if self.da_l4 is not None:
189
+ x2, _, _ = self.da_l4(x)
190
+ x = x + x2
191
+ last_hidden_state = x.flatten(2).transpose(1, 2)
192
+ pooler_output = self.encoder.avgpool(x).flatten(1)
193
+ return last_hidden_state, pooler_output
194
+
195
+ def _forward_vit(
196
+ self,
197
+ pixel_values: torch.Tensor,
198
+ *,
199
+ interpolate_pos_encoding: bool = True,
200
+ ) -> tuple[torch.Tensor, torch.Tensor]:
201
+ outputs = self.encoder(
202
+ pixel_values=pixel_values,
203
+ interpolate_pos_encoding=interpolate_pos_encoding,
204
+ return_dict=True,
205
+ )
206
+ last_hidden_state = outputs.last_hidden_state
207
+ pooler_output = last_hidden_state[:, 0]
208
+ return last_hidden_state, pooler_output
209
+
210
+ def _forward_segformer(self, pixel_values: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
211
+ outputs = self.encoder(pixel_values=pixel_values, return_dict=True)
212
+ hidden = outputs.last_hidden_state
213
+ batch_size, channels, height, width = hidden.shape
214
+ sequence = hidden.permute(0, 2, 3, 1).reshape(batch_size, height * width, channels)
215
+ pooler_output = sequence.mean(dim=1)
216
+ return sequence, pooler_output
217
+
218
+ def forward(
219
+ self,
220
+ pixel_values: Optional[torch.Tensor] = None,
221
+ return_dict: Optional[bool] = None,
222
+ **kwargs: Unpack[TransformersKwargs],
223
+ ) -> BaseModelOutputWithPooling:
224
+ if pixel_values is None:
225
+ raise ValueError("You must specify `pixel_values`")
226
+
227
+ pixel_values = pixel_values.to(dtype=self.dtype)
228
+ if return_dict is None:
229
+ return_dict = self.config.use_return_dict
230
+
231
+ if self.config.backbone == "resnet50":
232
+ last_hidden_state, pooler_output = self._forward_resnet(pixel_values)
233
+ elif self.config.backbone == "vits16":
234
+ interpolate_pos_encoding = kwargs.pop("interpolate_pos_encoding", True)
235
+ last_hidden_state, pooler_output = self._forward_vit(
236
+ pixel_values,
237
+ interpolate_pos_encoding=interpolate_pos_encoding,
238
+ )
239
+ else:
240
+ last_hidden_state, pooler_output = self._forward_segformer(pixel_values)
241
+
242
+ if not return_dict:
243
+ return (last_hidden_state, pooler_output)
244
+
245
+ return BaseModelOutputWithPooling(last_hidden_state=last_hidden_state, pooler_output=pooler_output)
246
+
247
+
248
+ class DeCURForImageClassification(DeCURPreTrainedModel):
249
+ def __init__(self, config: DeCURConfig):
250
+ super().__init__(config)
251
+ self.decur = DeCURModel(config)
252
+ self.classifier = (
253
+ nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
254
+ )
255
+ self.post_init()
256
+
257
+ def forward(
258
+ self,
259
+ pixel_values: Optional[torch.Tensor] = None,
260
+ labels: Optional[torch.Tensor] = None,
261
+ return_dict: Optional[bool] = None,
262
+ **kwargs: Unpack[TransformersKwargs],
263
+ ) -> ImageClassifierOutput:
264
+ outputs = self.decur(pixel_values=pixel_values, return_dict=True, **kwargs)
265
+ logits = self.classifier(outputs.pooler_output)
266
+ loss = None
267
+ if labels is not None:
268
+ loss = self.loss_function(labels, logits, self.config, **kwargs)
269
+ if not return_dict:
270
+ output = (logits,) + outputs[1:]
271
+ return ((loss,) + output) if loss is not None else output
272
+ return ImageClassifierOutput(
273
+ loss=loss,
274
+ logits=logits,
275
+ hidden_states=outputs.hidden_states,
276
+ attentions=outputs.attentions,
277
+ )
278
+
279
+
280
+ __all__ = [
281
+ "DeCURConfig",
282
+ "DeCURForImageClassification",
283
+ "DeCURModel",
284
+ "DeCURPreTrainedModel",
285
+ ]
decur-resnet50-rda-s1/pipeline_decur.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2024 The DeCUR Authors and The HuggingFace Inc. team.
2
+ """DeCUR image feature extraction pipeline."""
3
+
4
+ from typing import Any, Union
5
+
6
+ from transformers.pipelines.base import GenericTensor, build_pipeline_init_args
7
+ from transformers.pipelines.image_feature_extraction import ImageFeatureExtractionPipeline
8
+ from transformers.utils import add_end_docstrings, is_vision_available
9
+
10
+
11
+ if is_vision_available():
12
+ from transformers.image_utils import load_image
13
+
14
+
15
+ @add_end_docstrings(
16
+ build_pipeline_init_args(has_image_processor=True),
17
+ """
18
+ pool (`bool`, *optional*, defaults to `False`):
19
+ Whether or not to return the pooled output. If `False`, the model will return the raw hidden states.
20
+ """,
21
+ )
22
+ class DeCURImageFeatureExtractionPipeline(ImageFeatureExtractionPipeline):
23
+
24
+ def _sanitize_parameters(
25
+ self,
26
+ image_processor_kwargs=None,
27
+ return_tensors=None,
28
+ pool=None,
29
+ **kwargs,
30
+ ):
31
+ preprocess_params = {} if image_processor_kwargs is None else dict(image_processor_kwargs)
32
+ if "timeout" in kwargs:
33
+ preprocess_params["timeout"] = kwargs["timeout"]
34
+
35
+ postprocess_params = {}
36
+ if pool is not None:
37
+ postprocess_params["pool"] = pool
38
+ if return_tensors is not None:
39
+ postprocess_params["return_tensors"] = return_tensors
40
+
41
+ return preprocess_params, {}, postprocess_params
42
+
43
+ def preprocess(self, image, timeout=None, **image_processor_kwargs) -> dict[str, GenericTensor]:
44
+ if not isinstance(image, (list, tuple)) and not hasattr(image, "shape"):
45
+ image = load_image(image, timeout=timeout)
46
+ model_inputs = self.image_processor(image, return_tensors="pt", **image_processor_kwargs)
47
+ model_inputs = model_inputs.to(self.dtype)
48
+ return model_inputs
49
+
50
+ def __call__(
51
+ self,
52
+ *args: Union[str, Any, list[Any]],
53
+ **kwargs: Any,
54
+ ) -> list[Any]:
55
+ return super().__call__(*args, **kwargs)
56
+
57
+
58
+ __all__ = ["DeCURImageFeatureExtractionPipeline"]
decur-resnet50-rda-s1/preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor_type": "DeCURImageProcessor",
3
+ "size": {
4
+ "height": 224,
5
+ "width": 224
6
+ },
7
+ "do_resize": false,
8
+ "do_rescale": true,
9
+ "do_normalize": false,
10
+ "do_convert_rgb": false,
11
+ "rescale_factor": 0.00392156862745098,
12
+ "auto_map": {
13
+ "AutoImageProcessor": "image_processing_decur.DeCURImageProcessor"
14
+ }
15
+ }