CLIP图文多模态对比学习

CLIP简介

Repo: https://github.com/openai/CLIP

CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.

方法

  • 在大规模数据集上使用 NLP 监督预训练图像分类器
  • 4亿对来自网络的图文数据对,将文本作为图像标签,进行训练。进行下游任务时,只需要提供和图上的 concepts 对应的文本描述,就可以进行 zero-shot transfer
    • 考虑到大部分的数据集的标签都是以单词的形式存在的,比如“bird”,“cat”等等,然而在预训练阶段的文本描述大多都是某个短句,为了填补这种数据分布上的差别,作者考虑用“指示上下文”(guide context)对标签进行扩展。可以用a photo of a “object".作为文本端的输入,其中的 object 恰恰是需要预测的zero-shot标签。(100个类别就是100个文本描述)
  • 双流,2 个 encoder 分别处理文本和图片数据,text encoder 使用 Transformer,image encoder 用了 2 种模型,ResNetVision Transformer(ViT)
    • ResNet-50(5种ResNet:ResNet-50, ResNet-101,三种缩放的 EfficientNet-style 的ResNet,包括RN50x4, RN50x16, RN50x64)
    • Vision Transformer(ViT)(3 种 ViT:ViT-B/32, ViT-B/16, ViT-L/14)
  • encoder representation直接线性投影到 multi-modal embedding space;
  • 计算 2 模态之间的cosine similarity,让 N 个匹配的图文对相似度最大,不匹配的图文对相似度最小;
  • 对称的 cross-entropy loss;
    • 目标函数定义为:最大化the cosine similarity of the image and text embeddings of the N real pairs in the batch,最小化the cosine similarity of the embeddings of the incorrect pairings. 在这些 similarity scores 上使用了一个对称的 cross entropy loss,称之为 multi-class N-pair loss。
  • 数据增强:对resized图片进行random square crop。
  • CLIP 是从头开始训练的,没有使用预训练的初始参数

超参数

  • We train all models for 32 epochs.

  • Initial hyperparameters were set using a combination of grid searches, random search, and manual tuning on the baseline ResNet-50 model when trained for 1 epoch.

  • The learnable temperature parameter τ was initialized to the equivalent of 0.07 from (Wu et al.,2018) and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training instability.

  • We usea very large minibatch size of 32,768.

  • Mixed-precision (Micikevicius et al., 2017) was used to accelerate training and save memory.

  • To save additional memory, gradient checkpointing (Griewank & Walther, 2000; Chen et al., 2016), half-precision Adam statistics (Dhariwalet al., 2020), and half-precision stochastically rounded text encoder weights were used.

  • The largest ResNet model, RN50x64, took 18 days to train on 592 V100 GPUs while the largest Vision Transformer took 12 days on 256 V100 GPUs.

关键代码

官方

model.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
from collections import OrderedDict
from typing import Tuple, Union

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn


class Bottleneck(nn.Module):
expansion = 4

def __init__(self, inplanes, planes, stride=1):
super().__init__()

# all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)

self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)

self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()

self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
self.bn3 = nn.BatchNorm2d(planes * self.expansion)

self.relu = nn.ReLU(inplace=True)
self.downsample = None
self.stride = stride

if stride > 1 or inplanes != planes * Bottleneck.expansion:
# downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
self.downsample = nn.Sequential(OrderedDict([
("-1", nn.AvgPool2d(stride)),
("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
("1", nn.BatchNorm2d(planes * self.expansion))
]))

def forward(self, x: torch.Tensor):
identity = x

out = self.relu(self.bn1(self.conv1(x)))
out = self.relu(self.bn2(self.conv2(out)))
out = self.avgpool(out)
out = self.bn3(self.conv3(out))

if self.downsample is not None:
identity = self.downsample(x)

out += identity
out = self.relu(out)
return out


class AttentionPool2d(nn.Module):
def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
super().__init__()
self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
self.k_proj = nn.Linear(embed_dim, embed_dim)
self.q_proj = nn.Linear(embed_dim, embed_dim)
self.v_proj = nn.Linear(embed_dim, embed_dim)
self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
self.num_heads = num_heads

def forward(self, x):
x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1) # NCHW -> (HW)NC
x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0) # (HW+1)NC
x = x + self.positional_embedding[:, None, :].to(x.dtype) # (HW+1)NC
x, _ = F.multi_head_attention_forward(
query=x, key=x, value=x,
embed_dim_to_check=x.shape[-1],
num_heads=self.num_heads,
q_proj_weight=self.q_proj.weight,
k_proj_weight=self.k_proj.weight,
v_proj_weight=self.v_proj.weight,
in_proj_weight=None,
in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
bias_k=None,
bias_v=None,
add_zero_attn=False,
dropout_p=0,
out_proj_weight=self.c_proj.weight,
out_proj_bias=self.c_proj.bias,
use_separate_proj_weight=True,
training=self.training,
need_weights=False
)

return x[0]


class ModifiedResNet(nn.Module):
"""
A ResNet class that is similar to torchvision's but contains the following changes:
- There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
- Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
- The final pooling layer is a QKV attention instead of an average pool
"""

def __init__(self, layers, output_dim, heads, input_resolution=224, width=64):
super().__init__()
self.output_dim = output_dim
self.input_resolution = input_resolution

# the 3-layer stem
self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(width // 2)
self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(width // 2)
self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
self.bn3 = nn.BatchNorm2d(width)
self.avgpool = nn.AvgPool2d(2)
self.relu = nn.ReLU(inplace=True)

# residual layers
self._inplanes = width # this is a *mutable* variable used during construction
self.layer1 = self._make_layer(width, layers[0])
self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
self.layer4 = self._make_layer(width * 8, layers[3], stride=2)

embed_dim = width * 32 # the ResNet feature dimension
self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)

def _make_layer(self, planes, blocks, stride=1):
layers = [Bottleneck(self._inplanes, planes, stride)]

self._inplanes = planes * Bottleneck.expansion
for _ in range(1, blocks):
layers.append(Bottleneck(self._inplanes, planes))

return nn.Sequential(*layers)

def forward(self, x):
def stem(x):
for conv, bn in [(self.conv1, self.bn1), (self.conv2, self.bn2), (self.conv3, self.bn3)]:
x = self.relu(bn(conv(x)))
x = self.avgpool(x)
return x

x = x.type(self.conv1.weight.dtype)
x = stem(x)
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.layer4(x)
x = self.attnpool(x)

return x


class LayerNorm(nn.LayerNorm):
"""Subclass torch's LayerNorm to handle fp16."""

def forward(self, x: torch.Tensor):
orig_type = x.dtype
ret = super().forward(x.type(torch.float32))
return ret.type(orig_type)


class QuickGELU(nn.Module):
def forward(self, x: torch.Tensor):
return x * torch.sigmoid(1.702 * x)


class ResidualAttentionBlock(nn.Module):
def __init__(self, d_model: int, n_head: int, attn_mask: torch.Tensor = None):
super().__init__()

self.attn = nn.MultiheadAttention(d_model, n_head)
self.ln_1 = LayerNorm(d_model)
self.mlp = nn.Sequential(OrderedDict([
("c_fc", nn.Linear(d_model, d_model * 4)),
("gelu", QuickGELU()),
("c_proj", nn.Linear(d_model * 4, d_model))
]))
self.ln_2 = LayerNorm(d_model)
self.attn_mask = attn_mask

def attention(self, x: torch.Tensor):
self.attn_mask = self.attn_mask.to(dtype=x.dtype, device=x.device) if self.attn_mask is not None else None
return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]

def forward(self, x: torch.Tensor):
x = x + self.attention(self.ln_1(x))
x = x + self.mlp(self.ln_2(x))
return x


class Transformer(nn.Module):
def __init__(self, width: int, layers: int, heads: int, attn_mask: torch.Tensor = None):
super().__init__()
self.width = width
self.layers = layers
self.resblocks = nn.Sequential(*[ResidualAttentionBlock(width, heads, attn_mask) for _ in range(layers)])

def forward(self, x: torch.Tensor):
return self.resblocks(x)


class VisionTransformer(nn.Module):
def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int):
super().__init__()
self.input_resolution = input_resolution
self.output_dim = output_dim
self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)

scale = width ** -0.5
self.class_embedding = nn.Parameter(scale * torch.randn(width))
self.positional_embedding = nn.Parameter(scale * torch.randn((input_resolution // patch_size) ** 2 + 1, width))
self.ln_pre = LayerNorm(width)

self.transformer = Transformer(width, layers, heads)

self.ln_post = LayerNorm(width)
self.proj = nn.Parameter(scale * torch.randn(width, output_dim))

def forward(self, x: torch.Tensor):
x = self.conv1(x) # shape = [*, width, grid, grid]
x = x.reshape(x.shape[0], x.shape[1], -1) # shape = [*, width, grid ** 2]
x = x.permute(0, 2, 1) # shape = [*, grid ** 2, width]
x = torch.cat([self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1) # shape = [*, grid ** 2 + 1, width]
x = x + self.positional_embedding.to(x.dtype)
x = self.ln_pre(x)

x = x.permute(1, 0, 2) # NLD -> LND
x = self.transformer(x)
x = x.permute(1, 0, 2) # LND -> NLD

x = self.ln_post(x[:, 0, :])

if self.proj is not None:
x = x @ self.proj

return x


class CLIP(nn.Module):
def __init__(self,
embed_dim: int,
# vision
image_resolution: int,
vision_layers: Union[Tuple[int, int, int, int], int],
vision_width: int,
vision_patch_size: int,
# text
context_length: int,
vocab_size: int,
transformer_width: int,
transformer_heads: int,
transformer_layers: int
):
super().__init__()

self.context_length = context_length

if isinstance(vision_layers, (tuple, list)):
vision_heads = vision_width * 32 // 64
self.visual = ModifiedResNet(
layers=vision_layers,
output_dim=embed_dim,
heads=vision_heads,
input_resolution=image_resolution,
width=vision_width
)
else:
vision_heads = vision_width // 64
self.visual = VisionTransformer(
input_resolution=image_resolution,
patch_size=vision_patch_size,
width=vision_width,
layers=vision_layers,
heads=vision_heads,
output_dim=embed_dim
)

self.transformer = Transformer(
width=transformer_width,
layers=transformer_layers,
heads=transformer_heads,
attn_mask=self.build_attention_mask()
)

self.vocab_size = vocab_size
self.token_embedding = nn.Embedding(vocab_size, transformer_width)
self.positional_embedding = nn.Parameter(torch.empty(self.context_length, transformer_width))
self.ln_final = LayerNorm(transformer_width)

self.text_projection = nn.Parameter(torch.empty(transformer_width, embed_dim))
self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))

self.initialize_parameters()

def initialize_parameters(self):
nn.init.normal_(self.token_embedding.weight, std=0.02)
nn.init.normal_(self.positional_embedding, std=0.01)

if isinstance(self.visual, ModifiedResNet):
if self.visual.attnpool is not None:
std = self.visual.attnpool.c_proj.in_features ** -0.5
nn.init.normal_(self.visual.attnpool.q_proj.weight, std=std)
nn.init.normal_(self.visual.attnpool.k_proj.weight, std=std)
nn.init.normal_(self.visual.attnpool.v_proj.weight, std=std)
nn.init.normal_(self.visual.attnpool.c_proj.weight, std=std)

for resnet_block in [self.visual.layer1, self.visual.layer2, self.visual.layer3, self.visual.layer4]:
for name, param in resnet_block.named_parameters():
if name.endswith("bn3.weight"):
nn.init.zeros_(param)

proj_std = (self.transformer.width ** -0.5) * ((2 * self.transformer.layers) ** -0.5)
attn_std = self.transformer.width ** -0.5
fc_std = (2 * self.transformer.width) ** -0.5
for block in self.transformer.resblocks:
nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)

if self.text_projection is not None:
nn.init.normal_(self.text_projection, std=self.transformer.width ** -0.5)

def build_attention_mask(self):
# lazily create causal attention mask, with full attention between the vision tokens
# pytorch uses additive attention mask; fill with -inf
mask = torch.empty(self.context_length, self.context_length)
mask.fill_(float("-inf"))
mask.triu_(1) # zero out the lower diagonal
return mask

@property
def dtype(self):
return self.visual.conv1.weight.dtype

def encode_image(self, image):
return self.visual(image.type(self.dtype))

def encode_text(self, text):
x = self.token_embedding(text).type(self.dtype) # [batch_size, n_ctx, d_model]

x = x + self.positional_embedding.type(self.dtype)
x = x.permute(1, 0, 2) # NLD -> LND
x = self.transformer(x)
x = x.permute(1, 0, 2) # LND -> NLD
x = self.ln_final(x).type(self.dtype)

# x.shape = [batch_size, n_ctx, transformer.width]
# take features from the eot embedding (eot_token is the highest number in each sequence)
x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection

return x

def forward(self, image, text):
image_features = self.encode_image(image)
text_features = self.encode_text(text)

# normalized features
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)

# cosine similarity as logits
logit_scale = self.logit_scale.exp()
logits_per_image = logit_scale * image_features @ text_features.t()
logits_per_text = logits_per_image.t()

# shape = [global_batch_size, global_batch_size]
return logits_per_image, logits_per_text


def convert_weights(model: nn.Module):
"""Convert applicable model parameters to fp16"""

def _convert_weights_to_fp16(l):
if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
l.weight.data = l.weight.data.half()
if l.bias is not None:
l.bias.data = l.bias.data.half()

if isinstance(l, nn.MultiheadAttention):
for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
tensor = getattr(l, attr)
if tensor is not None:
tensor.data = tensor.data.half()

for name in ["text_projection", "proj"]:
if hasattr(l, name):
attr = getattr(l, name)
if attr is not None:
attr.data = attr.data.half()

model.apply(_convert_weights_to_fp16)


def build_model(state_dict: dict):
vit = "visual.proj" in state_dict

if vit:
vision_width = state_dict["visual.conv1.weight"].shape[0]
vision_layers = len([k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
image_resolution = vision_patch_size * grid_size
else:
counts: list = [len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in [1, 2, 3, 4]]
vision_layers = tuple(counts)
vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
vision_patch_size = None
assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
image_resolution = output_width * 32

embed_dim = state_dict["text_projection"].shape[1]
context_length = state_dict["positional_embedding"].shape[0]
vocab_size = state_dict["token_embedding.weight"].shape[0]
transformer_width = state_dict["ln_final.weight"].shape[0]
transformer_heads = transformer_width // 64
transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith(f"transformer.resblocks")))

model = CLIP(
embed_dim,
image_resolution, vision_layers, vision_width, vision_patch_size,
context_length, vocab_size, transformer_width, transformer_heads, transformer_layers
)

for key in ["input_resolution", "context_length", "vocab_size"]:
if key in state_dict:
del state_dict[key]

convert_weights(model)
model.load_state_dict(state_dict)
return model.eval()

Calculating cosine similarity

Milvus 中的内积 (IP)https://milvus.io/cn/docs/v2.0.0/metric.md#floating

image-20211111003358477

We normalize the features and calculate the dot product of each pair.

1
2
3
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = text_features.cpu().numpy() @ image_features.cpu().numpy().T

Zero-Shot Image Classification

1
2
3
4
5
6
7
8
9
10
11
12
13
# label 转换为 text
text_descriptions = [f"This is a photo of a {label}" for label in cifar100.classes]
text_tokens = clip.tokenize(text_descriptions).cuda()

# Calculate features
with torch.no_grad(): # 关闭自动求导引擎,model.eval() 在测试/验证不用 Dropout 表示 dropout=0
text_features = model.encode_text(text_tokens).float()
text_features /= text_features.norm(dim=-1, keepdim=True)

# You can classify images using the cosine similarity (times 100) as the logits to the softmax operation.
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
# Pick the top 5 most similar labels for the image
top_probs, top_labels = text_probs.cpu().topk(5, dim=-1)

非官方

OpenAI CLIP with train

  • 训练
  • FAISS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
_tokenizer = SimpleTokenizer()
# Copied from https://github.com/openai/CLIP/blob/beba48f35392a73c6c47ae67ddffced81ad1916d/clip/clip.py#L164
# but with relaxed exception
def tokenize(texts, context_length: int = 77) -> torch.LongTensor:
if isinstance(texts, str):
texts = [texts]

sot_token = _tokenizer.encoder["<|startoftext|>"]
eot_token = _tokenizer.encoder["<|endoftext|>"]
all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)

for i, tokens in enumerate(all_tokens):
n = min(len(tokens), context_length)
result[i, :n] = torch.tensor(tokens)[:n]
if len(tokens) > context_length:
result[i, -1] = tokens[-1]

return result


# Remove EMOJI
## compile 函数根据一个模式字符串和可选的标志参数生成一个正则表达式对象。该对象拥有一系列方法用于正则表达式匹配和替换。
RE_EMOJI = re.compile(r"\\x[A-Za-z0-9./]+", flags=re.UNICODE)
def strip_emoji(text):
return RE_EMOJI.sub(r'', text)


class RollingMean():
def __init__(self):
self.n = 0
self.mean = 0

def update(self, value):
self.mean = (self.mean * self.n + value) / (self.n + 1)
self.n += 1

def result(self):
return self.mean

'''
Sampler and dataset
We implement a sampler that ensures that in every batch, two samples of the same group are always present.
This is important in order to use Triplet SemiHardLoss (I'm using this implementation)
'''
class SameGroupSampler(Sampler):
def __init__(self, df ,ds):
super().__init__(ds)

# Create a dictionary of posting_id -> index in dataset
self.index_to_position = dict(zip(df.index, range(len(df))))

# Create a Series of label_group -> set(posting_id)
self.label_group = df.reset_index().groupby('label_group')['posting_id'].apply(set).map(sorted).map(np.array)

def __len__(self):
return len(self.label_group)

def __iter__(self):
for _ in range(len(self)):
# Sample one label_group
label_group_sample = self.label_group.sample(1).iloc[0]

# Sample two posting_id's
sample1, sample2 = np.random.choice(label_group_sample, 2, replace=False)

yield self.index_to_position[sample1]
yield self.index_to_position[sample2]

class MyDataset(Dataset):
def __init__(self, df, images_path):
super().__init__()
self.df = df
self.images_path = images_path
self.has_target = ('label_group' in df)

def __len__(self):
return len(self.df)

def __getitem__(self, idx):
row = self.df.iloc[idx]

image = preprocess(Image.open(self.images_path / row['image']))
text = tokenize([strip_emoji(row['title'])])[0]

if self.has_target:
return image, text, row['label_group']
else:
return image, text, 0

Finetune CLIP on train data

1
2
3
4
5
6
7
# Load CLIP
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("../input/openai-clip/ViT-B-32.pt", device=device, jit=False)

# Get embedding size
embed_dim = model.text_projection.shape[1]
embed_dim
1
2
3
4
5
6
# Load train data
train_images_path = Path('../input/shopee-product-matching/train_images')
df_train = pd.read_csv('../input/shopee-product-matching/train.csv', index_col='posting_id')

dstrain = MyDataset(df_train, train_images_path)
dltrain = DataLoader(dstrain, batch_size=128, num_workers=2, sampler=SameGroupSampler(df_train, dstrain))
1
2
3
4
5
6
7
n_epochs = 1

# optim = torch.optim.AdamW(model.parameters(), lr=1e-4, eps=1e-8, weight_decay=1e-2)
optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.2)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optim, 1e-2, total_steps=n_epochs * (2*len(dltrain)-1),
base_momentum=0.0, max_momentum=0.5, pct_start=0.1, div_factor=1e2, final_div_factor=1e4)
criterion = TripletLoss(device)

Triplet loss

Here we use the triplet loss principe to ajust CLIP:

对于三元组 anchor,positive,negative 而言,anchor 为训练集中的一个随机样本,positive 为与 anchor 同类的一个样本,negative 为与 anchor 不同类的一个样本。Tript loss 的作用是最小化 positive 与 anchor 之间的距离,而最大化 negative 与 anchor 之间的距离。

img

[公式] [公式] 表示anchor和positive的欧氏距离,[公式] 表示 positive 和 negative 之间的距离至少为此数。

  • easy triplets (简单三元组):triplet 对应的损失为 0 的三元组,形式化定义为d(a,n)>d(a,p)+margin,也就是负样本的距离远大于正样本的距离。
  • hard triplets(困难三元组):negative example 与 anchor 距离小于 anchor 与 positive example 的距离,形式化定义为 $$d(a,n)<d(a,p)$$,也就是负样本的距离远小于正样本的距离,意味着是易混淆的 case。
  • semi-hard triplets(一般三元组):negative example 与 anchor 距离大于 anchor 与 positive example 的距离,但还不至于使得 loss 为0,即 $$d(a,p)<d(a,n)<d(a,p)+margin$$,依旧是介于能区分与容易区分之间,有差距但是差距不够大。负样本的距离虽比正样本大,但不满足间隔裕量margin。此时损失 [公式] 大于 0,但小于margin。

Triplet loss

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
for epoch in range(n_epochs):
with tqdm(total=2*len(dltrain)-1) as bar:
loss_mean = RollingMean()
for images, texts, targets in dltrain:
targets = targets.to(device)

# Generate train and text features
images_features = model.encode_image(images.to(device))
texts_features = model.encode_text(texts.to(device))

optim.zero_grad()

# Join train and test features
features = torch.hstack([images_features, texts_features])

# L2-normalize features
features = features / features.norm(2, dim=1, keepdim=True)

# from triplet_loss import TripletLoss
# criterion = TripletLoss(device)
# Apply Triplet SemiHardLoss
loss = criterion(features, targets)

loss.backward()
optim.step()
scheduler.step()

# Update metric and progress bar
loss_mean.update(loss.item())
bar.update()
bar.set_description('{:.4f}'.format(loss_mean.result()))

Run on train

In this section we will generate features using CLIP and perform a similiarity search to find the closest matches.

We create the final set by taking away those results bellow a threshold similiarity (less 0.7)

FAISS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
def find_similarities_and_indexes(df, images_path, top_n=100, features_file=None):
# Create pytorch Dataset/DataLoader
ds = MyDataset(df, images_path)
dl = DataLoader(ds, batch_size=32, shuffle=False, num_workers=2)

# Allocate memory for features
features = np.empty((len(df), 2*embed_dim), dtype=np.float32)

# Begin predict
i = 0
for images, texts, _ in tqdm(dl):
n = len(images)
with torch.no_grad():
# Generate image and text features
images_features = model.encode_image(images.to(device))
texts_features = model.encode_text(texts.to(device))

# Concat features (first images then texts)
features[i:i+n, :embed_dim] = images_features.cpu()
features[i:i+n, embed_dim:] = texts_features.cpu()

i += n

# Option to save these features (may be usefull to tune cut value)
if features_file is not None:
np.save(features_file, features)

# l2-normalize
features /= np.linalg.norm(features, 2, axis=1, keepdims=True)

# Create index
index = faiss.IndexFlatIP(2*embed_dim)
index.add(features)

# Search index
return index.search(features, top_n)

# TODO: try range_search
# lims, similarities, indexes = index_test.range_search(test_features, GROUP_CUT)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
if RUN_ON_TRAIN:
# Perform search of similiar items
similarities, indexes = find_similarities_and_indexes(df_train, train_images_path, features_file='features-no-norm.npy')

# `similarities` will have shape (n, 100) and will have the similarites scores for closest matches
# `indexes` will have shape (n, 100) and have the index closest matches.
# Both arrays are aligned

# Convert index to groups, will have shape (n, 100)
found_groups = df_train['label_group'].values[indexes]

# Check if matches are from same group. Will create a boolean vector of (n, 100)
is_same_group = (found_groups == df_train['label_group'].values[:, np.newaxis])

# Plot similarities score from same group and different groups
plt.figure(figsize=(10, 5))
plt.hist([similarities[is_same_group], similarities[~is_same_group]], density=False, bins=51,
label=['Same group', 'Different group'], histtype='stepfilled', alpha=0.75)
plt.xlim(0, 1)
plt.xlabel('Similarity score')
plt.legend();

Tune CUT

In this last step we will move the cut_value to find optimal F1-score.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# SRC: https://www.kaggle.com/c/shopee-product-matching/discussion/224782#1233338
# With some adaptation
def row_wise_f1_score(y_true, y_pred):
tp = np.array([len(x[0] & x[1]) for x in zip(y_true, y_pred)])
fp = y_pred.apply(lambda x: len(x)).values - tp
fn = y_true.apply(lambda x: len(x)).values - tp

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * ((precision * recall) / (precision + recall))
return f1

def calc_score(cut_value):
# Apply cutoff of similarities
groups_are_same = (similarities > cut_value)

# Build results
results = []
for i, (group_is_same, index_result) in enumerate(zip(groups_are_same, indexes)):
row_results = df_train.index[index_result[group_is_same]]

# Keep found matches as a `set`
results.append(set(row_results))

df_results = pd.Series(results, index=df_answer.index)

# Evaluate results
return row_wise_f1_score(df_answer, df_results).mean()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
if RUN_ON_TRAIN:
# Create answer dataframe. This will have posting_id on index and a set of label_group as values
groups = df_train.reset_index().groupby('label_group')['posting_id'].apply(set)
df_answer = df_train['label_group'].map(groups)

# Cut values to evaluate
cuts = np.linspace(0.5, 0.95, 51)
scores = [calc_score(c) for c in tqdm(cuts)]

# Plot curve
plt.plot(cuts, scores)
plt.xlabel('Cutoff value')
plt.ylabel('F1 score')

print('Best cutoff is {:.2f} with expected F1 score of {:.4f}'.format(cuts[np.argmax(scores)], max(scores)))

OpenAI CLIP simple implementation

  • DistilBert + RN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# ImageEncoder
class ImageEncoder(nn.Module):
"""
Encode images to a fixed size vector
"""

def __init__(
self, model_name=CFG.model_name, pretrained=CFG.pretrained, trainable=CFG.trainable
):
super().__init__()
self.model = timm.create_model(
model_name, pretrained, num_classes=0, global_pool="avg"
)
for p in self.model.parameters():
p.requires_grad = trainable

def forward(self, x):
return self.model(x)

# TextEncoder
class TextEncoder(nn.Module):
def __init__(self, model_name=CFG.text_encoder_model, pretrained=CFG.pretrained, trainable=CFG.trainable):
super().__init__()
if pretrained:
self.model = DistilBertModel.from_pretrained(model_name)
else:
self.model = DistilBertModel(config=DistilBertConfig())

for p in self.model.parameters():
p.requires_grad = trainable

# we are using the CLS token hidden representation as the sentence's embedding
self.target_token_idx = 0

def forward(self, input_ids, attention_mask):
output = self.model(input_ids=input_ids, attention_mask=attention_mask)
last_hidden_state = output.last_hidden_state
return last_hidden_state[:, self.target_token_idx, :]

Projetcion Head

通过线性映射层 Projetcion Head 将图片特征[公式]和文本特征[公式]都映射到相同的嵌入特征维度[公式]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Projetcion Head
class ProjectionHead(nn.Module):
def __init__(
self,
embedding_dim,
projection_dim=CFG.projection_dim,
dropout=CFG.dropout
):
super().__init__()
self.projection = nn.Linear(embedding_dim, projection_dim)
self.gelu = nn.GELU()
self.fc = nn.Linear(projection_dim, projection_dim)
self.dropout = nn.Dropout(dropout)
self.layer_norm = nn.LayerNorm(projection_dim)

def forward(self, x):
projected = self.projection(x)
x = self.gelu(projected)
x = self.fc(x)
x = self.dropout(x)
x = x + projected
x = self.layer_norm(x)
return x

CLIPModel

直接用文本的 encoding 结果做为图像的监督信号,显然噪声太大了?是否可以早点跟图像特征早点交叉。

采用 InfoNCE: info Noise Contrastive Estimation loss [公式]

$$t$$ 温度系数的作用是调节对困难样本的关注程度:越小的温度系数越关注于将本样本和最相似的困难样本分开,去得到更均匀的表示。然而困难样本往往是与本样本相似程度较高的,很多困难负样本其实是潜在的正样本,过分强迫与困难样本分开会破坏学到的潜在语义结构,因此,温度系数不能过小。

考虑两个极端情况,温度系数趋向于 0 时,对比损失退化为只关注最困难的负样本的损失函数;当温度系数趋向于无穷大时,对比损失对所有负样本都一视同仁,失去了困难样本关注的特性。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
class CLIPModel(nn.Module):
def __init__(
self,
temperature=CFG.temperature,
image_embedding=CFG.image_embedding,
text_embedding=CFG.text_embedding,
):
super().__init__()
self.image_encoder = ImageEncoder()
self.text_encoder = TextEncoder()
self.image_projection = ProjectionHead(embedding_dim=image_embedding)
self.text_projection = ProjectionHead(embedding_dim=text_embedding)
self.temperature = temperature

def forward(self, batch):
# Getting Image and Text Features
image_features = self.image_encoder(batch["image"])
text_features = self.text_encoder(
input_ids=batch["input_ids"], attention_mask=batch["attention_mask"]
)
# Getting Image and Text Embeddings (with same dimension)
image_embeddings = self.image_projection(image_features)
text_embeddings = self.text_projection(text_features)

# Normalize

# Calculating the Loss
# Temperature
logits = (text_embeddings @ image_embeddings.T) / self.temperature

images_similarity = image_embeddings @ image_embeddings.T
texts_similarity = text_embeddings @ text_embeddings.T

targets = F.softmax(
(images_similarity + texts_similarity) / 2 * self.temperature, dim=-1
)

# cross_entropy
texts_loss = cross_entropy(logits, targets, reduction='none')
images_loss = cross_entropy(logits.T, targets.T, reduction='none')

loss = (images_loss + texts_loss) / 2.0 # shape: (batch_size)
return loss.mean()

def cross_entropy(preds, targets, reduction='none'):
log_softmax = nn.LogSoftmax(dim=-1)
loss = (-targets * log_softmax(preds)).sum(1)
if reduction == "none":
return loss
elif reduction == "mean":
return loss.mean()

CLIP:从自然语言监督中学习可迁移的视觉模型

img

CLIP应用

微博 W-CLIP

对博文-图片来进行表示学习。除了正例构造方法和模型小细节外,博文-图片多模态模型的整体结构和 CLIP 比较接近,所以,我们将这个使用包含大量噪音微博文图数据的多模态模型称为 W-CLIP(Weibo-CLIP)。

img

其他

img
  • 本文作者: Ashin Wang
  • 本文链接: https://blog.ashin.wang/clip/
  • 版权声明: 本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!