About sim Loss

[muyuuuu]

Hello, thanks for you code, I'm als prepare to use SSL to improve robustness. Some question:

RoCL/src/loss.py

Line 59 in b6d5185

def NT_xent(similarity_matrix, adv_type):

This loss function is from paper simCLR ? I also decided to use this loss function.

RoCL/src/rocl_train.py

Lines 154 to 155 in b6d5185

	if not (args.advtrain_type=='None'):
	loss = simloss + adv_loss

But I found that it converges very slowly and has much more value than classification function. Have you encountered this problem?

Here is my implement and log.

def InfoNCE(x1, x2, device):

    bs = x1.shape[0]
    feature = torch.cat((x1, x2), dim=0)
    feature = torch.nn.functional.normalize(feature, dim=1)

    similarity_matrix = torch.matmul(feature, feature.T)
    labels = torch.cat([torch.arange(bs) for i in range(2)], dim=0)
    labels = (labels.unsqueeze(0) == labels.unsqueeze(1)).float()
    mask = torch.eye(labels.shape[0], dtype=torch.bool)
    labels = labels[~mask].view(labels.shape[0], -1)
    similarity_matrix = similarity_matrix[~mask].view(similarity_matrix.shape[0], -1)
    positives = similarity_matrix[labels.bool()].view(labels.shape[0], -1)
    negatives = similarity_matrix[~labels.bool()].view(similarity_matrix.shape[0], -1)

    logits = torch.cat((positives, negatives), dim=1).to(device)
    labels = torch.zeros(logits.size()[0], dtype=torch.long).to(device)

    return logits, labels

Model: Wed Dec 29 01:10:13 2021 : clean class loss is 0.031855, attack class loss is 0.889221, sim loss is 4.037420.
Model: Wed Dec 29 01:10:48 2021 : clean class loss is 0.034873, attack class loss is 0.386264, sim loss is 4.034396.
Model: Wed Dec 29 01:11:23 2021 : clean class loss is 0.034596, attack class loss is 0.376193, sim loss is 4.035690.