0

I have two views of one object and using multi view cnn from .https://github.com/SAMY-ER/Multi-View-Image-Classification. This works great and I am getting better results. Though how can I apply grad cam to it. any help is appreciated . I want to see the cam on both views to see what parts in two images contributing to classification This is my model

MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE

class MVCNN(nn.Module):
   def __init__(self, num_classes=1000, pretrained=True):
    super(MVCNN, self).__init__()
   
    
    resnet = models.resnet50(pretrained = pretrained)
    
    fc_in_features = resnet.fc.in_features
    self.features = nn.Sequential(*list(resnet.children())[:-1])
    self.classifier = nn.Sequential(
        nn.Dropout(),
        ## multiplying by 2 to take care of two views
        nn.Linear(fc_in_features * 2, 2048),
        nn.ReLU(inplace=True),
        nn.Dropout(),
        nn.Linear(2048, 2048),
        nn.ReLU(inplace=True),
        nn.Linear(2048, num_classes)
    )
        
def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
    inputs = inputs.transpose(0, 1)
    view_features = [] 
    for view_batch in inputs:
        view_batch = self.features(view_batch)
        view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
        view_features.append(view_batch)   
        
    concat_views = torch.cat(view_features,-1)
    #pooled_views, _ = torch.max(torch.stack(view_features), 0)
    #outputs = self.classifier(pooled_views)
    outputs = self.classifier(concat_views)
    return outputs
user1631306
  • 81
  • 5
  • grad-cam is calculated w.r.t. a target class - you should start with calculating the predictions score (usually `1-Predicted_score`) and backpropagating the error to a desired CNN layer – Aray Karjauv May 08 '23 at 11:57
  • true, but i am merging two CNN layers from two different brances together. So, ideally i would like to see different activations in different branches/images. How do i do that? – user1631306 May 08 '23 at 14:44
  • I'm not familiar with this architecture but if I understand correctly you have different tails (output branches). In this case, you can backpropagate each branch separately to a shared CNN layer and then merge results or you could do the same for one of the last layers in each branch with the same shape (not sure in this case if it will make sense) – Aray Karjauv May 08 '23 at 16:06
  • can you provide me an example/code? – user1631306 May 08 '23 at 21:37

0 Answers0