0

I was training an autoencoder for anomaly detection and I wish to extract the bottleneck features of the encoder for K-NN. The model architecture is as such:

class Autoencoder(torch.nn.Module):
   def __init__(self):
      super().__init__()

      self.encoder = torch.nn.Sequential(
         torch.nn.Linear(curr_col_count, 300),
         torch.nn.ReLU(),
         torch.nn.Linear(300, 150),
         torch.nn.ReLU(),
         torch.nn.Linear(150, 75),
         torch.nn.ReLU(),
         torch.nn.Linear(75, 50),
         # torch.nn.ReLU(),
         # torch.nn.Linear(125, 50)
      )

      self.decoder = torch.nn.Sequential(
         torch.nn.Linear(50, 75),
         torch.nn.ReLU(),
         torch.nn.Linear(75, 150),
         torch.nn.ReLU(),
         torch.nn.Linear(150, 300),
         torch.nn.ReLU(),
         # torch.nn.Linear(500, 1000),
         # torch.nn.ReLU(),
         torch.nn.Linear(300, curr_col_count),
         torch.nn.Tanh()
      )
      
   def forward(self, x):
      encoded = self.encoder(x)
      decoded = self.decoder(encoded)
      return decoded

and I am using the Pytorch feature extractor as such:

import torchvision as tv
import torchextractor as tx

model = Autoencoder()
model = tx.Extractor(
    model,
    ["encoder.6"]
)
loss_function = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 1e-4)

Before training the model, I retrieved the bottleneck features for each row of data and every row had different values. However, after training the model for just 1 epoch, I realized that every row had the same values for the bottleneck features. Why is this the case?

desertnaut
  • 1,005
  • 10
  • 19
Aengus
  • 1
  • Do you see the same thing after training a few more epochs? – lpounng Aug 29 '23 at 03:49
  • Yes, i see the same thing with 30 epochs – Aengus Aug 29 '23 at 05:07
  • 1
    Are the two snippets from the same file, so you are training and attempting some analysis of the extracted features in the same run? That should be fine, but you should stil show how you are extracting the features with the lines of code that perform that (currently you only show that you are using `torchextractor`). Also that would appear to be in two places (before and after training with only one of them failing), so please show both – Neil Slater Aug 29 '23 at 08:08
  • Well, first one needs to clarify what one means by "bottleneck features". What do you mean by that? I don't see anything in your code that prints or plots any features. Yes, I see `tx.Extractor`, but nothing is being done after that. Can you provide more info about your data you're training the model with? Dimensions, etc – nbro Aug 31 '23 at 11:38

0 Answers0