1

I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.

Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?

Note that I would still have this problem with a regular autoencoder.

mhdadk
  • 233
  • 2
  • 7
  • I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though – Neil Slater Nov 10 '20 at 15:48
  • Yes you are correct. [Here](https://towardsdatascience.com/building-autoencoders-on-sparse-one-hot-encoded-data-53eefdfdbcc7) is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas? – mhdadk Nov 10 '20 at 16:23

0 Answers0