15

I just learned about GAN and I'm a little bit confused about the naming of Latent Vector.

  • First, In my understanding, a definition of a latent variable is a random variable that can't be measured directly (we needs some calculation from other variables to get its value). For example, knowledge is a latent variable. Is it correct?

  • And then, in GAN, a latent vector $z$ is a random variable which is an input of the generator network. I read in some tutorials, it's generated using only a simple random function:

    z = np.random.uniform(-1, 1, size=(batch_size, z_size))

then how are the two things related? why don't we use the term "a vector with random values between -1 and 1" when referring $z$ (generator's input) in GAN?

malioboro
  • 2,729
  • 3
  • 20
  • 46
  • 2
    I will answer here your specific question (because you are asking too many questions in one single post and because I don't want to clutter my answer below). `z = np.random.uniform(-1, 1, size=(batch_size, z_size))` is a sampling operation. You're sampling $z$ from a uniform distribution. In the context e.g. of VAEs, a latent vector is sampled from some distribution. This is a "latent" distribution because this distribution outputs a compact (and hidden) representation of the inputs (e.g. images). This latent distribution is trained to learn such compact representation. – nbro May 24 '19 at 20:58
  • I am actually not very well familiar with GANs, but the concrete sampled latent vector $z$ is not a random variable. Mathematically, a hidden variable can be a random variable. The concept of a random variable is more related to the concept of a distribution than to the concept of a vector. Each random variable has an associated distribution. A vector is a sample from a distribution (defined over vectors). – nbro May 24 '19 at 21:05
  • @nbro "a hidden variable can be a random variable", based on your answer, is that means "a latent variable can be a random variable"? – malioboro May 25 '19 at 15:59
  • @nbro so, do you think, can I call $z$ as a vector with random value instead of latent vector? – malioboro May 25 '19 at 16:00

2 Answers2

10

It is called a Latent variable because you cannot access it during train time (which means manipulate it), In a normal Feed Forward NN you cannot manipulate the values output by hidden layers. Similarly the case here.

The term originally came from RBM's (they used term hidden variables). The interpretation of hidden variables in the context of RBM was that these hidden nodes helped to model the interaction between 2 input features (if both activate together, then the hidden unit will also activate). This principle can be traced to Hebb's rule which states "Neurons that fire together, wire together." Thus RBM's were used to find representation of models, in a space (generally lower dimensional than than the original). This is the principal used in Auto Encoder's also. Thus as you can see we are explicitly, not modelling the interaction between 2 features, how the process is occurring is "hidden" from us.

So, the term latent basically can be attributed to the following ideas:

  • We map higher dimensional data to a lower dimensional data with no prior convictions of how the mapping will be done. The NN trains itself for the best configuration.
  • We cannot manipulate this lower dimensional data. Thus it is "hidden from us.
  • As we do not know what each dimension means, it is "hidden" from us.
  • ohh... I see.. your answer and @nbro's are quite clear! it's good to know other examples that give a clear similarity in their meaning of latent variable.. Thank you! But, what do you think why $z$ in GAN is not just called "a vector with random values"? – malioboro May 25 '19 at 15:52
  • 1
    @malioboro You can call $z$ a vector whose values have be uniformly sampled or a vector with random values. However, in that specific case, it is likely that vector of "random values" represents/implements a latent variable. – nbro May 25 '19 at 16:07
  • @malioboro I am not familiar with GANs but in general when an encoder decoder unsupervised learning is present it is generally derived from idea of RBM..I would highly suggest you to check out chap 16 structured pbblity model in deep learning book by Goodfellow..It is freely available. –  May 25 '19 at 16:31
  • So latent... Surprisingly there are at least 2-3 ways to interpret why it is latent, but a wise one will see that all interpretation are actually the same. –  May 25 '19 at 16:37
5

Latent is a synonym for hidden.

Why is it called a hidden (or latent) variable? For example, suppose that you observe the behaviour of a person or animal. You can only observe the behaviour. You cannot observe the internal state (e.g. the mood) of this person or animal. The mood is a hidden variable because it cannot be observed directly (but only indirectly through its consequences).

A good example of statistical model that is highly based on the notion of latent variables is the hidden Markov model (HMM). If you understand the HMM, you will understand the concept of a hidden variable.

nbro
  • 39,006
  • 12
  • 98
  • 176