1

I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.

The answer tells that it is due to the law of large numbers which is based on the assumption that random variables are independent and identically distributed.

If I have a dataset of all possible $32 \times 32$ grayscale images. Then my sample space consists of $256^{32 \times 32}$ elements. Suppose I define 1024 random variables as

$$X_i = \text{ intensity of } i^{th} \text{ pixel for } 1 \le i \le 1024$$

Then it is clear that all the random variables are iid since

  1. $X_i \perp X_j$ for all $i, j$ such that $i \ne j$ and
  2. $p(X_i = k) = \dfrac{1}{256}$ for all $i$

But these properties do not hold if I take a dataset of (say flower) images since pixel intensities are not independent of each other and the intensity values are not uniformly distributed as well.

Then how can the law of large numbers be applicable for GAN as the dataset (sample space) does not cover all the possible elements? If I am wrong, then what is the sample space they are considering and what are the random variables they are using implicitly that leads to the satisfaction of iid condition and then the law of large numbers?

hanugm
  • 3,571
  • 3
  • 18
  • 50
  • GANs do not learn pixels directly, but a mapping between latent features (e.g., color, rotation, width, etc.) and an output image. – Aray Karjauv Aug 01 '21 at 14:52
  • @ArayKarjauv But the value function is saying $x∼p_{data}$ under the expectation, which then I understood $x$ as a random vector containing random variables which says the intensities of pixels. Is it wrong? – hanugm Aug 01 '21 at 15:07
  • 1
    $x$ is an image with abstract attributes, which a GAN aims to learn. Specifically, GAN tries to approximate a lower dimensional manifold that represents the real data. Of course, all modalities must be evenly represented in the dataset, but I think that the mean is rather related to the loss than to the iid of the random variables. – Aray Karjauv Aug 01 '21 at 15:48
  • @ArayKarjauv Your second comment is close to a full answer IMO – Neil Slater Aug 01 '21 at 18:54
  • @ArayKarjauv So, is [this](https://ai.stackexchange.com/a/23163/18758) answer wrong? – hanugm Aug 01 '21 at 21:39
  • 1
    I find this answer incomplete. It just discusses the expectation itself, as the author stated at the beginning. – Aray Karjauv Aug 01 '21 at 22:26

1 Answers1

1

Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The most common example is a coin toss: as you flip the coin, one outcome does not influence or predict the next one.

As for a dataset of flowers, we assume that the photos were made independently and all kinds of flowers are evenly represented, which in practice is not true. Each flower won't be represented in the same lighting condition and camera view.

Each pixel in an image is highly dependent on the neighboring pixels (and sometimes distant ones, e.g., when generating faces as they are usual symmetric) and the main goal of GAN is to discover this dependence by exploring a latent features manifold that represents abstract attributes of the real data, and to find a mapping between these features and output pixels. Of course, if some of the features are poorly represented, the network will not be able to generate them correctly. For instance, to increase the fidelity, there is the so-called truncation trick, where we sample the latent features vector $z$ from a truncated normal distribution, which ignores rare features. But on the other hand, it reduces diversity.

As for the loss function,

$$\nabla_{\theta_{d}} \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$

the mean is associated with the loss rather than the iid of the random variables and is used to obtain the average gradient of a batch. Here are related questions: one, tow.

hanugm
  • 3,571
  • 3
  • 18
  • 50
Aray Karjauv
  • 907
  • 8
  • 15