5

I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter 5.1.2 The Performance Measure, P, the authors say the following:

The choice of performance measure may seem straightforward and objective, but it is often difficult to choose a performance measure that corresponds well to the desired behavior of the system.

In some cases, this is because it is difficult to decide what should be measured. For example, when performing a transcription task, should we measure the accuracy of the system at transcribing entire sequences, or should we use a more fine-grained performance measure that gives partial credit for getting some elements of the sequence correct? When performing a regression task, should we penalize the system more if it frequently makes medium-sized mistakes or if it rarely makes very large mistakes? These kinds of design choices depend on the application.

In other cases, we know what quantity we would ideally like to measure, but measuring it is impractical. For example, this arises frequently in the context of density estimation. Many of the best probabilistic models represent probability distributions only implicitly. Computing the actual probability value assigned to a specific point in space in many such models is intractable. In these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion.

It is this part that interests me:

Many of the best probabilistic models represent probability distributions only implicitly.

I don't have the experience to understand what this means (what does it mean to represent distributions "implicitly"?). I would greatly appreciate it if people would please take the time to elaborate upon this.

Community
  • 1
The Pointer
  • 527
  • 3
  • 17

1 Answers1

2

The probabilistic models that represent distributions implicitly are, for example, the GANs. (Goodfellow is one of the authors of the original GAN model).

In the paper Variational Inference using Implicit Distributions (2017), the authors write

Implicit distributions are probability models whose probability density function may be intractable, but there is a way to

  1. sample from them exactly and/or calculate and approximate expectations under them, and

  2. calculate or estimate gradients of such expectations with respect to model parameters.

Implicit models have been successfully applied to generative modelling in generative adversarial networks (GANs)

A popular example of implicit models are stochastic generative networks: samples from a simple distribution - such as uniform or Gaussian - are transformed nonlinearly and non-invertably by a deep neural network

They also provide a table (table 1) that shows some probabilistic models that use an implicit distribution. Here $I$ denotes inference only. VI stands for variational inference.

enter image description here

See this blog post Variational Inference using Implicit Models, Part I: Bayesian Logistic Regression. See also the paper Learning in Implicit Generative Models (2017).

Community
  • 1
nbro
  • 39,006
  • 12
  • 98
  • 176
  • Thanks for the answer. Do you mind briefly clarifying what is meant by "are transformed nonlinearly and non-invertably by a deep neural network" for a machine learning novice such as myself? I was actually recently studying exercise 1.4. of *Pattern Recognition and Machine Learning* by Christopher Bishop, which was an exercise on the linear/non-linear transformation of a density function, so I'm curious about this. – The Pointer Feb 28 '20 at 18:41
  • 1
    @ThePointer Neural networks **non**-linearly transform their inputs because neural networks (usually) use **non**-linear activation functions (i.e. functions that are non-linear, such as the **sigmoid** function). – nbro Feb 28 '20 at 18:42
  • Wait, neural networks *linearly* transform their inputs because they use *non-linear* activation functions? Is that correct is there a typo here? – The Pointer Feb 28 '20 at 18:43
  • 1
    @ThePointer It was a typo. – nbro Feb 28 '20 at 18:44
  • 1
    Ok. Thank you for taking the time to post this answer. – The Pointer Feb 28 '20 at 18:44
  • I think your answer is correct but you are focusing too much on GANs. –  Feb 29 '20 at 03:48
  • @DuttaA Well, I'm listing other models/algorithms that make use of implicit distributions (table 1 of the first paper), so I don't think I am focusing on the GAN. I just mentioned the GAN because Goodfellow was probably referring to it. To be honest, I wasn't aware of this terminology before this question came up. In fact, if you see the edit history of this answer, I initially said that the VAE also represented distributions implicitly, while this isn't apparently the case according to some papers, because I thought they were referring to the variational distribution. – nbro Feb 29 '20 at 03:55