For questions about the concept of independent and identically distributed (often shortened as i.i.d., iid or IID) random variables (or samples) and assumption, which is often made in statistics and machine learning.
Questions tagged [iid]
12 questions
14
votes
3 answers
Why exactly do neural networks require i.i.d. data?
In reinforcement learning, successive states (actions and rewards) can be correlated. An experience replay buffer was used, in the DQN architecture, to avoid training the neural network (NN), which represents the $Q$ function, with correlated (or…
nbro
- 39,006
- 12
- 98
- 176
2
votes
1 answer
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$p_1(X_1=1, X_2 = 2) = \dfrac{1}{4}$$
$$p_1(X_1=2,…
hanugm
- 3,571
- 3
- 18
- 50
2
votes
1 answer
How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.?
How would the performance of federated learning (FL) compare to the performance of centralized machine learning (ML), when the data is independent and identically distributed (i.i.d.)?
Moreover, what is the difference in the performance of FL when…
Jared
- 35
- 5
2
votes
1 answer
Are training sequences for LMs sampled in an IID fashion?
If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 chunks of 10 tokens each (token 1-11, 2-12, and so…
Opt
- 121
- 2
2
votes
1 answer
If the i.i.d. assumption holds, shouldn't the training and validation trends be exactly the same?
If the i.i.d. (independent and identically distributed) assumption holds for a training-validation set pair, shouldn't their loss trends be exactly the same, since every batch from the validation set is equivalent to having a batch from the training…
ashenoy
- 1,409
- 4
- 18
1
vote
1 answer
Do adversarial samples violate the i.i.d. assumption?
I am trying to understand why adversarial attacks work in theory.
I have read, that the image is perturbed by a special perturbation $X_{adv}=X_1+p$, but i can't find any reference on that perturbation being an i.i.d. violation by itself.
If the…
Jan
- 351
- 3
- 13
1
vote
0 answers
Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?
In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch of unit transitions to train neural networks as…
hanugm
- 3,571
- 3
- 18
- 50
1
vote
2 answers
How can we "draw i.i.d" from any probability distribution?
Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges
Supervised machine learning, in its simplest formalisation, considers
a set of $N$…
hanugm
- 3,571
- 3
- 18
- 50
1
vote
1 answer
What are the iid random variables for a dataset in the GAN framework?
I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.
The answer tells that it is due to the law of large numbers which is based on the assumption that random variables are independent and…
hanugm
- 3,571
- 3
- 18
- 50
0
votes
0 answers
Does it make sense to compare images (samples) with words (features)?
Consider the following paragraphs from the introduction of the chapter named Recurrent Neural Networks from the textbook titled Dive into Deep Learning
So far we encountered two types of data: tabular data and image data.
For the latter we designed…
hanugm
- 3,571
- 3
- 18
- 50
0
votes
0 answers
What are the different possible usages of the word "i.i.d" in machine learning?
The acronym "iid" stands for "independent and identically distributed". It is a property of a sequence of random variables. You can read here for more details. This question is just about the usage of the word "iid" in contemporary machine learning…
hanugm
- 3,571
- 3
- 18
- 50
0
votes
2 answers
Is knowing underlying probability distribution mandatory for deciding iid property of random variables?
Consider the following information regarding iid random variables
The acronym IID stands for "Independent and Identically Distributed".
A sequence of random variables (or random vectors) is IID if and only
if the following two conditions are…
hanugm
- 3,571
- 3
- 18
- 50