For questions about state spaces, in the context of reinforcement learning or other AI sub-fields.
Questions tagged [state-spaces]
47 questions
21
votes
2 answers
How to define states in reinforcement learning?
I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP.
What I don't understand is the process of defining the states of the MDP. In most examples…
Andy
- 313
- 1
- 2
- 6
8
votes
2 answers
What is the relation between the context in contextual bandits and the state in reinforcement learning?
Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)?
Specifically, in RL, we can use a function approximator (e.g. a neural network) to generalize to other states.…
Maxim Volgin
- 183
- 2
- 8
6
votes
3 answers
What is a high dimensional state in reinforcement learning?
In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology.
Suppose my state is a high dimensional vector of length $N$, where $N$ is a huge number. Let's say I solve this task using…
Siddhant Tandon
- 163
- 1
- 5
6
votes
3 answers
What exactly are partially observable environments?
I have trouble understanding the meaning of partially observable environments. Here's my doubt.
According to what I understand, the state of the environment is what precisely determines the next state and reward for any particular action taken. So,…
6
votes
1 answer
What are the state space and the state transition function in AI?
I'm studying for my AI final exam, and I'm stuck in the state space representation. I understand initial and goal states, but what I don't understand is the state space and state transition function. Can someone explain what are they with…
İsmail Uysal
- 63
- 1
- 4
6
votes
1 answer
What techniques are used to make MDP discrete state space manageable?
Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality.
Supposed my state has a few simple features:
Feeling: Happy/Neutral/Sad
Feeling: Hungry/Neither/Full
Food left:…
Brendan Hill
- 263
- 1
- 6
5
votes
1 answer
It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?
I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space is the only thing that really matters, in this…
FS93
- 145
- 6
4
votes
1 answer
How to approach a blackjack-like card game with the possibility of cards being counted?
Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.:
You're playing against a dealer with fixed rules.
You have one card deck which is played…
J. M. Arnold
- 41
- 2
4
votes
0 answers
How to update the observation probabilities in a POMDP?
How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model?
The POMDP relies on observation probabilities that match an observation to a state. This poses an issue as the probabilities are…
Pluxyy
- 85
- 3
3
votes
1 answer
How DFS may expand the same state many times via different paths in an acyclic state space?
I am reading the book titled Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig (4th edition) and came across this sentence about depth-first search (page 79, line 12):
For acyclic state spaces it may end up expanding the…
user153245
- 123
- 8
3
votes
1 answer
What is the difference between terminal state, nonterminal states and normal states?
In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following:
Each episode ends in a special state called the terminal state
But the authors also say:
the episodes can all be considered…
Daviiid
- 563
- 3
- 15
3
votes
1 answer
How can I compress the states of a reinforcement learning agent?
I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like convolutional neural networks will not work here.
Is…
Saeid Ghafouri
- 113
- 5
3
votes
2 answers
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
Problem description:
Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:
if current action == previous action, you get reward = $R(a,s)$
if…
FQT
- 33
- 6
3
votes
2 answers
What happens when the agent faces a state that never before encountered?
I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on the agent's action, the network status changes,…
krm76
- 155
- 7
3
votes
1 answer
What constitutes a large space state (in Q-learning)?
I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space?
I know that, according to multiple papers, tabular Q-learning is not suitable for these…
nickredsox
- 31
- 4