2

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in model-based RL such as in robotic manipulation to learn system dynamics, and can work effectively well. (Here, I mean in learning the model, not as an optimization method for the controller selecting the best actions).

In model-based RL, are there situations where a learning a linear model such as using a Lyapunov function would be better suited than using a neural network, or are the examples of problems framed to use linear models when addressing them using model-based RL?

mugoh
  • 531
  • 4
  • 20
  • This might be helpful: "Linear approximations of non-linear physical systems prove accurate and are well accepted in many fields. For the *comprehensive* study, see [Linear models of nonlinear systems](https://www.diva-portal.org/smash/get/diva2:21266/FULLTEXT01.pdf)" – mugoh Sep 26 '20 at 06:28

1 Answers1

1

This is just a case of supervised learning. You are trying to predict $s_{t+1}$ given $s_t$ and $a_t$, so the answer to your question depends on how complex your state dynamics are.

For example, if the state space is really complex, e.g. if your state space is an image and you want to predict the next image given the current image and an action, then linear methods are unlikely to work well.

David
  • 4,591
  • 1
  • 6
  • 25
  • Yes, state space will be key. [This work](https://arxiv.org/abs/2002.09093) uses a $32 * 32$ image on a linear model. Would that be termed as complex? – mugoh Sep 14 '20 at 15:14
  • I'm not sure, I'm not an image expert. In [this paper](https://arxiv.org/pdf/1912.01603) they learn state dynamics in a latent space -- the raw state space is an image -- and they use deep nets for this, so I would assume you need something more complex than a linear model to predict images. – David Sep 14 '20 at 15:46