Harel Lidar

Paper summary · Oct 2026

Reward Respecting Subtasks for Model-Based RL

Summary of Reward Respecting Subtasks for Model-Based RL · paper ↗

Provenance

Claim

The authors note that to achieve the ambitious goal of AI RL must be able to reason abstractly in both state and time. They claim that state reasoning has made progress but temporal abstraction lags far behind, largely due to the immense size of possible options. They flag the fact that previous proposals for option discovery do not take how the options will be used in planing into account as a severe limitation. To alleviate this, they propose reward-respecting subtasks, these come with strong contraints on the space of options and thereby provide a partial solution to the problem of option discovery. Further, they can be adapted to work natively with the “standard” algorithms.

The Challenge of Temporal Abstraction

The issue of abstraction is common in AI in general and RL in particular, the agent sensors work with low-level data such as pixels/tokens while the reward is given based on performance on much higher-level objects such as correct solution/navigation etc. Abstraction in planning in particular relate to how an AI model translates the transition dynamics it has learned into improvements in the policy/value function and corresponding behavior. The authors suggest options as the way if formulating temporal abstractions. One poses subsidiary tasks that should theoretically help with solving the problem. Solutions to these tasks produce an option, the consequences of this yield a model. The model is then used for planning (SubTask, Option, Model, Plan == STOMP). In the current work, the authors introduce the notion of reward respecting subtasks. That is, rewards that aren’t decoupled from the main task reward as is sometimes the case, and thereby avoiding situations in which solving the subtask is detrimental to solving the main task.

Reward Respecting Subtasks

GVF

The first step of the STOMP progression is subtask specification.

Option Learning

Now that we have specific subtasks, the next stage of the STOMP progression is specifying off-policy learning algorithms to approximate optimal VFs and options. An option here is a policy together with a stopping rule. The authors introduce a general TD error and an update procedure called UpdateWeights&Traces (UWT), so one template can describe learning values, policies, options, and the option models in the next STOMP step.

Model Learning

Moving on to the third stage of the STOMP progression, the authors learn models of the environment’s action and option transitions. An option is a policy plus a stopping function; an action is a special case whose policy always selects that action and whose stopping function always stops.

Planning with Options

The final step in STOMP uses the action and option models to improve the value function through approximate value iteration.

Questions

Anki