Harel Lidar

Article summary · Oct 2026

Environment Scaling for Agentic RL

Mostly someone else’s words. I keep it as reading, not as my own writing.

  1. Identifying source materials from which we can seed new environments. This can be:
    • code repos
    • tool documentation
    • a task set
    • a trajectory of past user interactions
  2. We take these source materials and translate them into a tuple of task and environment. This stage has a few common patterns:
    1. Skeleton to scenario: This is when we first create a skeleton of the environment, which can include state variables, tools, etc., and then inject or infer plausible tasks and scenarios to run on this environment, given the initial source and the skeleton we got.
    2. We have an environment-first inversion. That is instead of writing a task and then an environment, we take a working system that we sourced and we break it. We introduce bugs and ask the agent to fix them. Since we already know about this
  3. build and make it executable. We take the artifacts we produced in the previous step and we use an LLM with a repair loop until it works end to end. That is, we generate the code, try to build and run it, observe and monitor any errors, return them to the LLM, and ask it to iterate. At the end of this stage we have an environment that is fully executable
  4. Now that it’s executable the next stage is to verify, we check whether the task that we produce is satisfactory. We define satisfactory environments as tasks which we can verify and then, given that we can verify them, it makes no difference to us if they’re verifiable yet unsolvable. Once we know it’s verifiable we check whether it’s solvable as well as non-trivial. That is, two sides of the same coin.
  5. now that we have filtered the environments to those that are verifiable and solvable, we start collecting trajectories from strong teacher models. If we’re preparing a new model we can use the best state-of-the-art models that we currently have to bootstrap. We generate many candidate runs filtered to those that are successful and use them for an initial seed of SFT data. This also serves as the RL bootstrapping signal
    1. my own note here: I just got an idea for a research proposal. The common issue here is that state-of-the-art models are usually closed models from frontier labs but these typically hide the thinking traces from their models, which have been shown to be quite useful for training. What I’m now thinking is whether we can:
      1. see the initial trajectories from state-of-the-art models
      2. collect successful trajectories
      3. use the leading open-source models to fill in the trajectories between outputted blocks with plausible reasoning steps that led one from the first observation to the latter
  6. now we’re ready to train. We start with a short SFT warm-up stage where we train on the previously collected trajectories in order to obtain formatting prowess as well as a baseline of reasoning and execution capabilities to gain some form of signal. Then we move to the RL, where the model trains and gets rewarded based on the verifier
  7. A one-shot pipeline leaves signal on the table. After a round of training you know much more than you did at generation time: which skills the agent still fails, which tasks gave a clean learning signal, and which were a waste of compute. The evolve step turns that hindsight into the next batch of environmentsThree kinds of diagnostic signal are available, in increasing order of how hard they are to extract:
    • Inference-level signals — what the agent failed at. Per-skill failure clusters, and failure modes like looping on the same command, exhausting the turn budget, or terminating early. These are cheap and robust, and they drive coverage repair (generate more of the under-served skill) and failure-targeted generation.
    • Training-level signals — whether a task family is teaching anything. The fraction of tasks with all-zero or all-one reward, the per-task reward/advantage variance, entropy and KL trajectories. A family that produces no reward variance is producing no gradient (this is the difficulty-vs-trainability point of Difficulty ≠ trainability, used here as a control knob), and should be made easier, decomposed into subgoals, or dropped.
    • Transfer-level signals — whether the synthetic version is too easy. A gap between performance on synthesized development tasks and on a held-out benchmark flags skills whose synthetic rendition is a simplified caricature of the real thing.
  1. Robust verifiers, that is, automatic verifiers that give fine accurate descriptions and scores while being hard to hack.
  2. Generalization: improvement on the synthesized environments doesn’t always scale to production settings.
  3. Co-evolving the environments with the agents. As the agent becomes more capable the environments need to get progressively harder. Current systems usually rely on loops where we transition between some form of environment synthesis and further training but a better way of generating environments to fit the current agent is something to explore.