Scripted Simulation Controller and the Definition Meta-Loop¶

This document describes the scripted simulation controller and the definition-proposal capability it provides: a meta agent can make a definition decision before an environment is created, followed by an ordinary operating agent that acts inside the resulting environment. It introduces two cooperating capabilities:

the scripted simulation controller (ScriptedSimulationController), and
the definition interface, in which the environment itself proposes its meta-loop interface from its scenario definition.

Important

The current version does not provide a repeated meta (learning) loop that adapts the scenario across episodes. A non-empty episode_start selects a constrained, one-shot definition step: the meta agent acts once per SimulationStartRequest, and the validated candidate is used to start the operating environment. In-place episode resets do not rerun that decision. Repeated meta-agent decisions and scenario adaptation across episodes would require additional orchestration.

This page describes the framework generically, without committing to any particular environment. The concrete MIDAS example — the inline MIDAS scenario provider, automatic numeric discovery, and optional selector overrides — is documented separately in Inline MIDAS Scenarios.

This allows a run-file-configured agent to propose an initial change to an environment definition (for example, MIDAS scenario parameters), without writing Python outside of a run file. It should not be confused with a repeated meta-learning loop that observes outcomes and adapts the scenario over multiple episodes; that loop is not implemented in the current version.

Motivation¶

In the classic flow, an agent acts inside an environment whose configuration is fixed when the run starts. The brain-muscle loop is hard-coded: the agent observes sensors, proposes actuators, and the environment is stepped. Changing the environment itself — the grid topology, load levels, or generator setpoints of a MIDAS scenario — required a separate, external process.

The scripted definition step turns “how should the environment be configured?” into an initial decision for an agent. A dedicated meta agent inspects the environment definition and proposes a definition patch; the resulting definition is then used to create the operating environment. The operating agent runs the ordinary act/step loop against that environment. Although the operating return is handed to the meta agent at the terminal update, the current version does not run another definition decision on the next in-place-reset episode.

The Scripted Simulation Controller¶

ScriptedSimulationController replaces the hard-coded brain-muscle loop with a declarative lifecycle script supplied through the run file. The script is a mapping with three optional phases:

episode_start: Pre-creation phase selected at the start-request seam. In the constrained definition profile, this phase names the meta agent for the one-shot decision; in-place resets do not rerun that decision.
loop: Ordered, worker-local turn groups that execute the ordinary act -> step cycle. It is required and must be non-empty when a script is provided. This phase hosts the operating agent in a meta-loop run.
episode_end: Turn groups executed after the episode. In the constrained meta-loop profile it must be empty; otherwise it is a no-op.

Each turn group is a mapping describing which agents act on which environments:

script:
  episode_start:
    - agents: [meta_agent]
      environments: [midas]
  loop:
    - agents: [operating_agent]
      environments: [midas]
  episode_end: []

The controller is configured like any other simulation controller; its params block carries the script:

simulation:
  name: palaestrai.simulation:ScriptedSimulationController
  conditions:
    - name: palaestrai.simulation:VanillaSimControllerTerminationCondition
      params: {}
  params:
    script: { ... }

The meta-loop interface is not configured on the controller. It is proposed by the environment itself from its scenario definition (see The Environment-Proposed Interface). To support the free-form simulation.params mapping, the experiment run schema exposes simulation_params in palaestrai.experiment.run_schema.yaml.

The script structure is validated at construction time, before any worker or environment is started. A script with an empty episode_start keeps the generic scripted behaviour (turn groups only in loop); a non-empty episode_start selects the meta-loop profile described next.

The Definition Meta-Loop¶

Profile and constraints¶

A scripted run whose episode_start is non-empty selects the current one-shot definition-decision profile (sometimes called the definition meta-loop). It has a constrained shape:

exactly one agent declared in episode_start (the meta agent),
exactly one agent declared in loop (the operating agent),
the two agent names must be distinct,
episode_end must be empty, and
exactly one environment must propose a non-empty definition interface.

Membership in episode_start selects the definition-derived interface; membership in loop selects the real environment interface. Both agents are ordinary agent declarations (brain, muscle, objective, sensors, actuators); the meta agent is not a new agent class. No bootstrap or synthetic environment is created for it.

Lifecycle¶

The meta turn runs once per SimulationStartRequest, before the operating environment world exists. In the normal in-place-reset lifecycle, that means one initial definition decision, not a new meta decision for each episode:

The RunGovernor supplies a deep-copied mapping of the current phase’s environment configurations on the SimulationStartRequest (keyed by logical environment UID). From this, the controller derives the canonical definition.
The controller loads the configured environment without building a world and calls environment.propose_definition() to obtain the definition interface (selectors, scenario, and descriptors).
The returned descriptors are converted into concrete SensorInformation and ActuatorInformation values (see DefinitionAgentInterface), and provider.vectorize(definition, None) yields the full ordered initial readings for the declared interface.
The controller sends one ordinary AgentSetupRequest for the meta agent carrying those injected descriptors. There is no real environment static model.
The meta agent acts exactly once using the vectorized readings. It never calls step() and no environment setpoints are emitted.
The returned definition actuator values are mapped by UID and fed to provider.propose_patch(definition, actions) to obtain a declarative patch, then provider.validate(definition, patch). The resulting ValidatedCandidate .definition is the only definition used to start the operating environment. If the meta agent produces no patch, an empty action map is still validated into a candidate.
The controller derives a request-scoped environment-configuration override by replacing only the affected inline scenario subtree with the validated candidate’s matching subtree, and delegates to the base start handler. The override travels on EnvironmentSetupRequest as environment_configuration_overrides.
EnvironmentConductor loads the environment from the matching override (a deep copy) instead of its stored configuration; it never writes the override back. After the environment publishes its real descriptors, the ordinary setup path creates only the loop agent, which then runs the repeated act/step loop exactly as in a vanilla run.

The protocol fields are additive: without a meta turn, the base controller sends empty overrides and the conductor uses its stored configuration.

Important

An in-place _restart() resets the existing world and agents, clears override/reward state, and starts the operating simulation again. It does not call _run_definition_meta_turn or recreate the world. A new SimulationStartRequest is needed for another definition turn. The current version does not implement a new meta decision on every in-place episode reset. The fixtures therefore use episodes: 1.

How the meta turn is executed¶

On the RunGovernor’s SimulationStartRequest, _handle_simulation_start_request() detects the meta profile. It does not run episode_start for a meta-loop run; instead it seeds the request-scoped context (run governor UID, experiment run id, instance id, phase, and episode) and calls _run_definition_meta_turn() with the request’s environment_configurations, before delegating to the base handler which performs the normal environment/agent setup and starts simulate().

UML sequence of config-only proposal, agent action and validation — UML sequence of `_run_definition_meta_turn`. No world construction or environment `step()` occurs in this sequence. The setup/action request paths use the ordinary agent protocol.¶

`Graphviz source`

_run_definition_meta_turn then, in order:

resets _environment_configuration_overrides to {};
derives the canonical definition via _definition_for_meta_loop();
loads the definition environment config-only via _load_definition_environment(), which calls environment.propose_definition() and returns the environment UID together with a ProposedDefinition;
verifies the proposal is for the expected environment;
constructs the environment-agnostic DefinitionProvider from the proposal;
validates the baseline with provider.validate({environment: scenario});
materialises the baseline definition;
obtains the ordered DefinitionInterface (provider.describe);
obtains the full initial DefinitionVector (provider.vectorize(definition, None));
builds the concrete agent interface with build_definition_agent_interface();
resolves the meta agent name from _meta_loop_agent_names();
sets the meta agent up with the definition interface via _setup_definition_agent();
temporarily swaps the controller’s current sensor readings for the definition readings and calls act([meta_agent]) exactly once, restoring the prior readings afterwards (the meta agent never steps);
maps the returned actions to definition actuator UIDs (definition_actions_from_response);
proposes a declarative patch (provider.propose_patch);
validates it into a ValidatedCandidate (provider.validate);
derives the request-scoped override from the candidate definition and stores it in _environment_configuration_overrides.

Terminal reward handoff¶

The meta agent is a boundary actor: it makes one definition decision per start request before the environment exists. The controller hands the operating agent’s episode-level outcome to the meta agent as a terminal reward/update, but this does not cause another definition decision or scenario change on an in-place reset. The current version therefore does not implement the repeated meta-learning/adaptation loop. Such a loop would require additional orchestration to carry a candidate forward and schedule another meta decision. The current reward handoff works as follows:

while running the loop turn groups, _execute_turn_groups() accumulates the operating agent’s scalar rewards into _meta_loop_reward_values via _accumulate_meta_loop_step_rewards() (non-scalar or non-finite values are rejected by _coerce_meta_loop_reward_value());
at the episode barrier, flow_control() calls _set_meta_loop_terminal_reward(), which writes the summed return as a single reward for the meta agent with UID <operating_agent>.return (an unbounded Box), so the meta muscle observes that return on the terminal done update. This terminal update is not a scheduling hook for another meta action;
_restart() clears _meta_loop_reward_values (and pops the synthetic meta reward) before delegating to the base restart, so the next episode starts from an empty accumulator.

Enforcement¶

Invalid meta-loop structure fails during controller construction (before workers and environments are started).
Missing definition data, invalid descriptor/vector conversion, meta-agent setup failure, invalid actions, an invalid patch hash/revision, or a failed candidate validation all fail before the operating environment exists.
A patch that would change the definition interface is rejected by the provider’s fingerprint check, guaranteeing the meta agent cannot silently mutate the interface it is learning over.
A normal environment setup failure after a valid candidate follows the existing controller error handling.

The Environment-Proposed Interface¶

Each Environment implements propose_definition(), returning a ProposedDefinition for its own scenario configuration without building a world. The controller loads the configured environment config-only and asks it to propose the interface. The base Environment.propose_definition() returns an empty proposal; participating environments override it. For Mosaik, propose_mosaik_definition reads the inline scenario directly, without calling the MIDAS descriptor.

derive_selectors automatically visits every finite Python int or float leaf inside the scenario, descending into mappings and lists (including nested lists). Booleans, strings, nulls, NaN, and infinities are not selectors; integer mapping keys and list indices locate values but are not values to expose themselves. Discovery is not restricted to constant_* names. Bounds are derived from the current value (v > 0 gives Box[0, 2v], v < 0 gives Box[2v, 0], v == 0 gives Box[0, 1]) and normalisation {center: v, scale: max(abs(v), 1)}. These are continuous scalar Box spaces even for initial integer values; actions are not automatically rounded to integers. For structural or timing parameters that must remain integral or fixed, filter them out or explicitly declare a suitable discrete manifest selector.

An optional definition block in the environment’s inner params (include, exclude, selector_manifest) narrows or overrides discovery. The regular expressions match either the leaf key or its full RFC 6901 path; exclusion wins. For the old constant-only behaviour, use include: ['^constant_.*$']. A manifest replaces the automatically discovered selector at the same pointer, allowing a custom alias, bounds, or value map. Every selector is exposed as both a sensor and an actuator with UID <environment-uid>.definition.<alias>.

Unique named mapping leaves keep short aliases (for example, constant_load_p_mw). Repeated leaf names and numeric leaf tokens, including list indices, use the full escaped RFC 6901 pointer as their alias. Thus end under two mappings becomes two distinct path aliases; ~ and / inside a mapping key are escaped as ~0 and ~1. Declaration order is preserved. Empty mapping keys also use their full pointer. Ambiguous mappings whose keys resolve to the same pointer (for example, 3 and "3" together) are rejected instead of silently hiding a value. Paths remain stable for the same scenario structure; reordering a list or adding a duplicate named leaf can change its aliases, so persist the scenario and proposal together.

Nested numeric exposure and usage¶

The fixture tests/fixtures/Train_meta-loop_simple_change.yml mirrors the MIDAS training example. No selector manifest is needed for these values:

scenario:
  comdata_params:
    midasmv:
      active_mapping:
        3: [[Hospital, 0.427]]
        4: [[LargeHotel, 0.913]]
  der_params:
    midasmv:
      mapping:
        6: [[PV, 2.2]]

The automatically generated sensor and actuator UIDs are:

midas.definition./comdata_params/midasmv/active_mapping/3/0/1
midas.definition./comdata_params/midasmv/active_mapping/4/0/1
midas.definition./der_params/midasmv/mapping/6/0/1

Their initial readings are 0.427, 0.913, and 2.2; their bounds are [0, 0.854], [0, 1.826], and [0, 4.4]. A definition action of 0.5, 1.0, and 2.4 at these UIDs becomes a patch at precisely those three nested paths. The validated candidate supplies the new load and PV scales to environment setup, without changing the bus keys, profile names, or original configuration. The initial readings are raw values; the fixed normalisation policy is metadata, not an online rescaling of the readings.

The fixture uses definition.include to focus training on these three discovered values while leaving timing/topology parameters fixed. Without that filter, numeric leaves such as step_size, end, and buffer_size are also exposed. The controller injects the full proposed interface for the meta agent; listing fewer UIDs in its agent declaration does not narrow discovery. See Inline MIDAS Scenarios for the run-file wiring and patch example.

DefinitionProvider is the environment-agnostic implementation of the definition contract. It is constructed from the environment’s ProposedDefinition; it never imports an environment package, configures a scenario, or creates a world.

The contract has five methods:

describe(definition): Returns a DefinitionInterface — the ordered sensor and actuator descriptors derived from the definition.
vectorize(definition, selectors=None): Returns a DefinitionVector of scalar values. With selectors=None it returns the complete provider-declared interface in order; sensor descriptors must align positionally with actuator descriptors.
propose_patch(definition, actions): Translates actuator values into a DeclarativePatch — a copy-safe, versioned delta that describes a definition change without carrying runtime objects.
validate(definition, patch=None): Validates the definition (optionally with a patch applied) and returns a ValidatedCandidate carrying the validated definition, a descriptor fingerprint, a revision, and provenance.
interface_fingerprint(interface): Computes a stable DescriptorFingerprint for an interface, used to reject interface-changing patches.

The supporting value objects — SensorDescriptor, ActuatorDescriptor, DefinitionVector, DeclarativePatch, ValidatedCandidate, and DescriptorFingerprint — are all immutable frozen dataclasses wrapped in recursively immutable JSON-friendly containers.

Definition-to-agent bridge¶

DefinitionAgentInterface and its factory build_definition_agent_interface() translate the provider’s serialized descriptors (space mappings of type Box or Discrete) and vectors into the concrete SensorInformation / ActuatorInformation pairs the standard agent protocol already understands. Sensor and actuator UIDs must be identical and ordered; the definition vector must align with the sensor descriptors.

Versioned scenario contracts¶

The meta-loop also introduces an immutable, versioned model for scenario definitions under palaestrai.simulation:

ScenarioState — a versioned, immutable materialised environment definition,
ScenarioPatchAction and ScenarioActionCatalogue — a finite, ordered set of declarative patch actions mapping agent indices to stable patch IDs,
ScenarioDecisionRecord and ScenarioTransition — immutable evidence of a scenario planning decision, and
ScenarioAgentInterface — the agent boundary that projects a scenario state into sensors plus a single discrete catalogue action.

These contracts separate “which definition?” from “how does the environment step?”. They are separate public contracts available in the current version, but are not the execution path used by _run_definition_meta_turn: that method uses ProposedDefinition / DefinitionProvider and UID-valued numeric actions, not a ScenarioActionCatalogue or a scenario shield. Their presence, like the reusable generic scripted lifecycle hooks, does not provide the repeated meta-learning and across-episode scenario adaptation described above.

The Meta-Objective: WeightedBoundaryObjective¶

WeightedBoundaryObjective aggregates several configured reward components into one scalar utility for the meta agent:

sum(weight * clip((value - center) / scale, clip_low, clip_high))

Each component is a ComponentSpec (uid, weight, center, scale, clip_low, clip_high); components may also be given as mappings in run-file parameters. A condensed clip: [low, high] form is accepted. The objective deliberately applies no online normalisation.

Diagrams¶

The meta turn, end to end:

The flow of a definition through the environment-proposed provider: