*********************************************************** Scripted Simulation Controller and the Definition Meta-Loop *********************************************************** This document describes the *scripted simulation controller* and the definition-proposal capability it provides: a **meta agent** can make a definition decision before an environment is created, followed by an ordinary **operating agent** that acts inside the resulting environment. It introduces two cooperating capabilities: * the **scripted simulation controller** (:class:`~palaestrai.simulation.ScriptedSimulationController`), and * the **definition interface**, in which the environment itself proposes its meta-loop interface from its scenario definition. .. important:: The current version does **not** provide a repeated meta (learning) loop that adapts the scenario across episodes. A non-empty ``episode_start`` selects a constrained, one-shot definition step: the meta agent acts once per ``SimulationStartRequest``, and the validated candidate is used to start the operating environment. In-place episode resets do not rerun that decision. Repeated meta-agent decisions and scenario adaptation across episodes would require additional orchestration. This page describes the framework generically, without committing to any particular environment. The concrete MIDAS example — the inline MIDAS scenario provider, automatic numeric discovery, and optional selector overrides — is documented separately in :doc:`inline-midas-scenarios`. This allows a run-file-configured agent to propose an initial change to an *environment definition* (for example, MIDAS scenario parameters), without writing Python outside of a run file. It should not be confused with a repeated meta-learning loop that observes outcomes and adapts the scenario over multiple episodes; that loop is not implemented in the current version. Motivation ========== In the :doc:`classic flow `, an agent acts inside an environment whose configuration is fixed when the run starts. The brain-muscle loop is hard-coded: the agent observes sensors, proposes actuators, and the environment is stepped. Changing the environment itself — the grid topology, load levels, or generator setpoints of a MIDAS scenario — required a separate, external process. The scripted definition step turns *"how should the environment be configured?"* into an initial decision for an agent. A dedicated **meta agent** inspects the environment *definition* and proposes a *definition patch*; the resulting definition is then used to create the *operating* environment. The operating agent runs the ordinary act/step loop against that environment. Although the operating return is handed to the meta agent at the terminal update, the current version does not run another definition decision on the next in-place-reset episode. The Scripted Simulation Controller ================================== :class:`~palaestrai.simulation.ScriptedSimulationController` replaces the hard-coded brain-muscle loop with a declarative **lifecycle script** supplied through the run file. The script is a mapping with three optional phases: ``episode_start`` Pre-creation phase selected at the start-request seam. In the constrained definition profile, this phase names the meta agent for the one-shot decision; in-place resets do not rerun that decision. ``loop`` Ordered, worker-local turn groups that execute the ordinary *act -> step* cycle. It is required and must be non-empty when a script is provided. This phase hosts the operating agent in a meta-loop run. ``episode_end`` Turn groups executed after the episode. In the constrained meta-loop profile it must be empty; otherwise it is a no-op. Each turn group is a mapping describing which agents act on which environments:: script: episode_start: - agents: [meta_agent] environments: [midas] loop: - agents: [operating_agent] environments: [midas] episode_end: [] The controller is configured like any other simulation controller; its ``params`` block carries the script:: simulation: name: palaestrai.simulation:ScriptedSimulationController conditions: - name: palaestrai.simulation:VanillaSimControllerTerminationCondition params: {} params: script: { ... } The meta-loop interface is **not** configured on the controller. It is proposed by the environment itself from its scenario definition (see `The Environment-Proposed Interface`_). To support the free-form ``simulation.params`` mapping, the experiment run schema exposes ``simulation_params`` in ``palaestrai.experiment.run_schema.yaml``. The script structure is validated at **construction time**, before any worker or environment is started. A script with an empty ``episode_start`` keeps the generic scripted behaviour (turn groups only in ``loop``); a non-empty ``episode_start`` selects the meta-loop profile described next. The Definition Meta-Loop ======================== Profile and constraints ----------------------- A scripted run whose ``episode_start`` is non-empty selects the current one-shot *definition-decision* profile (sometimes called the definition meta-loop). It has a constrained shape: * exactly one agent declared in ``episode_start`` (the **meta agent**), * exactly one agent declared in ``loop`` (the **operating agent**), * the two agent names must be distinct, * ``episode_end`` must be empty, and * exactly one environment must propose a non-empty definition interface. Membership in ``episode_start`` selects the *definition-derived interface*; membership in ``loop`` selects the real *environment interface*. Both agents are ordinary agent declarations (brain, muscle, objective, sensors, actuators); the meta agent is *not* a new agent class. No bootstrap or synthetic environment is created for it. Lifecycle --------- The meta turn runs once per ``SimulationStartRequest``, before the operating environment world exists. In the normal in-place-reset lifecycle, that means one initial definition decision, not a new meta decision for each episode: .. figure:: _static/scripted-controller-lifecycle-sequence.svg :alt: UML sequence of the scripted meta-loop lifecycle and terminal handoff UML sequence diagram of the constrained meta-loop profile. Dashed vertical lines are lifelines; dashed horizontal arrows are responses. The loop frame repeats operating turns, not definition turns. Agent messages pass through their conductors and rollout workers; those hops are condensed here. :download:`Graphviz source ` 1. The :class:`~palaestrai.experiment.RunGovernor` supplies a deep-copied mapping of the current phase's environment configurations on the ``SimulationStartRequest`` (keyed by logical environment UID). From this, the controller derives the *canonical definition*. 2. The controller loads the configured environment *without building a world* and calls ``environment.propose_definition()`` to obtain the definition interface (selectors, scenario, and descriptors). 3. The returned descriptors are converted into concrete :class:`~palaestrai.agent.SensorInformation` and :class:`~palaestrai.agent.ActuatorInformation` values (see :class:`~palaestrai.simulation.DefinitionAgentInterface`), and ``provider.vectorize(definition, None)`` yields the full ordered initial readings for the declared interface. 4. The controller sends one ordinary ``AgentSetupRequest`` for the meta agent carrying those injected descriptors. There is no real environment static model. 5. The meta agent acts **exactly once** using the vectorized readings. It never calls ``step()`` and no environment setpoints are emitted. 6. The returned definition actuator values are mapped by UID and fed to ``provider.propose_patch(definition, actions)`` to obtain a declarative patch, then ``provider.validate(definition, patch)``. The resulting :class:`~palaestrai.simulation.ValidatedCandidate` ``.definition`` is the only definition used to start the operating environment. If the meta agent produces no patch, an empty action map is still validated into a candidate. 7. The controller derives a request-scoped environment-configuration override by replacing only the affected inline scenario subtree with the validated candidate's matching subtree, and delegates to the base start handler. The override travels on ``EnvironmentSetupRequest`` as ``environment_configuration_overrides``. 8. ``EnvironmentConductor`` loads the environment from the matching override (a deep copy) instead of its stored configuration; it never writes the override back. After the environment publishes its real descriptors, the ordinary setup path creates only the ``loop`` agent, which then runs the repeated act/step loop exactly as in a vanilla run. The protocol fields are additive: without a meta turn, the base controller sends empty overrides and the conductor uses its stored configuration. .. important:: An in-place ``_restart()`` resets the existing world and agents, clears override/reward state, and starts the operating simulation again. It does **not** call ``_run_definition_meta_turn`` or recreate the world. A new ``SimulationStartRequest`` is needed for another definition turn. The current version does not implement a new meta decision on every in-place episode reset. The fixtures therefore use ``episodes: 1``. How the meta turn is executed ----------------------------- On the :class:`~palaestrai.experiment.RunGovernor`'s ``SimulationStartRequest``, :meth:`~palaestrai.simulation.ScriptedSimulationController._handle_simulation_start_request` detects the meta profile. It does **not** run ``episode_start`` for a meta-loop run; instead it seeds the request-scoped context (run governor UID, experiment run id, instance id, phase, and episode) and calls :meth:`~palaestrai.simulation.ScriptedSimulationController._run_definition_meta_turn` with the request's ``environment_configurations``, before delegating to the base handler which performs the normal environment/agent setup and starts ``simulate()``. .. figure:: _static/scripted-controller-meta-turn-sequence.svg :alt: UML sequence of config-only proposal, agent action and validation UML sequence of ``_run_definition_meta_turn``. No world construction or environment ``step()`` occurs in this sequence. The setup/action request paths use the ordinary agent protocol. :download:`Graphviz source ` ``_run_definition_meta_turn`` then, in order: 1. resets ``_environment_configuration_overrides`` to ``{}``; 2. derives the canonical definition via :meth:`_definition_for_meta_loop`; 3. loads the definition environment *config-only* via :meth:`_load_definition_environment`, which calls ``environment.propose_definition()`` and returns the environment UID together with a :class:`~palaestrai.simulation.ProposedDefinition`; 4. verifies the proposal is for the expected environment; 5. constructs the environment-agnostic :class:`~palaestrai.simulation.DefinitionProvider` from the proposal; 6. validates the baseline with ``provider.validate({environment: scenario})``; 7. materialises the baseline definition; 8. obtains the ordered :class:`~palaestrai.simulation.DefinitionInterface` (``provider.describe``); 9. obtains the full initial :class:`~palaestrai.simulation.DefinitionVector` (``provider.vectorize(definition, None)``); 10. builds the concrete agent interface with :func:`~palaestrai.simulation.build_definition_agent_interface`; 11. resolves the meta agent name from :meth:`_meta_loop_agent_names`; 12. sets the meta agent up with the definition interface via :meth:`_setup_definition_agent`; 13. temporarily swaps the controller's current sensor readings for the definition readings and calls ``act([meta_agent])`` **exactly once**, restoring the prior readings afterwards (the meta agent never steps); 14. maps the returned actions to definition actuator UIDs (``definition_actions_from_response``); 15. proposes a declarative patch (``provider.propose_patch``); 16. validates it into a :class:`~palaestrai.simulation.ValidatedCandidate` (``provider.validate``); 17. derives the request-scoped override from the candidate definition and stores it in ``_environment_configuration_overrides``. Terminal reward handoff ----------------------- The meta agent is a *boundary* actor: it makes one definition decision per start request before the environment exists. The controller hands the operating agent's episode-level outcome to the meta agent as a terminal reward/update, but this does not cause another definition decision or scenario change on an in-place reset. The current version therefore does not implement the repeated meta-learning/adaptation loop. Such a loop would require additional orchestration to carry a candidate forward and schedule another meta decision. The current reward handoff works as follows: * while running the ``loop`` turn groups, :meth:`_execute_turn_groups` accumulates the operating agent's scalar rewards into ``_meta_loop_reward_values`` via :meth:`_accumulate_meta_loop_step_rewards` (non-scalar or non-finite values are rejected by :meth:`_coerce_meta_loop_reward_value`); * at the episode barrier, :meth:`flow_control` calls :meth:`_set_meta_loop_terminal_reward`, which writes the summed return as a single reward for the meta agent with UID ``.return`` (an unbounded ``Box``), so the meta muscle observes that return on the terminal ``done`` update. This terminal update is not a scheduling hook for another meta action; * :meth:`_restart` clears ``_meta_loop_reward_values`` (and pops the synthetic meta reward) before delegating to the base restart, so the next episode starts from an empty accumulator. Enforcement ----------- * Invalid meta-loop structure fails during controller construction (before workers and environments are started). * Missing definition data, invalid descriptor/vector conversion, meta-agent setup failure, invalid actions, an invalid patch hash/revision, or a failed candidate validation all fail **before** the operating environment exists. * A patch that would change the definition interface is rejected by the provider's fingerprint check, guaranteeing the meta agent cannot silently mutate the interface it is learning over. * A normal environment setup failure after a valid candidate follows the existing controller error handling. The Environment-Proposed Interface ================================== Each :class:`~palaestrai.environment.Environment` implements :meth:`~palaestrai.environment.Environment.propose_definition`, returning a :class:`~palaestrai.simulation.ProposedDefinition` for its own scenario configuration *without building a world*. The controller loads the configured environment config-only and asks it to propose the interface. The base ``Environment.propose_definition()`` returns an empty proposal; participating environments override it. For Mosaik, ``propose_mosaik_definition`` reads the inline scenario directly, without calling the MIDAS descriptor. ``derive_selectors`` automatically visits **every finite Python int or float leaf** inside the scenario, descending into mappings and lists (including nested lists). Booleans, strings, nulls, NaN, and infinities are not selectors; integer mapping keys and list indices locate values but are not values to expose themselves. Discovery is not restricted to ``constant_*`` names. Bounds are derived from the current value (``v > 0`` gives ``Box[0, 2v]``, ``v < 0`` gives ``Box[2v, 0]``, ``v == 0`` gives ``Box[0, 1]``) and normalisation ``{center: v, scale: max(abs(v), 1)}``. These are continuous scalar ``Box`` spaces even for initial integer values; actions are not automatically rounded to integers. For structural or timing parameters that must remain integral or fixed, filter them out or explicitly declare a suitable discrete manifest selector. An optional ``definition`` block in the environment's inner ``params`` (``include``, ``exclude``, ``selector_manifest``) narrows or overrides discovery. The regular expressions match either the leaf key or its full RFC 6901 path; exclusion wins. For the old constant-only behaviour, use ``include: ['^constant_.*$']``. A manifest replaces the automatically discovered selector at the **same pointer**, allowing a custom alias, bounds, or value map. Every selector is exposed as both a sensor and an actuator with UID ``.definition.``. Unique named mapping leaves keep short aliases (for example, ``constant_load_p_mw``). Repeated leaf names and numeric leaf tokens, including list indices, use the full escaped RFC 6901 pointer as their alias. Thus ``end`` under two mappings becomes two distinct path aliases; ``~`` and ``/`` inside a mapping key are escaped as ``~0`` and ``~1``. Declaration order is preserved. Empty mapping keys also use their full pointer. Ambiguous mappings whose keys resolve to the same pointer (for example, ``3`` and ``"3"`` together) are rejected instead of silently hiding a value. Paths remain stable for the same scenario structure; reordering a list or adding a duplicate named leaf can change its aliases, so persist the scenario and proposal together. Nested numeric exposure and usage --------------------------------- The fixture ``tests/fixtures/Train_meta-loop_simple_change.yml`` mirrors the MIDAS training example. No selector manifest is needed for these values:: scenario: comdata_params: midasmv: active_mapping: 3: [[Hospital, 0.427]] 4: [[LargeHotel, 0.913]] der_params: midasmv: mapping: 6: [[PV, 2.2]] The automatically generated sensor **and** actuator UIDs are: * ``midas.definition./comdata_params/midasmv/active_mapping/3/0/1`` * ``midas.definition./comdata_params/midasmv/active_mapping/4/0/1`` * ``midas.definition./der_params/midasmv/mapping/6/0/1`` Their initial readings are ``0.427``, ``0.913``, and ``2.2``; their bounds are ``[0, 0.854]``, ``[0, 1.826]``, and ``[0, 4.4]``. A definition action of ``0.5``, ``1.0``, and ``2.4`` at these UIDs becomes a patch at precisely those three nested paths. The validated candidate supplies the new load and PV scales to environment setup, without changing the bus keys, profile names, or original configuration. The initial readings are raw values; the fixed normalisation policy is metadata, not an online rescaling of the readings. The fixture uses ``definition.include`` to focus training on these three discovered values while leaving timing/topology parameters fixed. Without that filter, numeric leaves such as ``step_size``, ``end``, and ``buffer_size`` are also exposed. The controller injects the full proposed interface for the meta agent; listing fewer UIDs in its agent declaration does not narrow discovery. See :doc:`inline-midas-scenarios` for the run-file wiring and patch example. :class:`~palaestrai.simulation.DefinitionProvider` is the environment-agnostic implementation of the definition contract. It is constructed from the environment's :class:`~palaestrai.simulation.ProposedDefinition`; it never imports an environment package, configures a scenario, or creates a world. The contract has five methods: ``describe(definition)`` Returns a :class:`~palaestrai.simulation.DefinitionInterface` — the ordered sensor and actuator descriptors derived from the definition. ``vectorize(definition, selectors=None)`` Returns a :class:`~palaestrai.simulation.DefinitionVector` of scalar values. With ``selectors=None`` it returns the complete provider-declared interface in order; sensor descriptors must align positionally with actuator descriptors. ``propose_patch(definition, actions)`` Translates actuator values into a :class:`~palaestrai.simulation.DeclarativePatch` — a copy-safe, versioned delta that describes a definition change without carrying runtime objects. ``validate(definition, patch=None)`` Validates the definition (optionally with a patch applied) and returns a :class:`~palaestrai.simulation.ValidatedCandidate` carrying the validated definition, a descriptor fingerprint, a revision, and provenance. ``interface_fingerprint(interface)`` Computes a stable :class:`~palaestrai.simulation.DescriptorFingerprint` for an interface, used to reject interface-changing patches. The supporting value objects — :class:`~palaestrai.simulation.SensorDescriptor`, :class:`~palaestrai.simulation.ActuatorDescriptor`, :class:`~palaestrai.simulation.DefinitionVector`, :class:`~palaestrai.simulation.DeclarativePatch`, :class:`~palaestrai.simulation.ValidatedCandidate`, and :class:`~palaestrai.simulation.DescriptorFingerprint` — are all immutable frozen dataclasses wrapped in recursively immutable JSON-friendly containers. Definition-to-agent bridge -------------------------- :class:`~palaestrai.simulation.DefinitionAgentInterface` and its factory :func:`~palaestrai.simulation.build_definition_agent_interface` translate the provider's serialized descriptors (``space`` mappings of type ``Box`` or ``Discrete``) and vectors into the concrete :class:`~palaestrai.agent.SensorInformation` / :class:`~palaestrai.agent.ActuatorInformation` pairs the standard agent protocol already understands. Sensor and actuator UIDs must be identical and ordered; the definition vector must align with the sensor descriptors. Versioned scenario contracts ---------------------------- The meta-loop also introduces an immutable, versioned model for *scenario* definitions under :mod:`palaestrai.simulation`: * :class:`~palaestrai.simulation.ScenarioState` — a versioned, immutable materialised environment definition, * :class:`~palaestrai.simulation.ScenarioPatchAction` and :class:`~palaestrai.simulation.ScenarioActionCatalogue` — a finite, ordered set of declarative patch actions mapping agent indices to stable patch IDs, * :class:`~palaestrai.simulation.ScenarioDecisionRecord` and :class:`~palaestrai.simulation.ScenarioTransition` — immutable evidence of a scenario planning decision, and * :class:`~palaestrai.simulation.ScenarioAgentInterface` — the agent boundary that projects a scenario state into sensors plus a single discrete catalogue action. These contracts separate *"which definition?"* from *"how does the environment step?"*. They are separate public contracts available in the current version, but are not the execution path used by ``_run_definition_meta_turn``: that method uses ``ProposedDefinition`` / ``DefinitionProvider`` and UID-valued numeric actions, not a ``ScenarioActionCatalogue`` or a scenario shield. Their presence, like the reusable generic scripted lifecycle hooks, does not provide the repeated meta-learning and across-episode scenario adaptation described above. The Meta-Objective: WeightedBoundaryObjective ============================================= :class:`~palaestrai.agent.WeightedBoundaryObjective` aggregates several configured reward components into one scalar utility for the meta agent: .. code-block:: text sum(weight * clip((value - center) / scale, clip_low, clip_high)) Each component is a ``ComponentSpec`` (``uid``, ``weight``, ``center``, ``scale``, ``clip_low``, ``clip_high``); components may also be given as mappings in run-file parameters. A condensed ``clip: [low, high]`` form is accepted. The objective deliberately applies no online normalisation. Diagrams ======== The meta turn, end to end: .. image:: _static/scripted-simulation-controller-meta-turn.svg :alt: Scripted meta-turn sequence The flow of a definition through the environment-proposed provider: .. image:: _static/definition-provider-dataflow.svg :alt: Definition-to-candidate data flow