Portable phase checkpoints ========================== PalaestrAI checkpoints capable agents by default. A checkpoint is a complete, validated recovery point for one run instance and phase. PalaestrAI keeps only the **latest complete checkpoint** for that phase. Restarting is always explicit: a later run must ask to resume that latest checkpoint and name the original instance and phase. This page explains the user-facing behavior. The runtime configuration is summarized in :doc:`runtime-config`. Normal run behavior ------------------- No checkpoint configuration is needed for the normal case:: data_path: /srv/palaestrai/eaton-demo ``RuntimeConfig.data_path`` is resolved to an absolute directory. Checkpoint packages live below its ``checkpoints`` subdirectory; do not copy, edit, move, or select package files directly. A package is created only when all of the following are true: * checkpointing is enabled; * every participating Brain supports portable checkpoints; and * the run reaches a completed evaluation boundary. The boundary is quiescent: evaluation has finished and the agent contributes a consistent learning-state snapshot. It is not a mid-step or mid-episode save. Legacy Brains that do not provide the capability continue to run normally, but do not create a portable checkpoint. PalaestrAI writes a new package before replacing the latest pointer. If a run is interrupted during a replacement, the previously completed checkpoint remains the resume candidate. If no completed evaluation boundary has been reached, there is nothing to resume. Disable checkpointing --------------------- Set the single opt-out explicitly when a run must not create or resume portable checkpoints:: data_path: /srv/palaestrai/eaton-demo checkpoint: enabled: false Do not combine ``enabled: false`` with a resume request. Resume the latest checkpoint ---------------------------- Resume never happens automatically. To restart, use ``resume: latest`` and identify the exact historical run instance and phase that own the latest checkpoint:: data_path: /srv/palaestrai/eaton-demo checkpoint: enabled: true resume: latest resume_instance_uid: 4ed6c8bb-3d20-4ffb-b253-6c6ce0cb58a1 resume_phase: 0 ``resume_phase`` is the zero-based phase number. The selected package must match the run, instance, phase, configured agent contracts, and configuration. The new process performs setup first, restores the package before replay pretraining or online learning, and then continues from the completed evaluation boundary. PalaestrAI never scans other instances, chooses an older package, or resumes a partial package. Choose the same ``data_path`` that contains the original run's checkpoint packages and use the same Store. The Store is required for checkpoint evidence and is checked during resume. Verify a checkpoint or resume ----------------------------- For an evaluation-aware termination condition, the application log records an accepted deterministic evaluation with its ``evaluation_id``, ``checkpoint_id`` and ``checkpoint_hash``. A non-null checkpoint ID and hash identify the committed package associated with that evaluation. The Store contains a metadata-only ``CheckpointRef`` for each committed phase package. It contains the checkpoint ID, package hash, relative package path, run instance, phase, evaluation identity, completion time and retained state; it never contains checkpoint binaries. Query retained references to confirm the latest package for an instance and phase:: from palaestrai.store import Session from palaestrai.store.query import checkpoint_refs with Session() as session: latest = checkpoint_refs( session, experiment_run_instance_uids=["4ed6c8bb-3d20-4ffb-b253-6c6ce0cb58a1"], experiment_run_phases=[0], retained=True, ) print(latest[["package_hash", "relative_path", "completed_at", "evaluation_id"]]) After a successful resume, use the same log fields and Store query to confirm that later evaluation evidence refers to the restored run state and a newly committed latest package. Troubleshooting --------------- ``checkpoint.resume: latest requires ...`` Supply both ``resume_instance_uid`` and ``resume_phase``. PalaestrAI does not guess which historical run to restart. ``no latest checkpoint`` or a missing latest package The selected instance and phase have no completed checkpoint in this ``data_path``. Check that the run reached an evaluation boundary, that its Brain supports portable checkpoints, and that ``data_path`` points to the original output root. Corruption, a manifest/hash error, or an incompatibility error Stop the restart. PalaestrAI rejects incomplete, altered, or incompatible packages rather than loading them partially. Use the original environment, agent configuration, Store, and feature/action contracts, or start a new run without ``resume: latest``. Insufficient free space Free capacity on the filesystem containing ``data_path`` and rerun. Space for a complete replacement package is required; this safety margin is not a configuration option. ``worker: 1`` / multi-worker checkpointing error Portable checkpointing currently supports one rollout worker. Run with ``worker: 1`` or disable checkpointing. A coordinated multi-worker quiescence protocol is required before multi-worker recovery can be safe. Missing Store or CheckpointRef evidence error Configure and retain access to the Store used by the original run. A checkpoint package is not published without its metadata-only Store evidence. Safety and compatibility ------------------------ PalaestrAI retains exactly one fully committed checkpoint per phase. The current latest package is verified before it is resumed; package identity, content hashes, Store evidence, run/phase identity, and agent/configuration contracts must agree. Packages stay under the resolved ``data_path`` root and are protected against unsafe paths and incomplete replacement. These are fixed framework guarantees, not per-run tuning options. The portable checkpoint records algorithm state and PalaestrAI's run and condition state at a completed evaluation boundary. It does not claim to restore an interrupted environment step or a partially completed gradient update.