.. _observability: #################### OpenTelemetry Setup #################### This document describes the OpenTelemetry (OTel) instrumentation in palaestrAI, the metrics it exposes, and how to use Grafana to analyse runtime performance. ---- ============= OTel Overview ============= palaestrAI uses OpenTelemetry for **distributed tracing** and **application metrics**. Telemetry flows through the following pipeline: .. code-block:: text palaestrAI (worker subprocesses) │ │ OTLP (gRPC) ▼ OpenTelemetry Collector (:4317) │ ├── traces ──► Tempo ──► Grafana ├── logs ──► Loki ──► Grafana └── metrics ─► Prometheus exporter (:8889) │ ▼ Prometheus (:9090) ──► Grafana The OTel Collector is the central ingestion point. All signals use the ``otlp`` protocol when ``otel_enabled: true`` in the runtime configuration. ---- ================== Enabling Telemetry ================== In the runtime configuration file: .. code-block:: yaml otel_enabled: true otel_exporter: otlp # "otlp" or "jsonl" otel_collector_endpoint: http://otel-collector:4317 otel_service_name_prefix: "palaestrai" The ``otel_service_name_prefix`` is prepended to every service name, producing metrics such as ``palaestrai.SimulationController-abc123`` in the collector. When ``otel_enabled`` is ``false`` (the default), all OTel calls are no-ops and add zero overhead. The OTLP exporter uses **insecure gRPC** (no TLS). The endpoint must be reachable from the process. For local development this is typically ``http://localhost:4317``; in the Docker stack it is ``http://otel-collector:4317``. When the collector is unreachable, the OTel SDK queues metric data in memory. If the process is terminated before the next export interval (default 5 s), buffered metrics are lost. palaestrAI installs a SIGTERM handler (see :ref:`signal-flush`) to mitigate this. ---- .. _signal-flush: ============================================= SIGTERM Shutdown Flush (Metrics Preservation) ============================================= palaestrAI worker subprocesses are terminated by the parent ESM using ``process.terminate()`` (SIGTERM) and, as a fallback, ``process.kill()`` (SIGKILL). The OS default disposition for SIGTERM bypasses Python's ``atexit`` handlers, ``finally`` blocks, and the daemon export thread — so buffered metrics are silently dropped. To prevent this, ``init_tracer()`` in ``palaestrai/util/otel.py`` installs a best-effort SIGTERM/SIGINT handler that calls ``shutdown_tracer()`` before the process exits. This performs a synchronous flush of all pending metric data. The handler: 1. Calls ``shutdown_tracer()`` which forces the ``PeriodicExportingMetricReader`` to collect and export all buffered metrics. 2. Re-raises the signal so the process terminates normally (the parent ESM expects this). 3. Only overrides handlers that are still at their OS default or ``ignore`` value, so it does not clobber custom handlers installed by other components. This fix is applied in ``palaestrai/util/otel.py`` and covers the dominant SIGTERM case. SIGKILL (uncatchable) cannot be handled — for processes killed with SIGKILL, metrics will be lost. The solution is to ensure the parent ESM uses SIGTERM with a grace period before SIGKILL. ---- ============= Configuration ============= +-------------------------------+---------------------------+------------------------------------------+ | Key | Default | Description | +===============================+===========================+==========================================+ | ``otel_enabled`` | ``false`` | Enable OTel tracing and metrics | +-------------------------------+---------------------------+------------------------------------------+ | ``otel_collector_endpoint`` | ``http://localhost:4317`` | OTLP gRPC endpoint | +-------------------------------+---------------------------+------------------------------------------+ | ``otel_service_name_prefix`` | ``"palaestrai"`` | Prefix for service names | +-------------------------------+---------------------------+------------------------------------------+ | ``otel_exporter`` | ``"otlp"`` | ``"otlp"`` (gRPC) or ``"jsonl"`` (file) | +-------------------------------+---------------------------+------------------------------------------+ | ``otel_jsonl_file`` | ``""`` | Output path for ``jsonl`` trace exporter | +-------------------------------+---------------------------+------------------------------------------+ In the Docker stack, the configuration file is bind-mounted to ``/root/.config/palaestrai/runtime-conf.yaml`` inside the ``palaestrai`` container. ---- ============ Metric Names ============ Application metrics are prefixed with ``palaestrai.`` and follow the naming convention: .. code-block:: text palaestrai... The OTel Prometheus exporter converts metric names and adds the label ``exported_job="palaestrai"`` to all palaestrAI metrics. **Example converted name in Prometheus:** :: palaestrai_simulation_step_duration_seconds vs. the OTel API name: :: palaestrai.simulation.step.duration Always query Prometheus directly to confirm the exact exported metric name, as the OTel Prometheus exporter performs name translation. ---- .. _simulation-controller-metrics: ============================ SimulationController Metrics ============================ The ``SimulationController`` (via ``palaestrai/simulation/telemetry.py``) exposes the following metrics: Step Metrics ------------------------------ **``palaestrai.simulation.step.duration``** *(Histogram, seconds)* Duration of one complete ``step()`` call. Attributes: ``num_agents``, ``num_environments`` Use to understand how long each simulation step takes. **``palaestrai.simulation.step.count``** *(Counter)* Total number of completed ``step()`` calls. **``palaestrai.simulation.step.phase.duration``** *(Histogram, seconds)* Duration of each sub-phase within ``step()``. Attributes: ``phase`` Phase values: ``environment_wait``, ``environment_update_request``, ``result_processing``, ``flow_control``, ``pause_wait``, ``other`` Use to identify which part of the step is the bottleneck: - ``environment_wait`` → IPC/synchronization with environment processes - ``environment_update_request`` → local request dispatch - ``result_processing`` → local result handling - ``flow_control`` → flow control decisions Act Metrics ------------------------------ **``palaestrai.simulation.act.duration``** *(Histogram, seconds)* Duration of one complete ``act()`` call. Attributes: ``num_agents``, ``is_terminal`` **``palaestrai.simulation.act.count``** *(Counter)* Total number of completed ``act()`` calls. **``palaestrai.simulation.act.phase.duration``** *(Histogram, seconds)* Duration of each sub-phase within ``act()``. Attributes: ``phase`` Phase values: ``agent_wait``, ``agent_request``, ``result_processing``, ``flow_control``, ``pause_wait``, ``other`` Use to understand agent interaction overhead: - ``agent_wait`` → waiting for agent computation/communication - ``agent_request`` → local request dispatch to agents - ``result_processing`` → local result handling - ``flow_control`` → flow control decisions Loop Metrics ------------------------------ **``palaestrai.simulation.loop.iterations``** *(Counter)* Total simulation loop iterations. Attributes: ``num_agents`` ---- .. _rungovernor-metrics: =================== RunGovernor Metrics =================== The ``RunGovernor`` (via ``palaestrai/experiment/telemetry.py``) exposes: Run Lifecycle Metrics ------------------------------ **``palaestrai.runGovernor.run.startup.duration``** *(Histogram, seconds)* Duration of run-level startup/initialization. Attributes: ``num_phases`` **``palaestrai.runGovernor.phase.setup.duration``** *(Histogram, seconds)* Duration of phase setup (spawning processes). Attributes: ``phase``, ``num_simulation_controllers``, ``num_agent_conductors``, ``num_environment_conductors`` **``palaestrai.runGovernor.phase.execution.duration``** *(Histogram, seconds)* Duration of active phase computation. Attributes: ``phase`` **``palaestrai.runGovernor.phase.wait.duration``** *(Histogram, seconds)* Duration of waiting for phase processes to complete. Attributes: ``phase`` **``palaestrai.runGovernor.phase.shutdown.duration``** *(Histogram, seconds)* Duration of phase shutdown. Attributes: ``phase`` Counter Metrics ------------------------------ **``palaestrai.runGovernor.phases.count``** *(Counter)* Total phases executed. Attributes: ``phase`` **``palaestrai.runGovernor.simulationControllers.spawned``** *(Counter)* Total simulation controllers spawned. Attributes: ``type`` (``simulation_controller`` or ``evaluation_controller``) **``palaestrai.runGovernor.conductors.spawned``** *(Counter)* Total conductors spawned. Attributes: ``type`` (``agent`` or ``environment``) ---- ================= Grafana Dashboard ================= A Grafana dashboard for palaestrAI runtime analysis is provisioned at: :: containers/palaestrai-stack/grafana/provisioning/dashboards/palaestrai-runtime.json **UID:** ``palaestrai-runtime`` **Datasource:** Prometheus (uid: ``prometheus``) The dashboard is automatically provisioned when Grafana starts, using the standard Grafana dashboard provider configuration in: :: containers/palaestrai-stack/grafana/provisioning/dashboards/dashboards.yaml To view the dashboard: 1. Open Grafana at http://localhost:3001 (default credentials: admin/admin). 2. Navigate to **Dashboards → Browse → PalaestrAI → palaestrAI Runtime**. ---- Dashboard Sections ------------------------------ **palaestrAI Runtime Overview** High-level summary panels: - Total step calls (5m rate) - Mean step duration - p95 step duration - Total act calls - Mean act duration **SimulationController — Step** - Step duration over time (mean, p50, p95, p99) - Step phase breakdown (stacked time series) - Step count over time **SimulationController — Act** - Act duration over time - Act phase breakdown - Act count over time **Run Lifecycle** - Phase setup/execution/wait/shutdown durations over time - Phases count over time **Bottleneck Indicators** Stat panels highlighting high-latency conditions. ---- Example PromQL Queries ------------------------------ **Mean step duration:** .. code-block:: promql rate(palaestrai_simulation_step_duration_seconds_sum{exported_job="palaestrai"}[5m]) / rate(palaestrai_simulation_step_duration_seconds_count{exported_job="palaestrai"}[5m]) **Step p95 latency:** .. code-block:: promql histogram_quantile(0.95, sum(rate(palaestrai_simulation_step_duration_seconds_bucket{exported_job="palaestrai"}[5m])) by (le)) **Step phase breakdown (stacked):** .. code-block:: promql sum by (phase) ( rate(palaestrai_simulation_step_phase_duration_seconds_sum{exported_job="palaestrai"}[5m]) ) **Act p99 latency:** .. code-block:: promql histogram_quantile(0.99, sum(rate(palaestrai_simulation_act_duration_seconds_bucket{exported_job="palaestrai"}[5m])) by (le)) **Run phase execution rate:** .. code-block:: promql sum by (phase) ( rate(palaestrai_runGovernor_phase_execution_duration_seconds_sum{exported_job="palaestrai"}[5m]) ) ---- ================================= Example Bottleneck Interpretation ================================= These are **indicators**, not definitive causal diagnoses. .. list-table:: Bottleneck observations and likely causes :header-rows: 1 * - Observation - Possible interpretation * - High ``step()`` p95/p99 duration - Check the step phase breakdown * - + high ``environment_wait`` - Environment/IPC/synchronization issue * - + high ``result_processing`` - Local processing bottleneck * - High ``act()`` p95/p99 duration - Check the act phase breakdown * - + high ``agent_wait`` - Agent computation/communication issue * - High phase ``setup`` duration - Process spawning overhead * - High phase ``wait`` duration - Phase process sync/termination overhead * - High ``run.startup`` vs execution - Orchestration/process lifecycle overhead ---- =========================================== Developer Guide: Adding New Instrumentation =========================================== When instrumenting a new component, follow these conventions: When to Use Which Instrument ------------------------------ +------------------------+------------------+----------------------------------+ | What to measure | Instrument | Use when | +========================+==================+==================================+ | Count of events | Counter | Tracking total calls, errors, | | | | completions | +------------------------+------------------+----------------------------------+ | Distribution of values | Histogram | Measuring latency, duration, | | | | payload sizes | +------------------------+------------------+----------------------------------+ | Current value (gauge) | UpDownCounter | Tracking pool sizes, active | | | | connections | +------------------------+------------------+----------------------------------+ Naming Conventions ------------------------------ :: palaestrai... - Use **seconds** for durations (``s`` unit in OTel). - Use **count** for counters (unit ``{count}``). - Keep component and operation names lowercase with underscores. - Avoid high-cardinality attributes (e.g., do not put a UUID in an attribute). Adding Metrics to a Module ------------------------------ 1. Import the telemetry recording helpers or create a module-level ``_meter``: .. code-block:: python from opentelemetry import metrics _meter = metrics.get_meter(__name__, "1.0.0") 2. Create instruments at module level (created once, reused): .. code-block:: python _my_duration = _meter.create_histogram( name="palaestrai.my_component.operation.duration", unit="s", description="Duration of my operation", ) 3. Record measurements: .. code-block:: python _my_duration.record(duration_seconds, {"attribute": value}) 4. In ``init_tracer()``, the meter provider and readers are configured globally. No per-instrument configuration is needed. Unit Conventions ------------------------------ .. list-table:: Metric units and when to use them :header-rows: 1 * - Unit - When to use * - ``s`` - Durations (histogram for latencies) * - ``ms`` - Sub-second durations (optional) * - ``{count}`` - Counters (no specific quantity) Avoiding Performance Pitfalls ------------------------------ - **Do not add OTel calls in hot paths without guards.** Check ``otel_enabled`` or use the no-op pattern for expensive instrumentation. - **Avoid high-cardinality attributes** (e.g., per-request IDs, large numerical values). These explode metric cardinality. - **Minimize attribute count.** Each attribute combination creates a new time series in Prometheus. - **Use histograms for distributions** rather than trying to record every individual value. - **Keep instrument names stable** — renaming a metric creates a new time series and breaks historical queries. Testing Instrumentation ------------------------------ 1. Enable ``otel_enabled: true`` and run a small experiment. 2. Query Prometheus to confirm metrics appear: .. code-block:: bash promtool query instant 'palaestrai_your_metric_name' 3. Check the OTel Collector logs for any export errors. 4. Verify the metric appears in Grafana and the panel shows data. Adding a Grafana Panel ------------------------------ 1. Add the panel definition to ``containers/palaestrai-stack/grafana/provisioning/dashboards/palaestrai-runtime.json``. 2. Use ``{ "uid": "prometheus" }`` as the datasource. 3. Use ``exported_job="palaestrai"`` as a label filter. 4. Use ``histogram_quantile`` for histogram percentiles. 5. Use ``rate(...[5m])`` for rates over time. 6. Test by running an experiment and verifying data appears in the panel. ---- ============== Stack Commands ============== Start the stack: .. code-block:: bash cd containers/palaestrai-stack docker compose up -d Run the validation experiment: .. code-block:: bash docker compose exec palaestrai \ palaestrai -c /root/.config/palaestrai/runtime-conf.yaml \ start /palaestrai/tests/fixtures/Classic-ARL-Experiment_run-0.yml Restart after configuration changes: .. code-block:: bash docker compose restart otel-collector prometheus grafana View collector logs: .. code-block:: bash docker compose logs -f otel-collector