OpenTelemetry Setup

This document describes the OpenTelemetry (OTel) instrumentation in palaestrAI, the metrics it exposes, and how to use Grafana to analyse runtime performance.

OTel Overview

palaestrAI uses OpenTelemetry for distributed tracing and application metrics. Telemetry flows through the following pipeline:

palaestrAI (worker subprocesses)
    │
    │  OTLP (gRPC)
    ▼
OpenTelemetry Collector (:4317)
    │
    ├── traces ──► Tempo ──► Grafana
    ├── logs  ──► Loki  ──► Grafana
    └── metrics ─► Prometheus exporter (:8889)
                   │
                   ▼
              Prometheus (:9090) ──► Grafana

The OTel Collector is the central ingestion point. All signals use the otlp protocol when otel_enabled: true in the runtime configuration.

Enabling Telemetry

In the runtime configuration file:

otel_enabled: true
otel_exporter: otlp            # "otlp" or "jsonl"
otel_collector_endpoint: http://otel-collector:4317
otel_service_name_prefix: "palaestrai"

The otel_service_name_prefix is prepended to every service name, producing metrics such as palaestrai.SimulationController-abc123 in the collector.

When otel_enabled is false (the default), all OTel calls are no-ops and add zero overhead.

The OTLP exporter uses insecure gRPC (no TLS). The endpoint must be reachable from the process. For local development this is typically http://localhost:4317; in the Docker stack it is http://otel-collector:4317.

When the collector is unreachable, the OTel SDK queues metric data in memory. If the process is terminated before the next export interval (default 5 s), buffered metrics are lost. palaestrAI installs a SIGTERM handler (see SIGTERM Shutdown Flush (Metrics Preservation)) to mitigate this.

SIGTERM Shutdown Flush (Metrics Preservation)

palaestrAI worker subprocesses are terminated by the parent ESM using process.terminate() (SIGTERM) and, as a fallback, process.kill() (SIGKILL). The OS default disposition for SIGTERM bypasses Python’s atexit handlers, finally blocks, and the daemon export thread — so buffered metrics are silently dropped.

To prevent this, init_tracer() in palaestrai/util/otel.py installs a best-effort SIGTERM/SIGINT handler that calls shutdown_tracer() before the process exits. This performs a synchronous flush of all pending metric data.

The handler:

  1. Calls shutdown_tracer() which forces the PeriodicExportingMetricReader to collect and export all buffered metrics.

  2. Re-raises the signal so the process terminates normally (the parent ESM expects this).

  3. Only overrides handlers that are still at their OS default or ignore value, so it does not clobber custom handlers installed by other components.

This fix is applied in palaestrai/util/otel.py and covers the dominant SIGTERM case. SIGKILL (uncatchable) cannot be handled — for processes killed with SIGKILL, metrics will be lost. The solution is to ensure the parent ESM uses SIGTERM with a grace period before SIGKILL.

Configuration

Key

Default

Description

otel_enabled

false

Enable OTel tracing and metrics

otel_collector_endpoint

http://localhost:4317

OTLP gRPC endpoint

otel_service_name_prefix

"palaestrai"

Prefix for service names

otel_exporter

"otlp"

"otlp" (gRPC) or "jsonl" (file)

otel_jsonl_file

""

Output path for jsonl trace exporter

In the Docker stack, the configuration file is bind-mounted to /root/.config/palaestrai/runtime-conf.yaml inside the palaestrai container.

Metric Names

Application metrics are prefixed with palaestrai. and follow the naming convention:

palaestrai.<component>.<instrument>.<unit>

The OTel Prometheus exporter converts metric names and adds the label exported_job="palaestrai" to all palaestrAI metrics.

Example converted name in Prometheus:

palaestrai_simulation_step_duration_seconds

vs. the OTel API name:

palaestrai.simulation.step.duration

Always query Prometheus directly to confirm the exact exported metric name, as the OTel Prometheus exporter performs name translation.

SimulationController Metrics

The SimulationController (via palaestrai/simulation/telemetry.py) exposes the following metrics:

Step Metrics

``palaestrai.simulation.step.duration`` (Histogram, seconds)

Duration of one complete step() call.

Attributes: num_agents, num_environments

Use to understand how long each simulation step takes.

``palaestrai.simulation.step.count`` (Counter)

Total number of completed step() calls.

``palaestrai.simulation.step.phase.duration`` (Histogram, seconds)

Duration of each sub-phase within step().

Attributes: phase

Phase values: environment_wait, environment_update_request, result_processing, flow_control, pause_wait, other

Use to identify which part of the step is the bottleneck: - environment_wait → IPC/synchronization with environment processes - environment_update_request → local request dispatch - result_processing → local result handling - flow_control → flow control decisions

Act Metrics

``palaestrai.simulation.act.duration`` (Histogram, seconds)

Duration of one complete act() call.

Attributes: num_agents, is_terminal

``palaestrai.simulation.act.count`` (Counter)

Total number of completed act() calls.

``palaestrai.simulation.act.phase.duration`` (Histogram, seconds)

Duration of each sub-phase within act().

Attributes: phase

Phase values: agent_wait, agent_request, result_processing, flow_control, pause_wait, other

Use to understand agent interaction overhead: - agent_wait → waiting for agent computation/communication - agent_request → local request dispatch to agents - result_processing → local result handling - flow_control → flow control decisions

Loop Metrics

``palaestrai.simulation.loop.iterations`` (Counter)

Total simulation loop iterations.

Attributes: num_agents

RunGovernor Metrics

The RunGovernor (via palaestrai/experiment/telemetry.py) exposes:

Run Lifecycle Metrics

``palaestrai.runGovernor.run.startup.duration`` (Histogram, seconds)

Duration of run-level startup/initialization.

Attributes: num_phases

``palaestrai.runGovernor.phase.setup.duration`` (Histogram, seconds)

Duration of phase setup (spawning processes).

Attributes: phase, num_simulation_controllers, num_agent_conductors, num_environment_conductors

``palaestrai.runGovernor.phase.execution.duration`` (Histogram, seconds)

Duration of active phase computation.

Attributes: phase

``palaestrai.runGovernor.phase.wait.duration`` (Histogram, seconds)

Duration of waiting for phase processes to complete.

Attributes: phase

``palaestrai.runGovernor.phase.shutdown.duration`` (Histogram, seconds)

Duration of phase shutdown.

Attributes: phase

Counter Metrics

``palaestrai.runGovernor.phases.count`` (Counter)

Total phases executed.

Attributes: phase

``palaestrai.runGovernor.simulationControllers.spawned`` (Counter)

Total simulation controllers spawned.

Attributes: type (simulation_controller or evaluation_controller)

``palaestrai.runGovernor.conductors.spawned`` (Counter)

Total conductors spawned.

Attributes: type (agent or environment)

Grafana Dashboard

A Grafana dashboard for palaestrAI runtime analysis is provisioned at:

containers/palaestrai-stack/grafana/provisioning/dashboards/palaestrai-runtime.json

UID: palaestrai-runtime

Datasource: Prometheus (uid: prometheus)

The dashboard is automatically provisioned when Grafana starts, using the standard Grafana dashboard provider configuration in:

containers/palaestrai-stack/grafana/provisioning/dashboards/dashboards.yaml

To view the dashboard:

  1. Open Grafana at http://localhost:3001 (default credentials: admin/admin).

  2. Navigate to Dashboards → Browse → PalaestrAI → palaestrAI Runtime.

Dashboard Sections

palaestrAI Runtime Overview

High-level summary panels: - Total step calls (5m rate) - Mean step duration - p95 step duration - Total act calls - Mean act duration

SimulationController — Step

  • Step duration over time (mean, p50, p95, p99)

  • Step phase breakdown (stacked time series)

  • Step count over time

SimulationController — Act

  • Act duration over time

  • Act phase breakdown

  • Act count over time

Run Lifecycle

  • Phase setup/execution/wait/shutdown durations over time

  • Phases count over time

Bottleneck Indicators

Stat panels highlighting high-latency conditions.

Example PromQL Queries

Mean step duration:

rate(palaestrai_simulation_step_duration_seconds_sum{exported_job="palaestrai"}[5m])
/ rate(palaestrai_simulation_step_duration_seconds_count{exported_job="palaestrai"}[5m])

Step p95 latency:

histogram_quantile(0.95,
  sum(rate(palaestrai_simulation_step_duration_seconds_bucket{exported_job="palaestrai"}[5m])) by (le))

Step phase breakdown (stacked):

sum by (phase) (
  rate(palaestrai_simulation_step_phase_duration_seconds_sum{exported_job="palaestrai"}[5m])
)

Act p99 latency:

histogram_quantile(0.99,
  sum(rate(palaestrai_simulation_act_duration_seconds_bucket{exported_job="palaestrai"}[5m])) by (le))

Run phase execution rate:

sum by (phase) (
  rate(palaestrai_runGovernor_phase_execution_duration_seconds_sum{exported_job="palaestrai"}[5m])
)

Example Bottleneck Interpretation

These are indicators, not definitive causal diagnoses.

Bottleneck observations and likely causes

Observation

Possible interpretation

High step() p95/p99 duration

Check the step phase breakdown

  • high environment_wait

Environment/IPC/synchronization issue

  • high result_processing

Local processing bottleneck

High act() p95/p99 duration

Check the act phase breakdown

  • high agent_wait

Agent computation/communication issue

High phase setup duration

Process spawning overhead

High phase wait duration

Phase process sync/termination overhead

High run.startup vs execution

Orchestration/process lifecycle overhead

Developer Guide: Adding New Instrumentation

When instrumenting a new component, follow these conventions:

When to Use Which Instrument

What to measure

Instrument

Use when

Count of events

Counter

Tracking total calls, errors, completions

Distribution of values

Histogram

Measuring latency, duration, payload sizes

Current value (gauge)

UpDownCounter

Tracking pool sizes, active connections

Naming Conventions

palaestrai.<component>.<operation>.<unit>

  • Use seconds for durations (s unit in OTel).

  • Use count for counters (unit {count}).

  • Keep component and operation names lowercase with underscores.

  • Avoid high-cardinality attributes (e.g., do not put a UUID in an attribute).

Adding Metrics to a Module

  1. Import the telemetry recording helpers or create a module-level _meter:

    from opentelemetry import metrics
_meter = metrics.get_meter(__name__, "1.0.0")

  2. Create instruments at module level (created once, reused):

    _my_duration = _meter.create_histogram(
    name="palaestrai.my_component.operation.duration",
    unit="s",
    description="Duration of my operation",
)

  3. Record measurements:

    _my_duration.record(duration_seconds, {"attribute": value})

  4. In init_tracer(), the meter provider and readers are configured globally. No per-instrument configuration is needed.

Unit Conventions

Metric units and when to use them

Unit

When to use

s

Durations (histogram for latencies)

ms

Sub-second durations (optional)

{count}

Counters (no specific quantity)

Avoiding Performance Pitfalls

  • Do not add OTel calls in hot paths without guards. Check otel_enabled or use the no-op pattern for expensive instrumentation.

  • Avoid high-cardinality attributes (e.g., per-request IDs, large numerical values). These explode metric cardinality.

  • Minimize attribute count. Each attribute combination creates a new time series in Prometheus.

  • Use histograms for distributions rather than trying to record every individual value.

  • Keep instrument names stable — renaming a metric creates a new time series and breaks historical queries.

Testing Instrumentation

  1. Enable otel_enabled: true and run a small experiment.

  2. Query Prometheus to confirm metrics appear:

    promtool query instant 'palaestrai_your_metric_name'

  3. Check the OTel Collector logs for any export errors.

  4. Verify the metric appears in Grafana and the panel shows data.

Adding a Grafana Panel

  1. Add the panel definition to containers/palaestrai-stack/grafana/provisioning/dashboards/palaestrai-runtime.json.

  2. Use { "uid": "prometheus" } as the datasource.

  3. Use exported_job="palaestrai" as a label filter.

  4. Use histogram_quantile for histogram percentiles.

  5. Use rate(...[5m]) for rates over time.

  6. Test by running an experiment and verifying data appears in the panel.

Stack Commands

Start the stack:

cd containers/palaestrai-stack
docker compose up -d

Run the validation experiment:

docker compose exec palaestrai \
    palaestrai -c /root/.config/palaestrai/runtime-conf.yaml \
    start /palaestrai/tests/fixtures/Classic-ARL-Experiment_run-0.yml

Restart after configuration changes:

docker compose restart otel-collector prometheus grafana

View collector logs:

docker compose logs -f otel-collector