> ## Documentation Index
> Fetch the complete documentation index at: https://docs.polygon.technology/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Prometheus metrics exposed by the Agglayer node: per-network certificate heights, error flag, and bridging-time histograms.

## Overview

The Agglayer node exposes Prometheus metrics over an HTTP `/metrics` endpoint. Metric instruments are defined with OpenTelemetry and exported through `opentelemetry-prometheus`.

The listen address is configured under `[telemetry]` in the node's TOML configuration:

```toml theme={null}
[telemetry]
prometheus-addr = "0.0.0.0:3000"
```

Scrape `http://<prometheus-addr>/metrics` to collect the series described below.

## Per-network certificate state

Two observable gauges report the current certificate state of each network. Values are read from storage at scrape time, so exported values always match storage truth, including after restarts and admin edits.

| Metric                                              | Type           | Labels                | Description                                                                                |
| --------------------------------------------------- | -------------- | --------------------- | ------------------------------------------------------------------------------------------ |
| `agglayer_node_network_height`                      | Gauge          | `network_id`, `stage` | Height of the latest certificate for the network at the given lifecycle stage.             |
| `agglayer_node_network_latest_certificate_in_error` | Gauge (0 or 1) | `network_id`          | `1` if the latest pending certificate for the network has status `InError`, otherwise `0`. |

Stages exposed on `agglayer_node_network_height`:

| `stage`   | Meaning                                   |
| --------- | ----------------------------------------- |
| `pending` | Height of the latest pending certificate. |
| `proven`  | Height of the latest proven certificate.  |
| `settled` | Height of the latest settled certificate. |

Notes on semantics:

* A network with no recorded pointer for a given stage exports no series for that stage. A height of `0` is a valid height (the first certificate of a network) and is never used as a placeholder.
* `agglayer_node_network_latest_certificate_in_error` reports the storage truth for the latest pending certificate header. It does not consult the disabled-networks list, so a disabled network with an in-error pending certificate still exports `in_error=1`. Alerts that need to exclude disabled networks must join against that list separately.

## Certificate bridging-time histograms

Two histograms measure how long certificates take to move through the lifecycle. Both use the OpenTelemetry meter scope `agglayer_node_certificate`.

| Metric                                        | Type      | Labels                | Description                                                                 |
| --------------------------------------------- | --------- | --------------------- | --------------------------------------------------------------------------- |
| `agglayer_certificate_duration_seconds`       | Histogram | `network_id`          | End-to-end bridging time of a certificate, from `Pending` to `Settled`.     |
| `agglayer_certificate_stage_duration_seconds` | Histogram | `network_id`, `stage` | Time spent in each non-terminal lifecycle stage before the next transition. |

Stages exposed on `agglayer_certificate_stage_duration_seconds`:

| `stage`     | State timed | Ends at     | Covers                                                                    |
| ----------- | ----------- | ----------- | ------------------------------------------------------------------------- |
| `pending`   | `Pending`   | `Proven`    | Proof generation (certification).                                         |
| `proven`    | `Proven`    | `Candidate` | Calldata build and settlement job submission, including L1 `estimateGas`. |
| `candidate` | `Candidate` | `Settled`   | L1 inclusion and confirmation wait.                                       |

The three stages are contiguous, so their durations sum to `agglayer_certificate_duration_seconds` for the same certificate.

Both histograms share one bucket set (seconds):

```text theme={null}
0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, 900, 1800
```

<Note>
  Timings are measured with in-memory timers on the certificate task and are not persisted. Counts reset when the node restarts.

  Only fresh certificates observed from `Pending` through `Settled` within a single process lifetime contribute to these histograms. Certificates resumed after a restart (entering as `Proven` or `Candidate`) contribute no durations, which keeps end-to-end and per-stage distributions honest.

  A certificate that errors mid-lifecycle still contributes the stages that completed; only the end-to-end total requires reaching `Settled`.

  Timing starts when the certificate task begins processing the certificate, not at RPC receipt. Time spent waiting in the pending queue before pickup is not included.
</Note>

## Example PromQL

End-to-end p95 bridging time per network:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (le, network_id) (
    rate(agglayer_certificate_duration_seconds_bucket[$__rate_interval])
  )
)
```

Median time spent in each stage:

```promql theme={null}
histogram_quantile(
  0.5,
  sum by (le, stage) (
    rate(agglayer_certificate_stage_duration_seconds_bucket[$__rate_interval])
  )
)
```

Networks currently reporting the latest certificate in error:

```promql theme={null}
agglayer_node_network_latest_certificate_in_error == 1
```

Gap between the latest pending and latest settled height per network:

```promql theme={null}
agglayer_node_network_height{stage="pending"}
  - on (network_id)
agglayer_node_network_height{stage="settled"}
```

## Deployment labels

Labels such as `environment` or `cluster` are expected to be added at scrape time via Prometheus `external_labels` rather than emitted by the node.