Metrics & the ops view

This page covers what the ops listener tells you beyond health: Prometheus metrics, read-only views of what the server stores, and Go profiling. The listener itself, its loopback binding and the health endpoints are on Health & readiness.

Prometheus metrics

/metrics serves Prometheus exposition when features.enable_metrics is on, which is the default:

features:
  enable_metrics: true

With it off, /metrics is not mounted. The exposition comes from the server’s own registry, not Prometheus’s global one, so it holds exactly the series below plus the standard Go runtime and process collectors.

scrape_configs:
  - job_name: ricochet
    static_configs:
      - targets: ["127.0.0.1:9090"]

If Prometheus runs on another host, run it next to the server, tunnel to it, or move ops.bind to an interface on a network you trust.

What it publishes

Family	Series
Requests	`ricochet_requests_total`, `ricochet_request_duration_seconds` (histogram, 1 ms to 10 s), labelled `protocol`, `operation`, `outcome`
Admission	`ricochet_requests_in_flight`, `ricochet_admission_max_in_flight`, `ricochet_admission_admitted_total`, `ricochet_admission_shed_total`, `ricochet_admission_waited_total`, `ricochet_admission_tracked_peers`
Database pool	`ricochet_db_pool_conns_total`, `ricochet_db_pool_conns_idle`, `ricochet_db_pool_conns_acquired`, `ricochet_db_pool_conns_max`, `ricochet_db_pool_empty_acquire_total`, `ricochet_db_pool_canceled_acquire_total`, `ricochet_db_pool_acquire_seconds_total`
Buffer pool	`ricochet_buffer_pool_hits_total`, `ricochet_buffer_pool_misses_total`
Capacity	`ricochet_mailboxes`, `ricochet_messages_stored`, `ricochet_message_bytes`, `ricochet_database_bytes`, `ricochet_storage_max_bytes`, `ricochet_mailboxes_near_capacity`, `ricochet_mailbox_depth` (label `bucket`), `ricochet_stats_age_seconds`

Every series with its type and help text is in Prometheus metrics.

The outcome label takes six values: ok, client_error (a 4xx), server_error (a 5xx), rate_limited (a peer over a rate limit you set), overloaded (shed by admission control) and error (a pipeline failure such as a decode error). Keep rate_limited and overloaded apart: the first calls for a bigger budget, the second for more capacity.

Peer IDs never appear in a label. With thousands of clients that would be unbounded cardinality. Per-peer questions go to the ops views below.

The capacity series come from a sample taken every intervals.cleanup_min, not from live queries. Alert on ricochet_stats_age_seconds as well: a sampler that stopped would otherwise look like numbers that stopped changing.

Starting alerts

These are suggestions to adapt, not shipped rules:

groups:
  - name: ricochet
    rules:
      - alert: RicochetShedding
        expr: rate(ricochet_admission_shed_total[5m]) > 0
        for: 10m
      - alert: RicochetMailboxesNearCapacity
        expr: ricochet_mailboxes_near_capacity > 0
        for: 30m
      - alert: RicochetStorageBudget
        expr: ricochet_database_bytes / ricochet_storage_max_bytes > 0.85
      - alert: RicochetCapacitySampleStale
        expr: ricochet_stats_age_seconds > 1800

The ops view of stored data

Four read-only endpoints answer questions about what the server holds, for any owner, without a peer identity or a client library:

Endpoint	Answers
`/ops/mailboxes/top?n=20`	Which mailboxes are closest to full, across every owner. `sort=count` orders by message count instead of fill.
`/ops/mailboxes?owner=<peer-id>`	One peer’s mailboxes, fullest first
`/ops/storage`	Per-owner totals, plus the sampled server-wide figures
`/ops/limits`	The limits the server actually parsed and derived

OPS=http://127.0.0.1:9090

# Mailboxes already at their cap
curl -s "$OPS/ops/mailboxes/top?n=50" | jq '.mailboxes[] | select(.full)'

# Everything one peer is storing
curl -s "$OPS/ops/mailboxes?owner=12D3KooW..." | jq

# How much room is left, and the five biggest owners
curl -s "$OPS/ops/storage" | jq '{server, top: (.owners[:5])}'

# Did the change I made take effect?
curl -s "$OPS/ops/limits" | jq .admission

Things to know when reading them:

Each mailbox row carries owner, folderPath, messageCount, maxMessages, fillRatio, full, messageBytes and lastMessageAt.
Paging uses limit and offset (n is an alias for limit on /top). The default page is 20 rows and the maximum 500.
/ops/storage reports per-owner totals live, but its server block comes from the capacity sample. It carries sampledAt and ageSeconds, and is null with a note until the first sample completes.
/ops/limits reports effective values. A maxInFlight the server derived from the pool size is shown with the number it derived and maxInFlightDerivedFromPoolSize: true. Every rate-limit bucket has an explicit enabled.
A malformed parameter, such as n=twenty or an invalid peer ID, is a 400, not a silent default.
These queries scan every mailbox. Each is bounded by ops.query_timeout (10 s by default); one that runs over returns 504 and logs ops query timed out. Raise the timeout on a large server.

The full request and response shapes are in Ops HTTP endpoints.

pprof

Go’s profiler endpoints are off by default, because profiles include goroutine stacks and heap contents. Turn them on to diagnose, then turn them off:

ops:
  enable_pprof: true

After a restart, /debug/pprof/ is mounted on the ops listener:

go tool pprof http://127.0.0.1:9090/debug/pprof/heap
go tool pprof "http://127.0.0.1:9090/debug/pprof/profile?seconds=30"
curl -s "http://127.0.0.1:9090/debug/pprof/goroutine?debug=1" | head -50

Logs

The server logs to standard output in logfmt-style text: INFO by default, DEBUG with --development. At INFO every request is exactly one line, msg=request, with protocol, op, status, peer, duration and stream. A refused request adds one WARN (the client’s fault) or ERROR (the server’s) naming the cause, with the same stream ID. Library logging (DHT, relay, host) is controlled separately by --debug-dht and GOLOG_LOG_LEVEL.

# Requests answered with a 4xx or 5xx, from the .deb install's log
grep 'msg=request' /var/log/ricochet/stdout.log | grep -E 'status=[45][0-9]{2}'