Admission & rate limits

This page explains the controls that keep the server from being overloaded, which of them actually stop abuse, and how to tune them.

Identity is free, so per-peer limits are fairness controls

A peer identity is an Ed25519 key, and minting one costs nothing. Every limit keyed by peer ID (per-peer admission slots, per-peer rate limits, mailbox quotas) bounds what one well-behaved client can use. It does not bound what a determined client can use, because that client can rotate identities.

The controls a client cannot rotate out of are:

performance.max_concurrent_connections: a hard cap on open connections, for the whole server.
admission_control: a bound on work in flight against the database, however many identities ask.
performance.max_connections_per_ip: the one limit tied to a source address. It is off by default.

Treat the per-peer settings as fairness between clients, and the server-wide ones as protection.

Admission control

Admission control governs throughput. It bounds how many requests are in flight at once, not how many a client may send per minute. Work is admitted as fast as it completes, so throughput rises with your hardware and falls on its own when the database slows. A batch write holds its slot for as long as it runs, so expensive work costs more capacity than cheap work without a cost model.

admission_control:
  enabled: true
  max_in_flight: 0            # 0: derive from database.pool_size
  max_in_flight_per_peer: 64  # 0: no per-peer bound
  acquire_timeout: 5s

Key	Default	Meaning
`enabled`	`true`	Off removes the bound entirely. The server logs a warning at startup.
`max_in_flight`	`0`	Requests in flight server-wide. Zero derives it as 4 × `database.pool_size`: 100 at the default pool of 25, 200 with the production preset’s 50.
`max_in_flight_per_peer`	`64`	Requests in flight from one peer. Stops one client from holding every slot.
`acquire_timeout`	`5s`	How long a request waits for a slot before it is shed.

A request first waits for one of its peer’s slots, then for a server-wide slot. Waiting is normal: it is how a fast client is paced to the database. A request that gets no slot within acquire_timeout is shed with status 503 and a retry hint (a quarter of the timeout, between 50 ms and 1 s). The server logs request not admitted with the peer, the error and the current in-flight count.

A 503 from admission is a signal to add capacity (a pool connection, a replica, an instance), not a verdict on the client.

Watch it

Metric	Meaning
`ricochet_requests_in_flight`	Requests holding a slot now
`ricochet_admission_max_in_flight`	The effective server-wide bound
`ricochet_admission_waited_total`	Requests that had to wait for a server-wide slot
`ricochet_admission_shed_total`	Requests shed after `acquire_timeout`
`ricochet_requests_total{outcome="overloaded"}`	Shed requests, by protocol and operation

/readyz carries the same figures under details.admission. A busy server stays ready: saturation does not make /readyz fail, because pulling a busy instance out of rotation moves its load onto the busiest remaining ones.

Connection caps

performance:
  max_concurrent_connections: 10000
  max_connections_per_ip: 0

max_concurrent_connections is enforced by the libp2p resource manager. A connection manager trims idle peers once the count passes 90% of the cap, down to 80%. A new connection is safe from trimming for its first 30 seconds. The presets set 100 (development), 10,000 (default and production) or 50,000 (high-capacity).

max_connections_per_ip caps connections from one source address: an IPv4 address or an IPv6 /56. The default, 0, is no cap. Phones behind a carrier NAT and desks behind an office gateway share an address, and any cap turns the ones past it away. go-libp2p normally applies its own per-address limits (8 connections and 0.2 new connections a second); this server replaces them with no per-address cap and no per-address rate limit. Set max_connections_per_ip only where one address is known to be one client. Loopback is never capped.

Timeouts

Key	Default	Meaning
`performance.connection_timeout_sec`	`30`	How long a request or response may stall, with no bytes moving, before the stream is dropped. It does not limit how long a large transfer takes.
`performance.message_timeout_sec`	`10`	Deadline for the work a request does after it has been read: storage calls and the handler. A database that stops answering releases the request here.
`performance.worker_threads`	`4`	Push notifications in flight at once. Notifications past this during a burst are dropped; clients poll regardless.

Per-peer rate limits

Rate limits are off by default and are not the throughput mechanism. A rate limit is a guessed constant: set low, it caps a well-behaved client long before the hardware is busy; set high, it protects nothing. A saturated server needs to shed whatever arrives, which admission control already does.

Use rate limits for what a capacity bound does not cover: per-tenant billing caps, untrusted peers, or a client known to be looping.

rate_limiting:
  window: 1m
  protocols:
    msa:       { requests: { rate: 6000, burst: 12000 } }   # message submission
    msa_batch: { requests: { rate: 600,  burst: 1200 } }    # batch submission
    maa:       { requests: { rate: 6000, burst: 12000 } }   # message access
    mma:       { requests: { rate: 600,  burst: 1200 } }    # mailbox administration
    sda:                                                    # documents
      read:  { rate: 6000, burst: 12000 }
      write: { rate: 6000, burst: 12000 }

Every limit is per peer. Each bucket refills rate requests per window and lets an idle peer save up to burst to spend at once. A burst of 0 defaults to rate.
msa, msa_batch, maa, mma and mta take one requests bucket. sda, sfa and sca take separate read and write buckets.
mta is the routing backstop, charged per message.
A rate of -1 disables a bucket, and every bucket is disabled by default. A rate of 0 means “not set”, so it does not disable anything.
Limits count requests, not items: one batch write costs the same as one single write.
A protocol name the server does not recognise stops it at startup.
The legacy keys window_minutes (an alias for window) and max_requests_per_window (feeds only the mta backstop) are still honoured.

A peer over its limit gets status 429, counted as ricochet_requests_total{outcome="rate_limited"}. Keep the two outcomes apart when you read the metrics: rate_limited means a peer exceeded a budget you set, and the fix is the budget; overloaded means the server is full, and the fix is capacity.

Check what took effect

curl -s http://127.0.0.1:9090/ops/limits | jq '{admission, rateLimits}'

The response shows the effective maxInFlight, whether it was derived from the pool size, and each rate-limit bucket with an explicit enabled field. See Metrics & the ops view.

Troubleshooting

Clients see 503 “server at capacity” while the database is quiet: the pool and the derived bound are too small. Raise database.pool_size, or set max_in_flight explicitly.
One client sees 503 and others do not: it hit max_in_flight_per_peer. That is the bound working.
Clients see 429: a rate limit is on. Check /ops/limits for enabledProtocols.
Many clients from one office or carrier fail to connect: max_connections_per_ip is set. Set it back to 0.