Cluster

Several r3v3rs3 nodes can share one state in etcd or in the key-value store of Consul. Each node serves traffic with the same ports, proxies, access lists, certificates, ACME entries, admin accounts and settings. A change on one node reaches the other nodes without a restart.

This page is the reference. To build a cluster from nothing, follow High Availability: it covers every step from the store to the load balancer.

Architecture

• config.toml of each node holds the [cluster] section. The admin API and the WebUI do not change it, and the store does not hold it.
• The store holds the rest of the state: the settings, the ports, the proxies, the access lists, the certificates, the ACME entries, the admin accounts and the CDN IP ranges. It also holds the audit log and the sent certificate notifications. A node with the cluster on does not read ports.toml, proxies.toml, access_lists.toml, acme.toml, accounts.toml, notifications.json or the certificate files.
• Each node watches the store and applies every change. A change through the admin API of a node goes to the store first. A write that finds a newer value fails with 409 cluster_write_conflict, and the node gets the newer value from the store.
• The nodes share the admin sessions, the proxy sessions, the rate limit counts and, when share_cache is on, the cached responses.
• One node is the leader. Only the leader orders ACME certificates, sends the certificate notifications, removes expired certificates and old audit log entries, saves the downloaded CDN IP ranges to the store and removes the expired sessions and shared responses. The leader runs these tasks when it takes the lead and then at each background_task_interval.
• A load balancer in front of the nodes can send each request to any node.

The Settings page of the WebUI shows the state, the role and the applied revision of the node. GET /api/cluster/status returns the same data.

Set Up a Cluster

Four commands build a cluster: r3v3rs3 cluster keygen writes the encryption key, the [cluster] section of config.toml names the store on every node, r3v3rs3 cluster import copies the files of one node into an empty prefix, and r3v3rs3 start runs every node.

High Availability gives each step with its commands, the store install, the credentials and the load balancer.

Settings

Only config.toml sets these fields.

The admin API does not return password or token.

Keys in the Store

Every key starts with <prefix>/v1/.

Encryption

A node encrypts each value with AES-256-GCM before it writes the value. The KV key of the value is the associated data, so a value that somebody moves to another key does not decrypt. An encrypted value starts with the id of its key: the first 8 bytes of the SHA-256 digest of the key.

To replace a key:

1. Create a new key file with r3v3rs3 cluster keygen.
2. Put the new file first in encryption_key_files and keep the old file in the list. Do this on every node and restart the nodes, because a node reads the key files when it starts.
3. Run r3v3rs3 cluster rekey --config-dir <dir> on one node. The command encrypts every value again with the first key and reports the values that it changed.
4. Remove the old file from encryption_key_files on every node and restart the nodes.

Protect the store too. The encryption hides the values, but the key names show the ids of the ports, proxies and certificates.

etcd Permissions

The user of the nodes needs read and write access to the keys below the prefix:

$ etcdctl role add r3v3rs3
$ etcdctl role grant-permission --prefix=true r3v3rs3 readwrite r3v3rs3/
$ etcdctl user add r3v3rs3
$ etcdctl user grant-role r3v3rs3 r3v3rs3

With another prefix, grant the permission on <prefix>/. The nodes renew the authentication token when etcd revokes it.

Consul Permissions

The token of the nodes needs write access to the keys below the prefix and the permission to create sessions:

key_prefix "r3v3rs3/" {
  policy = "write"
}

session_prefix "" {
  policy = "write"
}

$ consul acl policy create -name r3v3rs3 -rules @r3v3rs3.hcl
$ consul acl token create -description "r3v3rs3 nodes" -policy-name r3v3rs3

The make test-cluster-e2e target runs the nodes with exactly these permissions.

Leader and ACME

The leader holds the lock/leader key with a lease of lock_ttl. It renews the lease three times in each lock_ttl. A store call of the leader waits one third of lock_ttl at most, so the leader gives up the lead before its lease can end in the store. When the leader stops, it releases the lock, and another node takes the lead at its next check.

For an HTTP-01 or TLS-ALPN-01 challenge, the leader writes the challenge to the store. Every node serves it. Each node writes the digest of the challenges that it serves to acks/. The leader waits up to 10 seconds until every present node reports the new challenges, and then asks the ACME server to validate.

Failure Modes

• A node checks the store three times in each lock_ttl. When the node has no successful read for lock_ttl, its state becomes degraded.
• A degraded node serves traffic with the last applied state. The admin API rejects changes with 503 cluster_unavailable, and every page of the WebUI shows a warning.
• The leader gives up the lead when its store calls fail, so a degraded node does not lead.
• A new login and a new proxy session need the store. Without the store, they fail with 503.
• When the store is back, the node reads the whole state again and becomes synced. The node does not write its local state to the store.
• A node that cannot reach the store when it starts stops after startup_timeout.

Sessions

The admin sessions and the sessions of the proxy authentication are in the store, so a session that one node starts is valid on every node. A logout removes the session for every node. A node reads the store for each request that carries a session. Without the store, a proxy session is not valid, and the admin API answers 503 cluster_unavailable.

Rate Limit Accuracy

A node does not read the store for each request. Each node counts the requests of each client and publishes the counts of the 2048 busiest clients of each limit at each rate_limit_sync_interval. A node adds the recent counts of the other nodes to its own counts. Counts that are older than three intervals, or older than 2 seconds when that is longer, do not count.

• A limit whose period is at least 10 times rate_limit_sync_interval uses the counts of every node. The counts of the other nodes are up to one interval old. A burst that reaches every node at the same time can pass up to (N − 1) × burst more requests than the limit, where N is the number of nodes.
• A limit with a shorter period divides the limit between the present nodes. Each node allows ceil(limit / N) requests. A load balancer that sends a client to one node gives the client less than the limit.
• Without the store, each node uses the divided limit with the last known number of nodes.

Shared Cache

With share_cache = true, a node queues each cached response that stays fresh for at least 60 seconds for the store. The node does not wait for the write. On a local miss, a node reads the store for at most 100 milliseconds.

• A response stays on its node when its encoded size is larger than cache_max_value_size, 1 MiB for etcd or 350 KiB for Consul, whichever is smaller. The body is stored in base64, so the stored size is about 4/3 of the body size. The node does not split a response into several keys.
• A purge of the cache of a proxy writes state/cache-purges/<proxy id> and deletes the shared responses of the proxy. Every node purges its local cache after the change.
• A shared response stays in the store for one hour after it becomes stale, so a node can revalidate it. The leader removes the expired responses at each background_task_interval. The removal reads and decrypts every shared response, so its cost grows with the number of stored responses.
• Each stored response is a write to the store. The store keeps every write in its history until a compaction, so a busy cache increases the size of etcd between compactions.

Clocks

The nodes compare Unix times for the session expiry, the rate limit windows, the freshness of the rate limit counts and the expiry of shared responses. Keep the clocks of the nodes in sync with NTP. A clock that is ahead or behind by seconds changes the rate limit decisions and the cache expiry.