Scaling and MQTT

One kraken node needs nothing else. When one node is not enough, there are two independent steps:

1. Cluster kraken nodes, so a message published on one node reaches subscribers on the others, and presence and connection limits are shared.
2. Move fan-out to an external MQTT broker (BROKER_BACKEND=mqtt), so delivery between nodes goes through a broker built for it.

Clustering

kraken nodes form a cluster over Erlang distribution. The default broker, syn, spreads its topic groups across the cluster, and so do presence, lobbies and the per-organization connection counts. Any node can take any connection; a load balancer in front only has to support WebSockets.

Every node needs:

The nodes must reach each other on port 4369 (the Erlang port mapper) and on ports 9100 to 9200, the distribution range kraken's release is configured with. Keep those ports on a private network.

epmd: a fixed list of nodes

Each node is given the full names of all the nodes, and connects to them, retrying every 30 seconds (CLUSTER_POLL_INTERVAL):

environment:
  - ERLANG_NODE_NAME=kraken@kraken1.cluster.local
  - ERLANG_COOKIE=replace-with-a-long-random-value
  - CLUSTER_STRATEGY=epmd
  - CLUSTER_HOSTS=kraken@kraken1.cluster.local,kraken@kraken2.cluster.local,kraken@kraken3.cluster.local

The kraken repository includes a three-node example built this way:

docker compose -f docker-compose.cluster.yml up -d --build

It publishes the nodes on ports 8081, 8082 and 8083. We connected a subscriber to ws://localhost:8083/ws and a publisher to ws://localhost:8081/ws, and the message crossed between nodes. Stop it with docker compose -f docker-compose.cluster.yml down.

dns: peers from DNS A records

Each node looks up a DNS name, and for every IP address it returns, connects to the node named <CLUSTER_NODE_BASENAME>@<ip>. This suits a Kubernetes headless service, or any network alias shared by the nodes. Each node's own name must follow the same pattern, so it has to know its IP address at startup:

We ran two nodes this way with Docker Compose, sharing the network alias kraken-peers, and they joined each other and passed messages between nodes. The image is kraken built locally with docker build -t kraken:0.9.0 . in the kraken checkout:

services:
  kn1:
    image: kraken:0.9.0
    environment:
      - CLUSTER_STRATEGY=dns
      - CLUSTER_DNS_QUERY=kraken-peers
      - CLUSTER_NODE_BASENAME=kraken
      - ERLANG_COOKIE=replace-with-a-long-random-value
    command: ["sh", "-c", "export ERLANG_NODE_NAME=kraken@$$(hostname -i); exec bin/kraken foreground"]
    networks:
      net:
        aliases: [kraken-peers]
  # kn2: the same again
networks:
  net: {}

A node looks its peers up when it starts and every CLUSTER_POLL_INTERVAL (30 seconds) after that, so a node that starts later is found within half a minute.

kraken's docs/CONFIG.md mentions CLUSTER_DNS_NAME for this strategy. v0.9.0 does not read it; use CLUSTER_DNS_QUERY.

gossip does not work in v0.9.0

CLUSTER_STRATEGY=gossip is accepted and opens a UDP multicast socket, but the node never announces itself or listens for others. Two nodes started this way stayed apart: neither saw the other, and a message published on one did not reach a subscriber on the other. Use epmd or dns.

Checking a cluster

Each node logs [ClusterManager] Node joined: '...' when it connects to a peer. To ask a running node directly:

docker exec <container> bin/kraken eval "nodes()."

An empty list ([]) means the node is on its own.

The MQTT broker backend

syn is a starter broker by design: it delivers inside one Erlang cluster, and on every publish it scans the cluster's subscription groups for wildcard matches, which is fine at modest scale. For more, point the broker slot at an external MQTT broker such as EMQX, Mosquitto or VerneMQ. kraken then publishes and subscribes on that broker, which does the fan-out. The wire protocol and the SDKs do not change.

The kraken repository includes an example with Mosquitto:

docker compose -f docker-compose.mqtt.yml up -d --build

kraken listens on ws://localhost:18081/ws. We ran the Quick Start clients against it, changing only the port in the URL, and they behaved the same. Stop it with docker compose -f docker-compose.mqtt.yml down.

What changes with the MQTT backend:

• QoS takes effect on the hop between kraken and the broker. The syn broker ignores QoS entirely. Neither extends QoS to the WebSocket between kraken and the client. See Quality of Service.
• Retained messages are kept by the external broker, under its own rules, instead of in kraken's memory for an hour.
• Persistent sessions. When the auth backend asks for one (@nolag/core does for agent and orchestrator actors), kraken opens an MQTT 5 session with clean start off and the given expiry, so the broker must support MQTT 5. What the broker holds for a disconnected session is then up to the broker.

What does not change: presence, lobbies and connection counting still run on kraken's own cluster. kraken nodes that share a broker but are not clustered deliver messages to each other through the broker, but each keeps its own presence. Cluster them as well if presence must span nodes.

Not to confuse with MQTT ingress

The MQTT broker backend is kraken acting as an MQTT client of your broker. kraken also has an MQTT listener on port 1883, meant for devices to connect to kraken directly. In v0.9.0 that listener accepts no connections; see Configuration.