[{"data":1,"prerenderedAt":756},["ShallowReactive",2],{"docs:/docs/self-hosting/scaling":3},{"id":4,"title":5,"body":6,"description":748,"extension":749,"meta":750,"navigation":751,"path":752,"seo":753,"stem":754,"__hash__":755},"docs/docs/self-hosting/scaling.md","Scaling and MQTT",{"type":7,"value":8,"toc":738},"minimark",[9,13,17,39,44,51,54,130,133,138,145,201,204,236,251,255,262,323,334,480,486,499,503,513,517,524,558,565,569,574,628,631,653,669,672,712,715,719,734],[10,11,5],"h1",{"id":12},"scaling-and-mqtt",[14,15,16],"p",{},"One kraken node needs nothing else. When one node is not enough, there are two independent steps:",[18,19,20,28],"ol",{},[21,22,23,27],"li",{},[24,25,26],"strong",{},"Cluster kraken nodes",", so a message published on one node reaches subscribers on the others, and presence and connection limits are shared.",[21,29,30,33,34,38],{},[24,31,32],{},"Move fan-out to an external MQTT broker"," (",[35,36,37],"code",{},"BROKER_BACKEND=mqtt","), so delivery between nodes goes through a broker built for it.",[40,41,43],"h2",{"id":42},"clustering","Clustering",[14,45,46,47,50],{},"kraken nodes form a cluster over Erlang distribution. The default broker, ",[35,48,49],{},"syn",", spreads its topic groups across the cluster, and so do presence, lobbies and the per-organization connection counts. Any node can take any connection; a load balancer in front only has to support WebSockets.",[14,52,53],{},"Every node needs:",[55,56,57,70],"table",{},[58,59,60],"thead",{},[61,62,63,67],"tr",{},[64,65,66],"th",{},"Setting",[64,68,69],{},"Value",[71,72,73,104,114],"tbody",{},[61,74,75,81],{},[76,77,78],"td",{},[35,79,80],{},"ERLANG_NODE_NAME",[76,82,83,84,87,88,91,92,95,96,99,100,103],{},"A unique full name, ",[35,85,86],{},"name@host",". kraken uses long names, so ",[35,89,90],{},"host"," must be a fully qualified domain name or an IP address. ",[35,93,94],{},"kraken@kraken1"," does not work; ",[35,97,98],{},"kraken@kraken1.cluster.local"," or ",[35,101,102],{},"kraken@10.0.0.5"," does.",[61,105,106,111],{},[76,107,108],{},[35,109,110],{},"ERLANG_COOKIE",[76,112,113],{},"The same value on every node. It is the only thing that keeps other Erlang nodes out, so make it long and random.",[61,115,116,121],{},[76,117,118],{},[35,119,120],{},"CLUSTER_STRATEGY",[76,122,123,124,99,127],{},"How the nodes find each other: ",[35,125,126],{},"epmd",[35,128,129],{},"dns",[14,131,132],{},"The nodes must reach each other on port 4369 (the Erlang port mapper) and on ports 9100 to 9200, the distribution range kraken's release is configured with. Keep those ports on a private network.",[134,135,137],"h3",{"id":136},"epmd-a-fixed-list-of-nodes","epmd: a fixed list of nodes",[14,139,140,141,144],{},"Each node is given the full names of all the nodes, and connects to them, retrying every 30 seconds (",[35,142,143],{},"CLUSTER_POLL_INTERVAL","):",[146,147,152],"pre",{"className":148,"code":149,"language":150,"meta":151,"style":151},"language-yaml shiki shiki-themes github-light github-dark","environment:\n  - ERLANG_NODE_NAME=kraken@kraken1.cluster.local\n  - ERLANG_COOKIE=replace-with-a-long-random-value\n  - CLUSTER_STRATEGY=epmd\n  - CLUSTER_HOSTS=kraken@kraken1.cluster.local,kraken@kraken2.cluster.local,kraken@kraken3.cluster.local\n","yaml","",[35,153,154,167,177,185,193],{"__ignoreMap":151},[155,156,159,163],"span",{"class":157,"line":158},"line",1,[155,160,162],{"class":161},"s9eBZ","environment",[155,164,166],{"class":165},"sVt8B",":\n",[155,168,170,173],{"class":157,"line":169},2,[155,171,172],{"class":165},"  - ",[155,174,176],{"class":175},"sZZnC","ERLANG_NODE_NAME=kraken@kraken1.cluster.local\n",[155,178,180,182],{"class":157,"line":179},3,[155,181,172],{"class":165},[155,183,184],{"class":175},"ERLANG_COOKIE=replace-with-a-long-random-value\n",[155,186,188,190],{"class":157,"line":187},4,[155,189,172],{"class":165},[155,191,192],{"class":175},"CLUSTER_STRATEGY=epmd\n",[155,194,196,198],{"class":157,"line":195},5,[155,197,172],{"class":165},[155,199,200],{"class":175},"CLUSTER_HOSTS=kraken@kraken1.cluster.local,kraken@kraken2.cluster.local,kraken@kraken3.cluster.local\n",[14,202,203],{},"The kraken repository includes a three-node example built this way:",[146,205,209],{"className":206,"code":207,"language":208,"meta":151,"style":151},"language-bash shiki shiki-themes github-light github-dark","docker compose -f docker-compose.cluster.yml up -d --build\n","bash",[35,210,211],{"__ignoreMap":151},[155,212,213,217,220,224,227,230,233],{"class":157,"line":158},[155,214,216],{"class":215},"sScJk","docker",[155,218,219],{"class":175}," compose",[155,221,223],{"class":222},"sj4cs"," -f",[155,225,226],{"class":175}," docker-compose.cluster.yml",[155,228,229],{"class":175}," up",[155,231,232],{"class":222}," -d",[155,234,235],{"class":222}," --build\n",[14,237,238,239,242,243,246,247,250],{},"It publishes the nodes on ports 8081, 8082 and 8083. We connected a subscriber to ",[35,240,241],{},"ws://localhost:8083/ws"," and a publisher to ",[35,244,245],{},"ws://localhost:8081/ws",", and the message crossed between nodes. Stop it with ",[35,248,249],{},"docker compose -f docker-compose.cluster.yml down",".",[134,252,254],{"id":253},"dns-peers-from-dns-a-records","dns: peers from DNS A records",[14,256,257,258,261],{},"Each node looks up a DNS name, and for every IP address it returns, connects to the node named ",[35,259,260],{},"\u003CCLUSTER_NODE_BASENAME>@\u003Cip>",". This suits a Kubernetes headless service, or any network alias shared by the nodes. Each node's own name must follow the same pattern, so it has to know its IP address at startup:",[55,263,264,272],{},[58,265,266],{},[61,267,268,270],{},[64,269,66],{},[64,271,69],{},[71,273,274,284,294,312],{},[61,275,276,280],{},[76,277,278],{},[35,279,120],{},[76,281,282],{},[35,283,129],{},[61,285,286,291],{},[76,287,288],{},[35,289,290],{},"CLUSTER_DNS_QUERY",[76,292,293],{},"The name to look up",[61,295,296,301],{},[76,297,298],{},[35,299,300],{},"CLUSTER_NODE_BASENAME",[76,302,303,304,307,308,311],{},"The part before ",[35,305,306],{},"@",". The default is ",[35,309,310],{},"kraken_proxy",", so set it to match your node names.",[61,313,314,318],{},[76,315,316],{},[35,317,80],{},[76,319,320],{},[35,321,322],{},"\u003CCLUSTER_NODE_BASENAME>@\u003Cthis node's IP>",[14,324,325,326,329,330,333],{},"We ran two nodes this way with Docker Compose, sharing the network alias ",[35,327,328],{},"kraken-peers",", and they joined each other and passed messages between nodes. The image is kraken built locally with ",[35,331,332],{},"docker build -t kraken:0.9.0 ."," in the kraken checkout:",[146,335,337],{"className":148,"code":336,"language":150,"meta":151,"style":151},"services:\n  kn1:\n    image: kraken:0.9.0\n    environment:\n      - CLUSTER_STRATEGY=dns\n      - CLUSTER_DNS_QUERY=kraken-peers\n      - CLUSTER_NODE_BASENAME=kraken\n      - ERLANG_COOKIE=replace-with-a-long-random-value\n    command: [\"sh\", \"-c\", \"export ERLANG_NODE_NAME=kraken@$$(hostname -i); exec bin/kraken foreground\"]\n    networks:\n      net:\n        aliases: [kraken-peers]\n  # kn2: the same again\nnetworks:\n  net: {}\n",[35,338,339,346,353,364,371,379,387,395,402,428,436,444,456,463,471],{"__ignoreMap":151},[155,340,341,344],{"class":157,"line":158},[155,342,343],{"class":161},"services",[155,345,166],{"class":165},[155,347,348,351],{"class":157,"line":169},[155,349,350],{"class":161},"  kn1",[155,352,166],{"class":165},[155,354,355,358,361],{"class":157,"line":179},[155,356,357],{"class":161},"    image",[155,359,360],{"class":165},": ",[155,362,363],{"class":175},"kraken:0.9.0\n",[155,365,366,369],{"class":157,"line":187},[155,367,368],{"class":161},"    environment",[155,370,166],{"class":165},[155,372,373,376],{"class":157,"line":195},[155,374,375],{"class":165},"      - ",[155,377,378],{"class":175},"CLUSTER_STRATEGY=dns\n",[155,380,382,384],{"class":157,"line":381},6,[155,383,375],{"class":165},[155,385,386],{"class":175},"CLUSTER_DNS_QUERY=kraken-peers\n",[155,388,390,392],{"class":157,"line":389},7,[155,391,375],{"class":165},[155,393,394],{"class":175},"CLUSTER_NODE_BASENAME=kraken\n",[155,396,398,400],{"class":157,"line":397},8,[155,399,375],{"class":165},[155,401,184],{"class":175},[155,403,405,408,411,414,417,420,422,425],{"class":157,"line":404},9,[155,406,407],{"class":161},"    command",[155,409,410],{"class":165},": [",[155,412,413],{"class":175},"\"sh\"",[155,415,416],{"class":165},", ",[155,418,419],{"class":175},"\"-c\"",[155,421,416],{"class":165},[155,423,424],{"class":175},"\"export ERLANG_NODE_NAME=kraken@$$(hostname -i); exec bin/kraken foreground\"",[155,426,427],{"class":165},"]\n",[155,429,431,434],{"class":157,"line":430},10,[155,432,433],{"class":161},"    networks",[155,435,166],{"class":165},[155,437,439,442],{"class":157,"line":438},11,[155,440,441],{"class":161},"      net",[155,443,166],{"class":165},[155,445,447,450,452,454],{"class":157,"line":446},12,[155,448,449],{"class":161},"        aliases",[155,451,410],{"class":165},[155,453,328],{"class":175},[155,455,427],{"class":165},[155,457,459],{"class":157,"line":458},13,[155,460,462],{"class":461},"sJ8bj","  # kn2: the same again\n",[155,464,466,469],{"class":157,"line":465},14,[155,467,468],{"class":161},"networks",[155,470,166],{"class":165},[155,472,474,477],{"class":157,"line":473},15,[155,475,476],{"class":161},"  net",[155,478,479],{"class":165},": {}\n",[14,481,482,483,485],{},"A node looks its peers up when it starts and every ",[35,484,143],{}," (30 seconds) after that, so a node that starts later is found within half a minute.",[14,487,488,489,492,493,496,497,250],{},"kraken's ",[35,490,491],{},"docs/CONFIG.md"," mentions ",[35,494,495],{},"CLUSTER_DNS_NAME"," for this strategy. v0.9.0 does not read it; use ",[35,498,290],{},[134,500,502],{"id":501},"gossip-does-not-work-in-v090","gossip does not work in v0.9.0",[14,504,505,508,509,99,511,250],{},[35,506,507],{},"CLUSTER_STRATEGY=gossip"," is accepted and opens a UDP multicast socket, but the node never announces itself or listens for others. Two nodes started this way stayed apart: neither saw the other, and a message published on one did not reach a subscriber on the other. Use ",[35,510,126],{},[35,512,129],{},[134,514,516],{"id":515},"checking-a-cluster","Checking a cluster",[14,518,519,520,523],{},"Each node logs ",[35,521,522],{},"[ClusterManager] Node joined: '...'"," when it connects to a peer. To ask a running node directly:",[146,525,527],{"className":206,"code":526,"language":208,"meta":151,"style":151},"docker exec \u003Ccontainer> bin/kraken eval \"nodes().\"\n",[35,528,529],{"__ignoreMap":151},[155,530,531,533,536,540,543,546,549,552,555],{"class":157,"line":158},[155,532,216],{"class":215},[155,534,535],{"class":175}," exec",[155,537,539],{"class":538},"szBVR"," \u003C",[155,541,542],{"class":175},"containe",[155,544,545],{"class":165},"r",[155,547,548],{"class":538},">",[155,550,551],{"class":175}," bin/kraken",[155,553,554],{"class":175}," eval",[155,556,557],{"class":175}," \"nodes().\"\n",[14,559,560,561,564],{},"An empty list (",[35,562,563],{},"[]",") means the node is on its own.",[40,566,568],{"id":567},"the-mqtt-broker-backend","The MQTT broker backend",[14,570,571,573],{},[35,572,49],{}," is a starter broker by design: it delivers inside one Erlang cluster, and on every publish it scans the cluster's subscription groups for wildcard matches, which is fine at modest scale. For more, point the broker slot at an external MQTT broker such as EMQX, Mosquitto or VerneMQ. kraken then publishes and subscribes on that broker, which does the fan-out. The wire protocol and the SDKs do not change.",[55,575,576,584],{},[58,577,578],{},[61,579,580,582],{},[64,581,66],{},[64,583,69],{},[71,585,586,598,615],{},[61,587,588,593],{},[76,589,590],{},[35,591,592],{},"BROKER_BACKEND",[76,594,595],{},[35,596,597],{},"mqtt",[61,599,600,608],{},[76,601,602,416,605],{},[35,603,604],{},"MQTT_BROKER_HOST",[35,606,607],{},"MQTT_BROKER_PORT",[76,609,610,611,614],{},"The broker's address. The default port is ",[35,612,613],{},"1884",", so set it.",[61,616,617,625],{},[76,618,619,416,622],{},[35,620,621],{},"MQTT_BROKER_USERNAME",[35,623,624],{},"MQTT_BROKER_PASSWORD",[76,626,627],{},"If the broker requires them",[14,629,630],{},"The kraken repository includes an example with Mosquitto:",[146,632,634],{"className":206,"code":633,"language":208,"meta":151,"style":151},"docker compose -f docker-compose.mqtt.yml up -d --build\n",[35,635,636],{"__ignoreMap":151},[155,637,638,640,642,644,647,649,651],{"class":157,"line":158},[155,639,216],{"class":215},[155,641,219],{"class":175},[155,643,223],{"class":222},[155,645,646],{"class":175}," docker-compose.mqtt.yml",[155,648,229],{"class":175},[155,650,232],{"class":222},[155,652,235],{"class":222},[14,654,655,656,659,660,665,666,250],{},"kraken listens on ",[35,657,658],{},"ws://localhost:18081/ws",". We ran the ",[661,662,664],"a",{"href":663},"/docs/getting-started","Quick Start"," clients against it, changing only the port in the URL, and they behaved the same. Stop it with ",[35,667,668],{},"docker compose -f docker-compose.mqtt.yml down",[14,670,671],{},"What changes with the MQTT backend:",[673,674,675,688,694],"ul",{},[21,676,677,680,681,683,684,250],{},[24,678,679],{},"QoS takes effect"," on the hop between kraken and the broker. The ",[35,682,49],{}," broker ignores QoS entirely. Neither extends QoS to the WebSocket between kraken and the client. See ",[661,685,687],{"href":686},"/docs/concepts/qos","Quality of Service",[21,689,690,693],{},[24,691,692],{},"Retained messages"," are kept by the external broker, under its own rules, instead of in kraken's memory for an hour.",[21,695,696,699,700,703,704,707,708,711],{},[24,697,698],{},"Persistent sessions."," When the auth backend asks for one (",[35,701,702],{},"@nolag/core"," does for ",[35,705,706],{},"agent"," and ",[35,709,710],{},"orchestrator"," actors), kraken opens an MQTT 5 session with clean start off and the given expiry, so the broker must support MQTT 5. What the broker holds for a disconnected session is then up to the broker.",[14,713,714],{},"What does not change: presence, lobbies and connection counting still run on kraken's own cluster. kraken nodes that share a broker but are not clustered deliver messages to each other through the broker, but each keeps its own presence. Cluster them as well if presence must span nodes.",[40,716,718],{"id":717},"not-to-confuse-with-mqtt-ingress","Not to confuse with MQTT ingress",[14,720,721,722,725,726,729,730,250],{},"The MQTT broker backend is kraken acting as an MQTT ",[24,723,724],{},"client"," of your broker. kraken also has an MQTT ",[24,727,728],{},"listener"," on port 1883, meant for devices to connect to kraken directly. In v0.9.0 that listener accepts no connections; see ",[661,731,733],{"href":732},"/docs/self-hosting/configuration","Configuration",[735,736,737],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":151,"searchDepth":169,"depth":169,"links":739},[740,746,747],{"id":42,"depth":169,"text":43,"children":741},[742,743,744,745],{"id":136,"depth":179,"text":137},{"id":253,"depth":179,"text":254},{"id":501,"depth":179,"text":502},{"id":515,"depth":179,"text":516},{"id":567,"depth":169,"text":568},{"id":717,"depth":169,"text":718},"Run several kraken nodes as one cluster over Erlang distribution (epmd or DNS discovery), or hand message fan-out to an external MQTT broker such as EMQX or Mosquitto.","md",{},true,"/docs/self-hosting/scaling",{"title":5,"description":748},"docs/self-hosting/scaling","RR7tL9aR5KI4oGv2IVj0-sEhW4HAzAmt37MdbO8j5Xw",1791536075168]