Scaling Websockets with Message Brokers
Some thoughts on how WebSockets can be used with message brokers.
If you don’t know how WebSockets work, check out my previous article.
Done? Let’s explore more about WebSockets!
When we talk about scalability; that too in the context of WebSockets, a server should be able to do:
- Maintain many active connections
- Send many messages to clients
- Support WebSocket fallback to scale to every client
- Authenticate incoming connections and invalidate connections
- Survive massive reconnect of all clients without losing messages
Here are some tips:
- Increase max open file descriptors: Every connection will cost you an open file descriptor, so you should tune a maximum number of open file descriptors your process can use. Make sure to limit your maximum number; it should always be lesser than the known file descriptor limit.
- Reduce Payload Size: WebSockets generally have lower overhead than TCP/IP. This article (Dissecting WebSocket Overhead) states that application-level message batching can be used to effectively reduce the overhead that is induced by TCP/IP.
While the above points look interesting, keep in mind that we might need to scale connections over different machines. However, we also need to find a way to deliver a message to a certain user. We can publish messages to all server instances, but this does not scale well.
This is where the Pub/Sub architecture comes into play; where you can connect WebSocket server instances over a central Pub/Sub broker. Clients that establish connections with your WebSocket server subscribe to topics (channels) in a broker, and as soon as you publish a message to that topic, it will be delivered to all active subscribers on WebSocket server instances.
The choice of message brokers are as follows:
But why do we need message brokers for scaling Websockets? Let’s see an example:
skribbl.io is a multiplayer drawing game. Let’s say we want to make something similar to skribbl.io: users can draw something, other users get real-time visualization of the drawing, etc.
If you think the client-server-database architecture would be sufficient, you’re probably moving in the wrong direction. Here’s why:
- WebSocket on Server 1 writes some data to the database
- How would WebSocket on Server 2 know when should it fetch the latest data from the database?
Adding Message Brokers
Now let’s add a message broker to our architecture:
When a user draws something, the drawing is saved in the database with an unique ID, and then, an event/message is published to the message broker based on the unique ID for the drawing.
Since the event has been published to the message broker, interested clients (which in our case will be the users in the same room) can fetch the latest changes from the database and update their local state (new drawing coordinates, etc.)
As you can see, this architecture is much more scalable than the previous one.
Now, what happens if a client gets disconnected during the game?
The drawings made during the period when the client was disconnected are saved in the message broker. When the client reconnects, it can check the latest events based on the timestamps, and fetch the latest changes from the database.
WebSockets are powerful. Combining message brokers with them makes them more powerful, as message brokers:
- Are scalable.
- Maintain message order in topics
- Can support millions of topics
- Support time-based retention and other features!
I hope this post motivates you to explore more about using message brokers like Kafka, RabbitMQ, etc. with WebSockets! 😄