Getting Started with NServiceBus Part 9: Horizontal Scaling with the Distributor

A single thread NServiceBus instance can handle quite of bit of messages. We're talking around 30 messages/second or roughly 2.5 million messages a day. It may be the case that if you need more throughput you can ramp up the number of threads NSB uses. If you would rather not do that and keep it single threaded but still scale out you can do so with the built-in Distributor process.  Check out the first slide at the end of the post for the general message flow.

Fundamentally the Distributor is comprised of at least 3 processes. The first is the Distributor process itself which holds onto the entire set of work that the Workers will operate upon. The reason the Distributor holds the work is so that if one of the Workers goes down, it can shift that work over to the other Worker(s). The other 2 processes are Worker processes that do the actual work and also report their status back to the Distributor. At its core the Distributor receives work from some other process and then dishes out the work based on the current status of the Workers.

The workflow is controlled by a few internal queues to the Distributor itself. The Distributor has a Control, Storage, and Data(work) queue. The Control queue is where the Workers will put their Ready messages. The Storage queue keeps track of the addresses of the Workers and is used to determine which Worker is available next for work. The Data queue keeps the set of work. All of these settings are kept in the app.config of the Distributor process.

Once the Distributor is configured we must tell each Worker how to communicate with the Distributor.

Once we have everything configured the workflow is a multi-step process that goes like this:

1. A Worker starts up and places a Ready message on the Distributor's Control queue.  This message is read and pushes a message on the Storage queue to indicate that we have an available Worker.
2. Step 1 is repeated for all Worker that start up.
3. A Client issuing the work sends a message to the Data queue.
4. The message is read from the Data queue and the Distributor pops the next available Worker from the Storage queue
5. The message is then forwarded to the Worker and the Worker handles the message.
6. Once the Worker is done it sends a Ready message back to the Control queue to signify that it is ready for more work.
7. If the first Worker is still working as more work comes in, the next available Worker gets the next message.  If all Workers are busy, then the Distributor queues up the messages in the Data queue.

Check out the second slide in the deck at the end of the post to have a visual of the flow.

The Worker message Handlers are written just like any old NSB handler.  The only other missing component is that you will need to send the Distributor some work.  When you fire up all the processes you can send messages to the Distributor and watch the work load balance.  Since the Distributor holds all the work you may want to make the Distributor highly available and put it into a cluster.

I have a full working sample on github: https://github.com/afyles/Blog/tree/master/NServiceBusExplorer/. All you need to do is fire up the Distributor and the 2 Workers in the Workers folder under the Source directory.

NServiceBus Distributor

View more presentations from afyles.