MassTransit and RabbitMQ

Not every application has aspirations to serve millions of requests a day, but to those who aspire to do so, technology has become so easy and friendly that there hasn’t been a better time to go web scale. There are usually two high level solutions to the problem of scaling:

  1. Scale Vertically.
  2. Scale Horizontally

Scaling Vertically

If you’ve worked in technology for any amount of time you’ve probably heard phrases like:

  • Memory is cheap.
  • Throw more money at the problem.
  • Disk space is cheap.
  • SSDs are soooo fast.
  • We need more power Jim!

These phrases are hints that you are about to scale vertically. Your application is most likely hosted on one server, and very little about your application is distributed.

Scaling vertically usually focuses on your host environment gaining more memory, disk space, processing power, or all three at the same time. The problem with scaling vertically, is your are constrained by Moore’s Law. If you continue to grow, you are likely to max out your hardware specifications.

Scaling Horizontally

Scaling horizontally usually starts with introspection, both personally and technologically: Do I really need to do this? Can my processing be distributed. It might also involve a lot of cursing or mentions of Fudge.

Scaling horizontally is a distribution problem. How can you relay your intentions to other machines? I’m here to to tell you that scaling horizontally is a problem of two. If you can distribute your process across two machines, then you can distribute your process across 100 machines. The second machine is the hardest.

This tutorial is to show you how easy it is to distribute your work across many machines using the following technologies: .NET, MassTransit, and RabbitMQ.

Getting Started

The first thing you will need, is to install RabbitMQ on your machine / server. You only need one instance of RabbitMQ for this tutorial, but in production it might be a good idea to load balance your installations.

Download the installer here, you will also need Erlang.

Publisher

First install MassTransit into your solution, with the RabbitMQ add-on.

PM> Install-Package MassTransit.RabbitMQ

So for the purposes of this tutorial, we will be using MassTransit to handle sending messages to our queue, and on MassTransit to consume the messages from the queues. So there are two parts to keep in mind:

  1. Publishers
  2. Consumers

To publish a message to RabbitMQ we first need to setup a Bus. This of the Bus as a service that facilitates communication across your network. Let’s set it up.

Bus.Initialize(cfg => {
    cfg.UseRabbitMq();
    cfg.ReceiveFrom("rabbitmq://localhost/test_queue");
});

Here is the quirky part that might not be intuitive, but each logical process needs its own separate queue. In this case our publisher has a queue of rabbitmq://localhost/test_queue. Let’s see how we can publish the message using the Bus.

Bus.Instance.Publish(new DoWorkItem { Text = string.Format("Message #{0} says: Hello World at {1}!", i, DateTime.Now) })

We’ve published a message! Now the message is out in our queueing system waiting to be picked up. Now to implement the other half.

Consumer

Consumers handle messages (very descriptive name). The process to consume the messages might be a separate application, and in this example it is, so we need to initialize another instance of the Bus.

Bus.Initialize(sbc =>
        {
            sbc.UseRabbitMq();
            sbc.ReceiveFrom("rabbitmq://localhost/test_queue_worker");
            sbc.Subscribe(subs =>
            {
                var del = new Action<IConsumeContext<DoWorkItem>, DoWorkItem>((context, msg) =>
                {
                    Console.WriteLine(msg.Text);
                });
                subs.Handler(del);
            });
        });

Notice a couple things about our implementation of the Bus this time.

  1. We have a different queue
  2. We are not subscribing to the DoWorkItem message.

So you remember how I said each logical process has to have its own queue, well now we can spawn up as many of the consumers as we want. The magic happens in RabbitMQ, which implements a competing consumer model. That means one message per one process at a time. Since all your consumers will be sharing the same queue, they will happily round robin responsibility. This means you can scale to hundreds, if not thousands of processors with no perceived differences in implementation. Let me show you a screenshot of how this all works.

w734

Awesome right? We have three consumers being delegated work from one publisher. You can try this out by cloning my GitHub repository.

https://github.com/algraps/RabbitMQ/tree/master/RabbitMQ.Scaling

Note: You will need NuGet package restore enabled. It helps to start the publisher and consumers at the same time, but that is optional as messages that are published will be persisted in the queue waiting for someone to consume them.

Conclusion

Web Scale is achievable with some OSS goodness and some practice. It really is an amazing time to be a developer. Give my code a try to get started, and let me know if you have any questions.

Lascia un commento

Inserisci i tuoi dati qui sotto o clicca su un'icona per effettuare l'accesso:

Logo WordPress.com

Stai commentando usando il tuo account WordPress.com. Chiudi sessione / Modifica )

Foto Twitter

Stai commentando usando il tuo account Twitter. Chiudi sessione / Modifica )

Foto di Facebook

Stai commentando usando il tuo account Facebook. Chiudi sessione / Modifica )

Google+ photo

Stai commentando usando il tuo account Google+. Chiudi sessione / Modifica )

Connessione a %s...