The AppScale implementation uses RabbitMQ as its engine for message passing in a distributed setting. We've found that RabbitMQ is well documented, has a relatively low memory footprint, and is very stable. The setup was simple for a distributed cluster with features such as acknowledgement of messages, high availability, and message durability. 

The figure above shows how tasks are distributed between nodes and application servers. Each machine may have multiple applications servers running either the same app or different apps. Within an application server (right box of the figure) we see there is a separate thread which listens for incoming tasks. When a message is received, this thread will then post to the local load balancer (orange arrow), where it will be balanced across one of the many application servers handing said application. If all application servers are unavailable, then the task will be re-enqueued and its retry number, which is stored in the header, will be incremented. Each time a failure occurs, the amount of time to backoff is a random number of seconds between 0 and 2^n, where n is the number of failures thus far.

Tasks names and state are stored in the datastore. There are three states a task can be in: running, completed, and error. Currently, only a single queue is used and features like rate limiting and task deferment are not implemented, but on the roadmap. Task deferment is not trivial as RabbitMQ does not support it delaying messages. This is the main feature currently lacking but is in the works.

The load balancer in AppScale is based on a combo of Nginx and HAProxy. HAProxy gives us the capability to queue up requests and serve them to the next available application server, while also doing health checks. Nginx gives the capability to do SSL and static file serving (spares the application server to handle only dynamic requests). 

The RabbitMQ Server runs on each node configured in a cluster. Any client that listens in on a particular queue will receive a message in a round robin fashion. If a client has a task and it fails, RabbitMQ will automatically distribute it to another client, providing fault tolerance. Our testing shows this works as promised. Thus far we are very happy with how RabbitMQ has been performing and encourage you to try it out for your message passing needs.