🤔 Difference between Message queue, broker and Celery

In the journey of developing systems, Microservice Architecture is widely adapted as the system scales. In general the idea is to have multiple small applications running as services performing their tasks and interacting with each other by some means of communication.

There are majorly 2 type of communication types: Synchronous and Asynchronous. Synchronous communication is where client calls the server and waits for request to get the response. Like HTTP(s) or gRPC protocol, client send a request and wait for server to send response. Asynchronous communication is where client sends a request and doesn’t waits for response to come, like it is done in messaging protocols.

Both Synchronous and Asynchronous Communication have their own use cases. In a case where next set of execution is depended on current request’s response, Synchronous Communication is used. Like a HTTP Request to get the details of a User.

However, in some use cases, we need to do some tasks which can happen at their own pace. For example, If a user places an order on an E-commerce website, there are some tasks like sending sms, email confirmation of the order which do not need to happen synchronously and can happen at their own pace. In case we do it synchronously, it makes the overall user experience bad as sending email or sms might take sometime and the user needs to wait till that time before getting any response.

Messages is the one of many ways by which the services can interact with each other. And for a beginner python developer who has just begun with celery for asynchronous communication, might get confused when introduced to messaging.

Now I will explain subtle difference between a message queue, message broker and Celery.

Message queue:

Message queue is just a container that stores the message sent by the sender or publisher (in bi-directional queues). The queue does not have its own intelligence, this is used by the broker to store the message.

Message data can be stored in-memory, or in file, or in database, whatever suits the broker.

Message broker:

A message broker is a software/application which has the intelligence to read/understand messages and send from sender to receiver. A broker can contain multiple queues which it will manage. Broker facilitates communication between multiple servers and allows them to communicate asynchronously.

Some popular message brokers are RabbitMQ, Apache Kafka, and Apache ActiveMQ.

Celery:

Celery is a task queue, that is specifically design for handling tasks which needs to run in background like delayed tasks, sending an email, pdf generation etc. Celery provides an easy way to execute tasks asynchronously.

Celery can work with message broker to leverage its messaging capabilities for building distributed systems with asynchronous. Like RabbitMQ can be used as message broker for sending and receiving tasks for Celery, which will provide scalability, fault tolerance and flexibility in handling background tasks in Python application.

Co-author: @pdhruvil29