Message Queues are a way to transfer information between two systems. This information – a message – can be data, metadata, signals, or a combination of all three. The systems that are communicating can be processes on the same computer, modules of the same application, services that might but running on different computers or technology stacks, or different kinds of systems altogether – like transferring information form your software into an email or an SMS on the global cellphone network.
The idea of a messaging system has been around a very long time, from the message boxes1 used for moving information between people or office departments, to telegrams, to the Pony Express. The messaging system in the physical world that comes closest to what we have in computing is probably the pnuematic tubes2 that moved messages through buildings and cities using compressed air until a few decades ago (and are still used in some places today).
These messages we want transferred today might be a note that something happened, like CPU usage exceeding a limit; or an event of interest, like a customer placing an order; or a signal, like a command that tells another service to do something. The contents will be driven entirely by the architecture of your application and its purposes – so for the rest of this book, we don't need to be concerned about what the contents of a message are – we're more concerned with how the message gets from the system where it originates (the producer or source) to the system where's it's supposed to go (the consumer, destination or receiver).
We need message queues because no system exists or works in isolation – all systems need to communicate with other systems in structured ways that they can both understand, and in controlled message quantities that they can both handle. Any work that requires a process needs a way to move information between each stage of the process; any workflow needs a way to move the intermediate product between the stages of that workflow. Message queues are a great way to do this. There are plenty of ways of getting these messages around using API calls, file systems, or many other abuses of the natural order of things; but all of these are ad-hoc implementations of the message queue that we often refuse to acknowledge we need.
The simplest mental model for a message queue is a tube that you can roll a ball into. You write your message on a ball, roll it into the tube, and someone or something else receives it at the other end. There are a lot of interesting benefits with this model, which we'll see the next chapter.
Literally from where we get the words ‘inbox’ and ‘outbox’, used to designate the boxes for messages that were coming in for processing and going out after. ↑