Modern applications are being built to take advantage of serverless technologies. One key component of serverless architectures are services that allow for asynchronous orchestration between microservices. “Messaging Services” play a critical role to allow distributed applications to communicate with each other in order to improve their resilience, scalability, and availability of applications.
In this post, I am going to talk about the key differences between two extremely important popular messaging Services: Simple Notificaiton Service (SNS) and Simple Queue Service (SQS).
Despite having very similar names, these services provide two very different functions, and most folks tend to get confused and wonder when its appropriate to use one over the other.
In this post, I am going to introduce you to both SNS and SQS, and run through a practical example to demonstrate the differences between the two.
Let’s get started.
If you prefer to watch a video version of this blog post, check out my YouTube video below on this topic.
What is Amazon SNS?
SNS stands for Simple Notification Service – it is a cloud service offered by AWS. It is a fully managed distributed publish-subscribe system, also sometimes referred to as shorthand pub-sub. It offers a push notification service that lets us send individual messages or bulk messages to a large set of subscribing clients or applications. The core functionality of Amazon SNS is to push a message to multiple subscribers as soon as it received it from a publisher.
To get started with Amazon SNS, developers first have to create a topic that acts as an access point for subscribers interested in receiving notifications about a specific subject. Whenever there is an update for subscribers, developers (or applications) publish a message to the topic, and this move prompts Amazon SNS to distribute a copy of the message to all its subscribers simultaneously.
It makes it simple and super cost-effective to send push notifications to mobile users across distributed devices such as Android, Apple, Windows, email recipients and other internet-connected devices. It also supports several endpoints such as email, SMS, HTTP endpoint and SQS.
Wish to learn about Amazon SNS in depth? Then do check out this playlist
What is Amazon SQS?
SQS stands for Simple Queue Service, another cloud service offered by AWS. It is a fully managed message queuing service that offers a secure, durable and available hosted queue, and it lets us integrate and decouple distributed software systems and components. Unlike SNS, messages are not pushed to receivers. Instead, receivers need to Poll SQS to receive messages.
The main purpose of an SQS queue is to allow for consumer applications to poll messages off the queue at their preferred rate. This can be important for applications that have certain client dependencies that support a certain threshold for API calls. SQS helps you slow down your processing so that you can stay below those levels.
Polling results in some latency in the delivery of the message system in SQS. Anyone thread in an application can receive, process, and delete messages. When a thread picks up a message, it is claimed, and becomes invisible to other threads attempting to poll messages until either the message is successfully processed (and subsequently deleted), or when a configurable timeout period elapses.
Amazon SQS operates on both Standard and FIFO Queue principles; standard queues offer maximum throughput and guarantee at-least-once delivery, FIFO queues are designed to guarantee that messages are processed exactly once, in the same order they are sent. FIFO is a useful property for many applications, you can learn more about FIFO queues here.
Wish to learn about Amazon SQS in depth then check out this playlist
SNS VS SQS- Tabular Comparison of Amazon SNS & SQS
AMAZON SNS | AMAZON SQS |
Publisher role in PubSub | Queuing service for message processing |
Publishing messages can deliver to many subscribers (fan-out) of different types (SQS, Lambda, Email) | A application must poll to discover new events, and messages are typically processed by a single consumer application. |
All the consumers can be of different types. | All the consumers are supposed to be of identical types |
It involves a push mechanism with entities such as topics and broadcasts. | It involves a pull mechanism using polling with entities such as a Queue. |
A Practical Example
So we’ve run through what each of these services do, lets use a practical example to describe how these services are different.
In this example, we’re going to be talking about credit card transaction processing system. Initially, a user will make a purchase on some website or through some POS terminal that communicates with TransactionProcessingWebService.
Now they’re making a purchase request to some kind of REST API, and we’re just calling this a transaction processing web service, and the payload of it may looks something like this-
Invoke:
You need to first communicate with some kind of credit card authority service, practically speaking, probably Visa, Mastercard, Amex, etc. So when an event occurs saying that a transaction is taking place, that should get validated against the authority to ensure the transaction is authenticated.
When step 1 is successfully passed, then there might be relevant subsystems that care about when new transactions take place. In other words, they want to get notified when a new transaction occurs in this application ecosystem. In this example, we have three different sub-systems that potentially care about this notification:
- CustomerReminderService (A Lambda Function)
- TransactionAnalyticsQueue (A SQS Queue)
- TransactionFraudDetectionQueue (Also a SQS Queue)
Use Case of Lambda (Email Reminders)
When a message gets delivered, maybe we have some kind of customer reminder service, and if you’ve ever ordered anything online, maybe on Amazon or any kind of web site you know that when you order something shortly after that, you get an email that says something like “hey thank you for ordering, we really appreciate your interest and here’s a summary of the purchase”.
This component is potentially responsible for that very handy information your customer receive very often. In the back end, whenever there is a message received from the Transaction SNS topic, the lambda function will be invoked with a copy of the message that was delivered as an input argument to the function. This function may have code in it to turn around and query some database for some additional details and then sends an email to your customer automatically.
There are two other subsystems here that care about these same events, but they do very different things, which conflate two other responsibilities. Lets examine them now.
Use case of 1st SQS system of the example (Analytics)
This system with SQS as queue can be some kind of analytics engine. So when an event occurs, in addition to publishing to the lambda function, we’re publishing that same event at some analytics queue. This system may be one of its responsibilities is it cares about how many orders are generated in a single day, or maybe you want to show that on some dashboard.
So when an event occurs, this payload gets delivered to this queue. It’s not going to really do anything; you need another system here to poll the queue to actually become aware that something is in there and respond to it. In this example, we configured some transaction analytic services hosted on ec2 and part of spinning up your service. It will have a series of threads that know how to pull messages off of this queue, and when there is a message that is present, it can increment the count of the number of orders per day.
Use case of 2nd SQS system of the example (Fraud Detection)
The other SQS is a queue to identify the fraudulent transactions that may occur in the e-commerce ecosystem. Think of this like a scenario where you’re concerned with fraud transactions because certain people have ordered something in the past, and the transaction eventually got reversed, and now you want to have a robust, proactive approach that attempts to detect and mitigate fraud before you ship your product.
Similar to the above, here too we may have some fraud detection service hosted on ec2. Again we’re going to poll the queue to figure out when events occur, and once we process the message, we delete that message from the queue so no other threads can receive it.
At the end of the day, this system tries to detect fraudulent transactions. It helps businesses identify the right orders to be delivered against valid transactions. It results in saving a lot of time, money, and effort eventually. This is where the usage of SNS and SQS shines because events are published and distributed to all the consumers accordingly.
It’s obvious that you may get a doubt that ” why should we couple all the services here and why you should not process each and every operation in a sequence directly?” Please watch this video where I briefly explained the answer to the same question.
Conclusion
I hope by now you got clear about the usage of Amazon SQS and Amazon SNS, that is when you have to publish messages and deliver them to systems that want to be notified of an event, then we need to go for SNS.
If you want a delivery point for this queue where a relevant sub-system can pick it up and process it later, you can use a service like SQS.
One thing to note is that we don’t need to couple SNS and SQS in all cases. We can have SNS send messages to email, SMS or HTTP endpoint apart from SQS. Coupling of SNS and SQS is a best practice when we desire to publish a message to one or more consumers, with sQS being one of them.
If you still feel a bit of ambiguity, then consider these statements
- If you want an unknown number and type of subscribers to receive identical copies of messages, you need SNS.
- If you want a simple queue to process these messages at your own rate, then you need SQS.
I believe that I am able to make things clear for you, and on this note, I encourage you to check out our complete playlist on tutorials of SNS and SQS to continue your learning further on these services.
Please do subscribe to our newsletter so that you can get our amazing blog posts directly into your inbox as soon as we publish.