AWS 19 - AWS Intergration & Messaging SQS, SNS & Kinesis

Integration & Messaging between services in AWS

• Synchronous between applications can be problematic if there are sudden spikes of traffic

• What if you need to suddenly encode 1000 videos but usually it’s 10?

• In that case, it’s better to decouple your applications,

• using SQS: queue model • using SNS: pub/sub model

• using Kinesis: real-time streaming model

• These services can scale independently from our application!

AWS SQS

The best way to think of SQS is as a highly durable, scalable to-do list for your distributed applications.

• Core Function: It provides a hosted queue for storing messages (tasks, jobs, data) as they travel between different parts of your application.

• Decoupling: SQS is the ultimate tool for decoupling. Instead of one service calling another directly (synchronously), the first service just drops a message into the SQS queue and moves on. The second service picks up the message when it's ready.

• Scalability & Resilience:

  • If a service suddenly gets swamped (like that $1000$ video-encode spike), the messages pile up safely in the queue. The processing service can then scale up (via Auto Scaling) to empty the queue.

  • If the processing service completely fails, the messages remain safely stored in SQS across multiple AWS servers, waiting to be picked up later, guaranteeing no data loss.

• Fully Managed: AWS handles all the hard parts of traditional message systems, including hardware provisioning, maintenance, and scaling—it's essentially limitless.

SQS Queue Types:

1. Standard Queue (Default): Offers maximum throughput and guarantees at least once message delivery (duplicates are possible but rare) and best-effort ordering.

2. FIFO Queue (First-In, First-Out): Guarantees that messages are processed exactly once and in the exact order they were sent. This is crucial for things like financial transactions where sequence is non-negotiable.

SQS - Standard Queue - Producing messages

Standard Queue (Default): Offers maximum throughput and guarantees at least once message delivery (duplicates are possible but rare) and best-effort ordering.

• Oldest offering (over 10 years old)

• Fully managed service, used to decouple applications

• Attributes: • Unlimited throughput, unlimited number of messages in queue

• Default retention of messages: 4 days, maximum of 14 days

• Low latency (<10 ms o n publish and receive) • Limitation of 256KB per message sent

• Can have duplicate messages (at least once delivery, occasionally)

• Can have out of order messages (best effort ordering)

SQS - Standard Queue - Consuming messages & Multiple EC2 Instances Consumers

• Consumers receive and process messages in parallel: Instead of one single machine being overwhelmed, multiple EC2 instances (the "consumers") are constantly pulling messages from the queue. This is the core principle of horizontal scaling for processing tasks.

• At least once delivery: Since the queue is designed for massive scale and speed, it can't guarantee a message will be delivered exactly once. It promises at least one delivery, meaning that under rare conditions (like a consumer failing just before it deletes a message), the same message might be received and processed twice.

SQS with Auto-Scaling Group (ASG)

1. The SQS Queue (The Waiting Line):

• The SQS Queue receives incoming tasks (messages) from the producer application.

• The EC2 Instances within the ASG constantly poll the queue, grabbing and processing these messages.

• CloudWatch Metric (The Measurement):

  • AWS CloudWatch tracks the ApproximateNumberOfMessages in the SQS Queue. This is the key performance indicator (KPI) that tells you how much work is currently waiting.

2. CloudWatch Alarm (The Trigger):

• A CloudWatch Alarm is set to monitor this metric.

• If the number of waiting messages crosses a certain threshold (a "breach," such as more than 100 messages waiting for five minutes), the alarm is triggered.

3. Auto Scaling Group (The Workforce Manager):

• The CloudWatch Alarm sends a "scale" signal directly to the Auto Scaling Group.

• The ASG immediately launches new EC2 Instances (more consumers) to help empty the queue faster. When the queue length drops back down, the ASG will terminate the excess instances to save costs.

SQS Queue Access Policies

While AWS IAM Roles and User Policies handle access for principals within your own account, SQS Queue Policies are essential for controlling cross-account and cross-service communication.

Policy Use Case	Explanation	Real-World Example
Cross-Account Access	Allows an AWS account different from the one the queue is in to send or receive messages.	A development account sends test messages to a QA account's queue.
Cross-Service Access	Allows another AWS service (like SNS or EventBridge) to push messages into your SQS queue.	You must use a Queue Policy to allow an Amazon SNS Topic to publish notifications to your SQS queue.
IP-Based Filtering	Restricts which IP addresses or VPC endpoints are allowed to interact with the queue.	Limits message consumption only to EC2 instances running within a specific VPC for security.
Denial (Least Privilege)	Explicitly denies an action to certain principals, even if they might have a broad Allow from another IAM policy.	Prevent a specific testing role from accidentally deleting messages (SQS:DeleteMessage) from a production queue.

In most microservice architectures, you will use a Queue Policy to grant permission to an SNS Topic to write messages to the queue. Without t his specific policy, the service-to-service communication will fail.

Message Visibility Timeout

The Message Visibility Timeout is a temporary invisibility setting applied to a message once it has been retrieved by a consumer. Its main job is to prevent duplicate processing.

. If a message is not processed within the visibility timeout, it will be processed twice

. A consumer could call the ChangeMessageVisibility API to get more time

. If visibility timeout is high (hours), and consumer crashes, re-processing will take time

. If visibility timeout is too low (seconds), we may get duplicates

AWS SQS - Dead Letter Queue (DLQ)

• If a consumer fails to process a message within the Visibility Timeout, the message goes back to the queue!

. We can set a threshold of how many times a message can go back to the queue

. After the MaximumReceives threshold is exceeded, the message goes into a dead letter queue (DLQ)

· Useful for debugging!

. DLQ of a FIFO queue must also be a FIFO queue

. DLQ of a Standard queue must also be a Standard queue

• Make sure to process the messages in the DLQ before they expire:

  • Good to set a retention of 14 days in the DLQ

AWS SQS - Delay Queue

An SQS Delay Queue (or the DelaySeconds setting) is used to intentionally postpone when a message is first made available to a consumer.

Common Use Cases for Delay Queues:

• Rate Limiting: If an external API you call can only handle 10 requests per second, you can set a delay on your SQS queue to throttle the rate at which your consumers receive tasks.

• Initial Consistency: In distributed systems, you might need a brief delay to ensure a newly written record in a database is fully replicated before a consumer tries to read it.

• Batch Processing Warm-up: Delaying the start of a large processing job until a specific time or until related resources are spun up.

AWS SQS - Long polling & SQS Extended Client

SQS Extended Client:

• It's a Library, Not a Service: It's an open-source client library for languages like Java and Python, not an AWS service you provision. It acts as an abstraction layer over the standard SQS and S3 APIs.

• The S3 Dance: The library handles the entire lifecycle: uploading the large file to S3 on the producer side, and downloading and deleting it from S3 on the consumer side (once the SQS message is successfully deleted).

• Why Use It? It lets developers send large media, complex documents, or large log files via SQS without having to write the boilerplate code for managing S3 upload/download/cleanup themselves.

AWS SQS - Must know API

SQS - Must know API

. CreateQueue (MessageRetentionPeriod), DeleteQueue

. PurgeQueue: delete all the messages in queue

· SendMessage (DelaySeconds), ReceiveMessage, DeleteMessage

. MaxNumberOfMessages: default I, max 10 (for ReceiveMessage API)

. ReceiveMessageWaitTimeSeconds: Long Polling

. ChangeMessageVisibility: change the message timeout

. Batch APIs for SendMessage, DeleteMessage, ChangeMessageVisibility helps decrease your costs

AWS SQS - FIFO Queue

Unlike Standard queue which sends messages to the consumers randomly

AWS SQS - FIFO Queues Advanced - Deduplication

Deduplication is the process of eliminating redundant or identical copies of data.

In the context of AWS messaging services, particularly Amazon SQS, it has a very specific meaning:

Context	Deduplication Goal	How it Works
General Storage (e.g., S3)	Saves space and cost by ensuring only one unique copy of a file or data block is stored.	If 10,000 users upload the exact same image file, the storage system only keeps one copy and points all users to it.
SQS (FIFO Queues)	Ensures a message is processed Exactly-Once.	The message producer provides a unique Deduplication ID (or the content itself is used). If a second message with the exact same ID is sent within a 5-minute window, SQS rejects the duplicate message.

Why SQS Needs Deduplication?

The SQS Standard Queue guarantees "At Least Once Delivery," meaning you might occasionally get duplicate messages (for instance, if a consumer receives a message but crashes before deleting it).

The SQS FIFO Queue uses the Deduplication feature to solve this problem, ensuring the message is processed only once, which is vital for use cases like financial transactions and order processing.

AWS SQS - FIFO Queues Advanced - Message Grouping

SQS FIFO Message Grouping Explained

Think of it like a meticulous, digital post office that only delivers letters in the exact order they arrived... unless you've marked them with a special group ID.

• The Golden Rule (for strict ordering):

  • If you set the MessageGroupID to the same value for every message in the queue, then all messages are processed in strict FIFO order, but you can only have one consumer working on the queue. Slow, but perfectly sequential!

• The Clever Hack (for parallel, yet ordered, processing):

  • To speed things up while keeping some order, you use different MessageGroupID values.

  • In-Group Order: Messages sharing the same MessageGroupID (like A1, A2, A3) will be processed in order, one after the other. It's their own mini-FIFO line.

  • Parallel Processing: Since each unique MessageGroupID forms its own independent sequence, multiple consumers can process different groups simultaneously (e.g., Consumer "A" processes group A while Consumer "B" processes group B). * The Catch: Ordering across different groups is NOT guaranteed. Group "A" might finish before Group "C" even if the first message in "C" arrived earlier. You only get ordering within a specific group.

In short: The MessageGroupID is your ticket to a bit of parallel processing without completely losing your mind (or your required sequence of events).

AWS SNS

SNS's Deep Integration with AWS Services (Fan-out) so many AWS services can send data directly to SNS for notifications

The main reason SNS is so integrated is that its subscribers can be many different things simultaneously. A single message published to an SNS Topic can be instantly fanned out to all of the following destinations:

Subscriber Type	Integration Use Case
Amazon SQS Queue	Decoupling & Durability: This is the most common pair! SNS broadcasts the message, and SQS catches it. This is essential for reliability, as SQS keeps the message safe until a consumer is ready to process it.
AWS Lambda	Real-Time Processing: SNS can directly trigger a Lambda function, which runs your custom code immediately upon receiving the message.
HTTP/S Endpoint	External Communication: Sends the notification directly to an external server or an application hosted outside of AWS (via a webhook).
AWS EventBridge	Advanced Routing: SNS can publish to EventBridge for more complex filtering and routing to many other services.
AWS CloudWatch	Alerts & Alarms: CloudWatch often uses an SNS Topic as the final action when a metric threshold (like high CPU) is breached, sending alerts to administrators.
End Users (A2P)	Direct Notification: Messages can be sent directly to end-users via Email or SMS/Mobile Push Notifications.

AWS SNS - How to public

Topic Publish (Using the AWS SDK)

This is the standard, flexible method used for Application-to-Application (A2A) messaging and general notifications.

• Create a topic: You first establish a named central channel (the "topic") within the SNS service (e.g., Order_Confirmation_Events).

• Create a subscription (or many): You register one or more recipients to listen to that topic. These subscribers can be SQS queues, Lambda functions, email addresses, or HTTP endpoints.

• Publish to the topic: A publisher application (like your EC2 instance) sends a single message to the topic's name. SNS then automatically fans out that message to all registered subscribers

Direct Publish (For Mobile Apps SDK)

This method is specifically designed for sending notifications directly to a single mobile device or application, which falls under Application-to-Person (A2P) messaging.

• Create a platform application: You register your mobile platform (e.g., Apple Push Notification Service (APNS) for iOS, Google Cloud Messaging (GCM) for Android) with SNS.

• Create a platform endpoint: For each user's device, SNS creates a unique, encrypted address (an endpoint ARN) that represents that specific device on that platform.

• Publish to the platform endpoint: You send a message directly to that unique device endpoint ARN.

• Works with Google GCM, Apple APNS, Amazon ADM...: SNS translates your message into the specific format required by the respective platform (Apple, Google, etc.) and ensures the notification reaches the target device.

SNS + SQS: Fan out pattern

Why This Combination Works

This hybrid push/pull pattern solves the major problems of distributed systems:

• Decoupling & Parallelism: A single "Order Placed" event published to the SNS Topic is instantly replicated into the ShippingQueue, the AnalyticsQueue, and the BillingQueue. All three services can start working on their tasks at the exact same time.

• Durability (No Data Loss): Since the message lands in SQS, it is stored redundantly until the specific service's consumer successfully processes it. If the Billing Service is down, its queue simply stores the message, guaranteeing no critical event is lost.

• Independent Scaling: The Billing service can scale its consumers to handle 10 messages per second, while the Analytics service can scale to handle 100 messages per second, all using the same original event without affecting each other.

• Message Filtering: You can attach a Subscription Filter Policy to an SQS queue so that it only receives a subset of the messages from the Topic (e.g., the Premium_Orders_Queue only receives messages with the attribute order_type: "premium").

SNS + SQS: Fan out pattern examples and architecture solutions

AWS SNS - Message Filtering

SNS Message Filtering: Customized Broadcast

The filtering is defined by a Subscription Filter Policy, which is a JSON document attached to the subscription (not the Topic itself).

How It Works

When a message is published to the SNS Topic, the service does the following for every single subscriber:

1. Check the Policy: SNS compares the incoming message's content against the JSON filter policy attached to that specific subscription.

2. Filter/Deliver:

   • Match: If the message meets all conditions of the policy, SNS pushes the message to the subscriber (e.g., the SQS Queue).

   • No Match: If the message does not match, SNS filters it out and the subscriber never receives it.

Two Main Types of Filtering

Type	How the Filter Reads the Message	Common Use Case
Attribute-Based	The filter checks Message Attributes—small metadata key-value pairs that are sent alongside the main message body.	Filtering by categories: {"order_type": ["digital", "physical"]}, {"region": ["EU", "US"]}.
Payload-Based	The filter checks specific fields within the main JSON body of the message itself.	Filtering based on document content, such as a field in an S3 event notification: {"s3_object_key": [{"prefix": "invoices/"}]}.

Example Use Case: Order Processing Fan-out

Imagine a single ORDER_EVENTS SNS Topic:

• Subscriber A (Email Service): Only needs messages where status: ["shipped", "delivered"]. The filter rejects all messages about payments or cancellations.

• Subscriber B (Inventory Service): Only needs messages where type: ["physical_goods"]. The filter rejects messages about digital downloads.

AWS Kinesis Data Streams (KDS)

KDS is used to collect and store streaming data in real-time 

Features of KDS:

· Retention between up to 365 days

· Ability to reprocess (replay) data by consumers

. Data can't be deleted from Kinesis (until it expires)

. Data up to IMB (typical use case is lot of "small" real-time data)

. Data ordering guarantee for data with the same "Partition ID"

. At-rest KMS encryption, in-flight HTTPS encryption

. Kinesis Producer Library (KPL) to write an optimized producer application

. Kinesis Client Library (KCL) to write an optimized consumer application

AWS KDS - Capacity Modes

In Amazon Kinesis Data Streams, a shard is the fundamental unit of capacity and parallelism. Each shard handles a portion of the data stream's read and write throughput.

What Is a Shard?

• A shard is a uniquely identified sequence of data records within a Kinesis stream.

• Each stream is composed of one or more shards, and each shard acts like a container for incoming data records.

• Shards enable parallel processing of data, allowing multiple consumers to read from different shards simultaneously.

📌 Why Shards Matter

• Shards determine how much data your stream can handle and how fast consumers can process it.

• They are crucial for scaling, performance tuning, and cost management in Kinesis Data Streams.

AWS Data Firehose

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to various destinations without requiring custom applications or infrastructure management.

Here are the key details:

. Note: used to be called "Kinesis Data Firehose"

. Fully Managed Service

. Amazon Redshift / Amazon S3 / Amazon OpenSearch Service

. 3rd party: Splunk / MongoDB / Datadog / NewRelic / ...

. Custom HTTP Endpoint

. Automatic scaling, serverless, pay for what you use

. Near Real-Time with buffering capability based on size / time

. Supports CSV,JSON, Parquet, Avro, Raw Text, Binary data

. Conversions to Parquet / ORC, compressions with gzip / snappy

. Custom data transformations using AWS Lambda (ex: CSV to JSON)

Amazon Manager Services

Amazon Managed Services (AMS) is a suite of operational support offerings from AWS that helps enterprises manage and operate their cloud infrastructure more efficiently and securely.

Here’s a detailed breakdown of what AMS provides:

• AMS helps enterprises operate AWS infrastructure by automating tasks like provisioning, monitoring, and patching.

• It enforces best practices for security, compliance, and operational efficiency.

• AMS provides 24/7 incident response and change management support.

• It assists with cost optimization and scaling cloud operations.

• Designed for large organizations needing robust governance and support.

SQS vs SNS vs Kinesis