| Feature | Standard Queue | FIFO Queue |
|---|---|---|
| Throughput | Unlimited | 300 TPS (default) 3,000 TPS (with high-throughput mode) |
| Message per second | Virtually unlimited | Limited to configured TPS |
| Ordering | Best-effort (out of order possible) | Strictly ordered |
| Duplicates | Possible (at-least-once delivery) | No duplicates (exactly-once processing) |
| Latency | Tens of milliseconds | Slightly higher due to ordering enforcement |
| Message Size | Up to 256 KB per message | Up to 256 KB per message |
| Message Retention | 1 minute to 14 days (default 4 days) | 1 minute to 14 days (default 4 days) |
| Visibility Timeout | 0 seconds to 12 hours | 0 seconds to 12 hours |
| Batching | Supports up to 10 messages per batch | Supports up to 10 messages per batch |
| Deduplication | Not available | Requires a deduplication ID |
| Cost | Lower cost per request | Slightly higher cost due to ordering and deduplication overhead |
Key Limitations
- FIFO queues have a lower throughput cap compared to standard queues.
- FIFO queues require a deduplication ID or content-based deduplication.
- Standard queues do not guarantee exactly-once processing, so duplicates should be handled at the application level.
- Both queues have a maximum message size of 256 KB.
- Visibility timeout must be set appropriately to prevent multiple consumers from processing the same message.
Comparison Table: Short Polling vs. Long Polling
| Feature | Short Polling | Long Polling |
|---|---|---|
| Polling Method | Polls immediately, returns quickly even with no messages. | Waits up to 20 seconds for messages, reduces empty responses. |
| Cost | Can result in unnecessary costs due to frequent empty responses. | More cost-effective because it reduces empty requests. |
| Latency | Potentially higher latency due to multiple empty requests. | Lower latency for message retrieval, reduces waiting time. |
| Efficiency | Less efficient because of repeated empty responses. | More efficient by reducing the number of empty requests. |
| Timeout | No wait time; responds immediately. | Waits up to 20 seconds (configurable). |
| Use Case | Good for quick, frequent checks, but can result in unnecessary polling. | Ideal for low-latency applications, reduces unnecessary costs. |
| Default Setting | Default polling method for SQS. | Can be enabled by setting WaitTimeSeconds in the SQS receive request. |
Visibility Timeout
The Visibility Timeout is a key setting in Amazon Simple Queue Service (SQS) that defines the amount of time a message remains hidden from other consumers after it has been received by a consumer. This prevents multiple consumers from processing the same message simultaneously.
1. How Visibility Timeout Works
- Message Received: When a consumer (e.g., EC2 instance, Lambda function) receives a message from the SQS queue, the message becomes invisible to other consumers for the duration of the Visibility Timeout period.
- Processing Time: During this timeout, the message is being processed, and no other consumer can pick it up until the timeout expires.
- Message Deletion or Visibility Reset:
- If processed successfully, the consumer deletes the message from the queue.
- If processing fails or times out, the message becomes visible again after the timeout period ends, allowing another consumer to attempt processing.
2. Visibility Timeout Settings
- Default Timeout: The default visibility timeout for a new SQS queue is 30 seconds.
- Maximum Timeout: The maximum allowed visibility timeout is 12 hours.
- Adjustable Timeout: You can configure the visibility timeout for each queue, and you can also override the default for individual message retrieval operations.
How Batching Works in Amazon SQS
Batching in Amazon SQS allows you to send, receive, and delete multiple messages in a single API request. This helps reduce costs and improves efficiency.
1. SendMessageBatch (Sending Messages in Bulk)
- Allows sending up to 10 messages in a single API call.
- Each message in the batch must be ≤ 256 KB.
- If any message in the batch fails, only that message fails—the rest are processed.
- Cost-saving: Batching reduces API costs by decreasing the number of requests.
2. ReceiveMessage (Receiving Messages in Bulk)
- Can retrieve up to 10 messages in a single API call.
- Messages are returned only if they are available.
- The batch size can be adjusted using the
MaxNumberOfMessagesparameter (1-10).
3. DeleteMessageBatch (Deleting Messages in Bulk)
- Allows deleting up to 10 messages in a single request.
- Each message is identified by its receipt handle.
- If deletion of a message fails, only that message needs retrying, not the whole batch.
Batching Limitations
| Limitation | Description |
|---|---|
| Max Batch Size | 10 messages per batch |
| Max Message Size | 256 KB per message |
| Partial Failures | Some messages in a batch may fail while others succeed |
| FIFO Queue Consideration | Maintains order within a batch but order across batches is not guaranteed |
Amazon SQS Throttling Limits
Throttling in Amazon SQS occurs when API requests exceed the allowed rate limits. These limits vary based on the queue type (Standard vs. FIFO) and whether high-throughput mode is enabled.
1. Throttling Limits for SQS Queues
| Queue Type | Max Requests per Second (TPS) | High-Throughput Mode | Notes |
|---|---|---|---|
| Standard Queue | Virtually unlimited | N/A | No hard limit, scales automatically |
| FIFO Queue (default) | 300 TPS per API action (send, receive, delete) | No | Limited to 300 TPS |
| FIFO Queue (high-throughput) | 3,000 TPS per API action | Yes | Requires enabling high-throughput mode |
2. What Happens When Throttling Occurs?
If you exceed the allowed API request rate:
🔴 AWS returns a ThrottlingException (HTTP 429).
🔴 Requests may be delayed or dropped.
🔴 Message processing may slow down due to retries.
AWS SQS FIFO Deduplication ID
In AWS SQS FIFO (First-In-First-Out) queues, the Deduplication ID is used to prevent duplicate messages from being processed within a 5-minute deduplication window.
How the Deduplication ID Works
- When you send a message to an SQS FIFO queue, AWS checks its Deduplication ID.
- If another message with the same Deduplication ID was sent within the last 5 minutes, SQS ignores the new message to prevent duplication.
- After 5 minutes, the same Deduplication ID can be used again.
Ways to Set the Deduplication ID
- Manually Specify the Deduplication ID
- You can provide a custom Deduplication ID when sending a message.
- Example: jsonCopyEdit
{ "MessageBody": "Hello, FIFO Queue!", "MessageGroupId": "group-123", "MessageDeduplicationId": "unique-id-456" } - The MessageDeduplicationId must be unique within the 5-minute window.
- Content-Based Deduplication (AWS-Generated ID)
- If you do not specify a Deduplication ID, AWS generates one automatically using an SHA-256 hash of the message body.
- Two messages with the same body will be considered duplicates within the 5-minute window.
AWS SQS FIFO – Message Group ID
In AWS SQS FIFO (First-In-First-Out) queues, the Message Group ID is used to ensure that messages in the same group are processed in order by a single consumer.
How the Message Group ID Works
- Every message in an SQS FIFO queue must have a Message Group ID.
- Messages with the same Message Group ID are processed sequentially by a single consumer.
- Different Message Group IDs can be processed in parallel by multiple consumers.
- A consumer must delete a message before SQS delivers the next message in the same group.