Amazon SQS (Simple Queue Service) is a fully managed message queue service that allows you to decouple and scale microservices, distributed systems, and serverless applications. SQS offers two types of polling mechanisms for retrieving messages from a queue:
- Short Polling
- Long Polling
Short Polling vs. Long Polling in SQS
Here’s a comparison of the two polling methods:
Short Polling:
How It Works:
- Short Polling is the default polling method for SQS. In this approach, when you request messages from a queue, SQS will return immediately, even if there are no messages in the queue.
- SQS checks a small subset of the queue’s servers (based on the internal partitioning of the queue) to see if there are any messages to retrieve. If there are no messages on those servers, SQS immediately returns an empty response.
Characteristics:
- Instant Response: Returns immediately, even if no messages are available.
- Higher Costs: If you repeatedly poll and receive empty responses, you might incur unnecessary requests and cost.
- Not Ideal for Low Latency: If there’s no message in the queue, it leads to wasted requests since you’ll get an empty response quickly.
Use Case:
- Short polling can be useful if you don’t mind making frequent polling requests and can handle receiving empty responses. It’s good when you need to get messages quickly and don’t mind dealing with the possibility of empty responses.
Long Polling:
How It Works:
- Long Polling is a more efficient alternative to short polling, where SQS waits for a message to become available in the queue before responding.
- With long polling, instead of immediately returning an empty response when no messages are found, SQS holds the request open for a specified amount of time (up to 20 seconds) while it waits for a message to arrive in the queue.
- Once a message is available or the maximum wait time has elapsed, SQS will respond with the message (if available) or an empty response.
Characteristics:
- Reduced Polling Costs: You only pay for the requests that return messages, as opposed to making frequent empty requests with short polling.
- Lower Latency: Reduces the need for multiple requests, leading to fewer empty responses and lower latencies when messages are available.
- More Efficient: Ideal for reducing the number of empty responses and associated costs, as the connection is kept open until either a message is available or the wait time expires.
Use Case:
- Long polling is ideal for applications where you want to reduce the frequency of empty responses and want to efficiently retrieve messages when they become available. It’s a better choice for use cases like background processing, event-driven architectures, or when you’re handling intermittent workloads.
Comparison Table: Short Polling vs. Long Polling
| Feature | Short Polling | Long Polling |
|---|---|---|
| Polling Method | Polls immediately, returns quickly even with no messages. | Waits up to 20 seconds for messages, reduces empty responses. |
| Cost | Can result in unnecessary costs due to frequent empty responses. | More cost-effective because it reduces empty requests. |
| Latency | Potentially higher latency due to multiple empty requests. | Lower latency for message retrieval, reduces waiting time. |
| Efficiency | Less efficient because of repeated empty responses. | More efficient by reducing the number of empty requests. |
| Timeout | No wait time; responds immediately. | Waits up to 20 seconds (configurable). |
| Use Case | Good for quick, frequent checks, but can result in unnecessary polling. | Ideal for low-latency applications, reduces unnecessary costs. |
| Default Setting | Default polling method for SQS. | Can be enabled by setting WaitTimeSeconds in the SQS receive request. |
Visibility Timeout
The Visibility Timeout is a key setting in Amazon Simple Queue Service (SQS) that defines the amount of time a message remains hidden from other consumers after it has been received by a consumer. This prevents multiple consumers from processing the same message simultaneously.
1. How Visibility Timeout Works
- Message Received: When a consumer (e.g., EC2 instance, Lambda function) receives a message from the SQS queue, the message becomes invisible to other consumers for the duration of the Visibility Timeout period.
- Processing Time: During this timeout, the message is being processed, and no other consumer can pick it up until the timeout expires.
- Message Deletion or Visibility Reset:
- If processed successfully, the consumer deletes the message from the queue.
- If processing fails or times out, the message becomes visible again after the timeout period ends, allowing another consumer to attempt processing.
2. Visibility Timeout Settings
- Default Timeout: The default visibility timeout for a new SQS queue is 30 seconds.
- Maximum Timeout: The maximum allowed visibility timeout is 12 hours.
- Adjustable Timeout: You can configure the visibility timeout for each queue, and you can also override the default for individual message retrieval operations.