Amazon Kinesis Data Firehose buffer size determines how much data Firehose collects before delivering it to the destination. This affects performance, latency, and cost.
🔹 Key Parameters for Buffering
Kinesis Data Firehose buffers data using two settings:
- Buffer Size (in MB)
- Buffer Interval (in seconds)
Firehose delivers data when either the buffer size is met or the buffer interval expires.
| Destination | Buffer Size Range | Buffer Interval Range |
|---|---|---|
| Amazon S3 | 1 MB – 128 MB | 60 – 900 sec |
| Amazon Redshift | 1 MB – 128 MB | 60 – 900 sec |
| Amazon OpenSearch | 1 MB – 100 MB | 60 – 900 sec |
| Amazon HTTP Endpoint | 1 MB – 64 MB | 60 – 900 sec |
🔹 How Buffering Works
- Lower Buffer Size = Faster Delivery (good for real-time streaming but increases cost)
- Higher Buffer Size = Lower Costs (batching reduces requests but adds delay)
- If the buffer interval reaches the limit before the buffer size is filled, Firehose flushes the data.