When processing my Amazon Kinesis data stream, my AWS Lambda function receives fewer records per invocation than the batch size that I configured in the Kinesis event source. Why is that?
The actual batch size your Lambda function processes is determined by several factors:
- The number of records received from the GetRecords call that Lambda makes when polling your event source.
- The user-configured batch size, which is the largest number of records that are read from your stream at once. For more information, see Creating an Event Source Mapping.
- The number of records that can fit within the Lambda invocation payload size limit of 6 MB. A larger record size means that fewer records can fit in the payload. For more information on limits, see AWS Lambda Limits.
The amount of data in the stream also plays a part. That is, the Lambda function doesn't wait until it has more records before invoking. The records are pulled at the normal rate, and the function processes the available records. If the traffic on the stream is low, then the batch size is low as well.
Estimate the approximate batch size with this formula:
6000 KB / record size in KB
For example, if each record in your batch is 64 KB, then you should expect to see about 90 records per batch, even if you configured a batch size of 1000 records.