Introducing queue consumption mode for Apache Kafka in AWS Lambda

Queue consumption mode for AWS Lambda event source mappings (ESMs) that subscribe to Apache Kafka is now available. Queue consumption mode is available on the self-managed Apache Kafka ESM. It covers both Kafka clusters you operate yourself and managed Kafka offerings such as Confluent Cloud and other providers that offer Kafka 4.2 or later with Kafka Queue support.

With this launch, you can build queue-style event-driven workloads on Kafka that process records with high parallelism that scales concurrency to match your backlog instead of your topic layout. Queue mode achieves this using Kafka Share Groups (KIP-932), introduced in Kafka 4.0. With share groups, multiple consumers read concurrently from the same partitions, so Lambda can process records from a single partition across many parallel invocations. The result is parallelism that is decoupled from partition count, without head-of-line blocking and without introducing a second messaging system into your architecture.

Many Kafka workloads treat each event as an independent unit of work rather than a step in an ordered sequence: generating image and video renditions after upload, offloading slow work from a user-facing API, scoring transactions for fraud, delivering notifications, enriching records for inference, processing bulk files, buffering ahead of a rate-limited dependency, and ingesting telemetry. Correctness for these workloads depends on processing records quickly, not in order. The Kafka consumer group protocol, however, assigns each partition to exactly one consumer, so a topic with 10 partitions supports up to 10 concurrent consumers. Over-partitioning raises that ceiling but increases broker resource usage, slows rebalances, and still leaves head-of-line blocking, where one slow record stalls every record behind it. Previously, teams moved this traffic to a separate queuing system, adding a platform to operate and new producer code.

In this post, you learn how queue consumption mode works compared to the existing stream mode, what broker and cluster requirements your Kafka 4.2 cluster must meet, and how to scale from on-demand defaults to provisioned mode for high-throughput workloads.

What’s new

This launch adds a ConsumptionMode parameter to the Kafka event source mappings in Lambda. It accepts two values: Stream (the default, matching today’s behavior) and Queue, which uses Kafka Share Groups for parallel consumption. The core primitive in both modes is the event poller, a compute resource that fetches records from Kafka and invokes your Lambda function. Queue mode changes the protocol an event poller uses to obtain those records. Because pollers are no longer bound to individual partitions, queue mode supports higher concurrency. A single poller reaches up to 10 concurrent invocations compared to 5 in stream mode, and queue mode scales independently of partition count. We cover the scaling model in detail later in this post.

Stream mode: One consumer per partition, ordered processing

ConsumptionMode defaults to Stream, so existing ESMs are unaffected by this launch. In Stream mode, event pollers join a Kafka consumer group, and the broker assigns each partition exclusively to one consumer. This guarantees ordered processing within each partition.

Queue mode: Multiple consumers per partition, unordered parallel processing

Setting ConsumptionMode to Queue directs Lambda to use the Kafka Share Consumer. Event pollers join a share group, where partitions can be assigned to multiple event pollers for parallel processing without guarantees about order. The broker manages a record lifecycle of Available, Acquired, Acknowledged, and Archived. Fetched records are marked Acquired under a visibility timeout called the record lock duration. A successful invocation acknowledges them. A failed invocation releases them back to Available for any poller to acquire. While an invocation is still running, the poller renews the lock without incrementing the delivery attempt count.

Choosing between the modes

Use Stream mode when per-partition ordering matters: change data capture, event sourcing, or per-entity state machines. Use Queue mode when records are independent, and you want parallelism that is not bounded by partition count. You can run a Queue mode ESM and a Stream mode ESM against the same topic. Each operates independently through its own protocol.

Minimum requirements

Before you configure an ESM with ConsumptionMode: Queue, confirm that your Kafka cluster and Lambda function meet the following requirements.

Cluster requirements

• Apache Kafka 4.2 or later, with share groups enabled.

Four broker settings interact with your ESM configuration:

• group.share.partition.max.record.locks – The maximum in-flight records per partition. In Queue mode, multiple pollers might acquire records from the same partition concurrently, so the total in-flight count can exceed a single poller’s batch size. Set this value high enough to accommodate your expected concurrency. When the limit is reached, pollers can’t acquire new records from that partition until existing records are acknowledged, or their locks expire.
• group.share.record.lock.duration.ms – The visibility timeout for acquired records. In most cases, the default of 30 seconds is sufficient. The poller renews the lock during processing. A long-running function does not typically require a duration longer than 30 seconds.
• group.share.delivery.count.limit – The maximum number of times the broker attempts to deliver a record before archiving it. This works alongside MaximumRetryAttempts to govern retry behavior.
• group.share.max.size – The maximum consumers allowed in the share group. When provisioned mode is active, keep MinimumPollers ≤ MaximumPollers ≤ group.share.max.size so that Lambda scaling isn’t capped by the broker. In on-demand mode, Lambda internally manages poller counts within broker-safe defaults.

Getting started

In the Lambda console, choose your function, choose Add trigger, and select Self-managed Apache Kafka. Enter your bootstrap servers and topic name, set Consumption mode to Queue, and configure your virtual private cloud (VPC) subnets and security groups. Optionally set Minimum event pollers and Maximum event pollers to use provisioned mode. For on-demand scaling leave those empty.

The following AWS CLI command creates the equivalent Queue mode ESM in on-demand mode:

aws lambda create-event-source-mapping
--function-name my-function
--self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["broker1.example.com:9092","broker2.example.com:9092"]}}'
--topics my-queue-topic
--batch-size 100
--self-managed-kafka-event-source-config '{"ConsumptionMode": "Queue"}'

ConsumptionMode is the only parameter required to enable Queue mode. BatchSize sets the maximum records per batch sent to your function, accepts values from 1–10,000, and defaults to 100.

With AWS CloudFormation, the equivalent resource looks as follows:

MyKafkaQueueESM:
  Type: AWS::Lambda::EventSourceMapping
  Properties:
    FunctionName: !Ref MyFunction
    SelfManagedEventSource:
      Endpoints:
        KafkaBootstrapServers:
          - broker1.example.com:9092
          - broker2.example.com:9092
    Topics:
      - my-queue-topic
    BatchSize: 100
    SelfManagedKafkaEventSourceConfig:
      ConsumptionMode: Queue

Queue mode is also configurable through the AWS SDKs, AWS Serverless Application Model (AWS SAM), and AWS Cloud Development Kit (AWS CDK). For the complete setup walkthrough, see the AWS Lambda Developer Guide.

Scaling with on-demand mode

Queue mode ESMs use on-demand mode for scaling by default. Lambda evaluates share group consumer lag at one-minute intervals and adds or removes event pollers as needed, completing scaling actions within three minutes. On-demand mode scales to five event pollers per ESM, each supporting up to 1 MB/s. It reaches its ceiling at either 5 MB/s aggregate throughput or 50 concurrent invocations, whichever comes first. This suits lower-concurrency workloads and development environments.

Scaling with provisioned mode

For workloads with strict latency requirements or spiky traffic, provisioned mode for scaling gives you direct control over the event pollers dedicated to your ESM. Set MinimumPollers from 1 –200 and MaximumPollers from 1 to 2,000. However, the broker’s group.share.max.size setting caps how many members can join a share group, and Kafka 4.2 limits this value to 1,000. Set MaximumPollers to 1,000 or below unless your broker supports a higher limit. Each provisioned event poller supports up to 5 MB/s.

aws lambda create-event-source-mapping
--function-name my-function
--self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["broker1.example.com:9092","broker2.example.com:9092"]}}'
--topics my-queue-topic
--batch-size 100
--self-managed-kafka-event-source-config '{"ConsumptionMode": "Queue"}'
--provisioned-poller-config '{"MinimumPollers": 5, "MaximumPollers": 100}'
--source-access-configurations '[{"Type":"VPC_SUBNET","URI":"subnet-0123456789abcdef0"},{"Type":"VPC_SUBNET","URI":"subnet-0123456789abcdef1"},{"Type":"VPC_SECURITY_GROUP","URI":"sg-0123456789abcdef0"}]'

To size MinimumPollers, divide your baseline ingestion rate by 5 MB/s. If your workload is concurrency-bound, divide your target concurrent invocations by 10 instead. A workload needing 300 concurrent invocations requires at least 30 pollers. Setting MinimumPollers equal to MaximumPollers pins capacity and disables autoscaling. For detailed sizing guidance, see Introducing provisioned mode for Kafka event source mappings with AWS Lambda on the AWS Compute Blog.

Low-latency processing, schema registry filtering for Avro and Protobuf, and ESM grouping are available only in provisioned mode. Provisioned mode also removes the requirement to configure AWS PrivateLink VPC endpoints, so you are not charged for those endpoints. To switch an ESM to on-demand scaling, pass an empty configuration:

aws lambda update-event-source-mapping \
--uuid a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \
--provisioned-poller-config '{}'

Cost optimization with provisioned mode

In provisioned mode, ESMs in the same VPC can share Event Poller Unit (EPU) capacity through a common PollerGroupName, letting low-throughput ESMs pack onto shared EPUs instead of each holding dedicated capacity. For details, see Cost optimization for provisioned mode. A poller group supports up to 100 ESMs and 2,000 aggregate maximum pollers.

Queue mode ESMs share EPUs only with other Queue mode ESMs. The two modes autoscale on different signals, and isolation keeps a new Queue mode ESM from destabilizing existing ordered consumers. Use a unique PollerGroupName for your Queue mode ESMs. If you reuse a PollerGroupName across modes, CreateEventSourceMapping returns a validation error.

Handling failures

MaximumRetryAttempts accepts values from 0 to 10,000 and defaults to -1, which retries indefinitely. BisectBatchOnFunctionError splits a batch in half when the function returns an error, isolating a poison-pill record from an otherwise healthy batch. To capture records that exhaust their retries, configure an on-failure destination: a Kafka topic, Amazon Simple Queue Service (Amazon SQS), or Amazon Simple Notification Service (Amazon SNS). ESM logging to Amazon CloudWatch Logs is available for diagnostics.

Key considerations for Queue mode

Keep the following in mind when configuring your event source mapping for Queue mode.

• Make your function idempoten – Lambda event source mappings process each record at least once, and duplicate processing can occur. See How do I make my Lambda function idempotent in the AWS Knowledge Center.
• No ordering guarantees – Records can arrive out of order, including records from the same partition.
• Batching window is not supported – MaximumBatchingWindowInSeconds returns a validation error. The Kafka Share Consumer requires every record in a batch to be acknowledged before fetching new records, which is incompatible with a time-based window.
• Starting position is not supported – StartingPosition and StartingPositionTimestamp can’t be used. Queue mode reads from the share group’s committed position. To control where a new share group begins reading, configure share.auto.offset.reset on the broker.
• Event filtering supports up to five filter patterns per event source mapping.

Pricing and availability

Queue consumption mode is available in all AWS Regions where Lambda event source mappings for self-managed Apache Kafka is supported. On-demand mode carries no charge beyond standard Lambda invocation charges. Provisioned mode is billed on EPU usage. For details, see AWS Lambda pricing.

Conclusion

Queue consumption mode brings queue-style processing to Apache Kafka topics without additional infrastructure and without changes to your producers, topic configuration, or function code. By replacing exclusive partition assignment with share groups, it removes the partition count as a ceiling on parallelism and eliminates head-of-line blocking for workloads built on independent records.

To get started, create an event source mapping with ConsumptionMode set to Queue on a self-managed Apache Kafka 4.2 or later cluster. For configuration details, see Polling Kafka queue topics in the AWS Lambda Developer Guide.

For more serverless learning resources, visit Serverless Land.

About the authors