Configure Share Groups for Docker in Confluent Platform

This topic provides a simple, single-node setup for running Confluent Platform with share group support enabled using Docker. Share groups allow for cooperative message consumption and scaling consumers beyond the number of topic partitions, which is ideal for high-throughput queuing use cases.

For more information, see Share Groups for Confluent Platform. For client-side details, see Share Consumers for Confluent Platform.

Important

A Preview feature is a Confluent Platform component that is being introduced to gain early feedback from developers. Preview features can be used for evaluation and non-production testing purposes or to provide feedback to Confluent. The warranty, SLA, and Support Services provisions of your agreement with Confluent do not apply to Preview features. Confluent may discontinue providing preview releases of the Preview features at any time in Confluent’s’ sole discretion.

Environment variables for single-broker clusters

In a single broker cluster, these variables set properties for the internal Apache Kafka® topic where the share coordinator persists the state of all share groups and their corresponding message locks and delivery attempts.

KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR

Defines the number of brokers that replicate each partition of the internal topic used to store share group metadata.

KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR

Defines the minimum number of replicas that must be in-sync for a successful write operation to the internal state topic.

Run Kafka with share groups enabled

In a single-broker cluster, you run Kafka with share groups enabled in the same way that you would normally start it, but you specify the KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR and KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR environment variables.

The following docker-compose.yml file sets up a single-node Kafka cluster using KRaft mode. Note the highlighted lines for enabling the share coordinator and configuring its internal state topic in a one-broker cluster.

 1services:
 2
 3  broker:
 4    image: confluentinc/cp-server:8.1.0
 5    hostname: broker
 6    container_name: broker
 7    ports:
 8      - "9092:9092"
 9      - "9101:9101"
10    environment:
11      KAFKA_NODE_ID: 1
12      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
13      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
14      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
15      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
16      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
17      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
18      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
19      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
20      KAFKA_JMX_PORT: 9101
21      KAFKA_JMX_HOSTNAME: localhost
22      KAFKA_PROCESS_ROLES: 'broker,controller'
23      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
24      KAFKA_LISTENERS: 'PLAINTEXT://:29092,CONTROLLER://:29093,PLAINTEXT_HOST://0.0.0.0:9092'
25      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
26      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
27      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
28      CONFLUENT_METRICS_ENABLE: 'false'
29      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'      
30      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
31      
32      # --- Share Group Configuration (Enabled) ---
33      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
34      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
Considerations:

  • If you are using Docker Compose V1, use a dash in the docker compose commands. For more information, see Migrate to Compose V2.

  • Set the share group feature version on the cluster after the broker has started.

  • Run the commands below from the host machine.

  • Confluent drops the .sh extension for CLI tools.

To run a single-broker Kafka cluster with share groups enabled:

  1. Copy the example compose file and save to a file named: docker-compose.yml

  2. In a terminal window, navigate to where you saved the compose file, and start the single-broker Kafka cluster with the -d option to run in detached mode:

    docker compose up -d

    Your output should resemble the following:

    ✔ Network qfk-quick-start.       Created       0.0s
✔ Container broker               Started       0.3s

  3. Verify that the broker is up and running:

    docker compose ps

  4. Enable the share group feature:

    docker compose exec broker kafka-features
        --bootstrap-server localhost:9092 \
        upgrade \
        --feature share.version=1

    Your output should resemble the following:

    share.version was upgraded to 1.

  5. Create a topic named share-test with a single partition:

    docker compose exec broker kafka-topics \
        --create \
        --topic share-test \
        --bootstrap-server localhost:9092 \
        --partitions 1 \
        --replication-factor 1 \
        --if-not-exists

    Your output should resemble the following:

    Created topic share-test.

  6. In a second terminal window, start the first share consumer:

    docker compose exec broker kafka-console-share-consumer \
        --topic share-test \
        --bootstrap-server localhost:9092 \
        --group share-group-v3

    Your output should resemble the following:

    [2025-12-12 05:06:58,091] WARN Share groups and KafkaShareConsumer are part
of a preview feature introduced by KIP-932, and are not recommended for use
in production. (org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator)

  7. In a third terminal window, start the second share consumer, using the same group as the first share consumer:

    docker compose exec broker kafka-console-share-consumer \
        --topic share-test \
        --bootstrap-server localhost:9092 \
        --group share-group-v3

    You get similar output as you did with the first share consumer.

  8. Open a fourth terminal window and run the producer:

    docker compose exec broker kafka-console-producer \
        --topic share-test \
        --bootstrap-server localhost:9092

  9. In the producer terminal window, type 10 or more unique messages and press Enter after each one. For example:

    • message 1

    • message 2

    • message 3 …

  10. Verify success. Records appear in both share consumers, distributed in batches or chunks, demonstrating successful message splitting across the single partition.

    If you’re done working with Confluent Platform, you can stop and remove the Docker containers and images. To stop the consumer and producer, use the key combination Control + C.

  11. Run the following command to stop the Docker containers:

    docker compose stop

  12. After stopping the Docker containers, run the following commands to prune the Docker system. Running these commands deletes containers, networks, volumes, and images, freeing up disk space:

    docker system prune -a --volumes --filter "label=io.confluent.docker"

Troubleshoot

Issue | Error message

Cause

Fix

Share consumers can join the group but receive no messages.

Incorrect KAFKA_LISTENERS prevents the controller from assigning consumption tasks to the broker.

Ensure internal listeners use the wildcard interface. For example, see Partitions not being assigned.

Consumers don’t read old messages.

The share consumer tool often defaults to latest offset, and the --from-beginning flag isn’t supported. To read from the earliest offset, set share.auto.offset.reset to EARLIEST using the share group configurations.

Use a new, unique group ID for each consumption test to force the broker to start reading from the beginning.

UNKNOWN_TOPIC_OR_PARTITION

Producer starts before the broker fully loads KRaft metadata on start up.

Wait 15-30 seconds after docker compose up before running any commands.

exec: “kafka-features.sh”: … not found

Incorrect path or use of the .sh extension in Confluent Platform Docker images.

Drop the .sh extension: use kafka-features.

Partitions not being assigned

If you are able to form a share group but partitions are not being assigned, change the hostname in your KAFKA_LISTENERS environment variable from a specific service name, such as broker, to the wildcard interface (:) as shown below.

If your KAFKA_LISTENERS specifies a particular service name, such as broker, and partitions are not being assigned:

  • KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'

Change your KAFKA_LISTENERS to the wildcard interface (:) like in the following example:

  • KAFKA_LISTENERS: 'PLAINTEXT://:29092,CONTROLLER://:29093,PLAINTEXT_HOST://0.0.0.0:9092'

Use the wildcard interface (:) for KAFKA_LISTENERS with share groups.

Related content