Connecting Kafka Streams to Confluent Cloud

You can connect Kafka Streams to your Confluent Platform Kafka cluster in Confluent Cloud.

Prerequisites

To connect Streams to Confluent Cloud, update your existing configs with the properties described here.

  1. Create a java.util.Properties instance.

  2. Configure your streams application. Kafka and Kafka Streams configuration options must be configured before using Streams. You can configure Kafka Streams by specifying parameters in a StreamsConfig instance from the java.util.Properties instance. For example:

    # Comma-separated list of the the Confluent Cloud broker endpoints. For example:
    # r0.great-app.confluent.aws.prod.cloud:9092,r1.great-app.confluent.aws.prod.cloud:9093,r2.great-app.confluent.aws.prod.cloud:9094
    bootstrap.servers=<broker-endpoint1, broker-endpoint2, broker-endpoint3>
    replication.factor=3
    security.protocol=SASL_SSL
    sasl.mechanism=PLAIN
    sasl.jaas.config=\
        org.apache.kafka.common.security.plain.PlainLoginModule required \
        username="<confluent cloud access key>" \
        password="<confluent cloud secret>";
    
    # Recommended performance settings
    producer.retries=2147483647
    producer.confluent.batch.expiry.ms=9223372036854775807
    producer.request.timeout.ms=300000
    producer.max.block.ms=9223372036854775807
    
    # Unique application ID string
    application.id=<app-id>
    
    # Local file system directory where application state is stored
    state.dir = <app-state-storage>
    
    # Client ID for your embedded producer, consumer and admin clients. This is useful for debugging to override
    client.id = <client-id>
    
    # Default is 0; override to non-zero values for fast rebalance
    num.standby.replicas = <num-replicas>
    
    # Key serde class name. This usually must be overridden
    default.key.serde = <serde-class-name>
    
    # Value serde class name. This setting must usually be overridden
    default.value.serde = <value-serde-name>
    

For more information, see Configuring a Streams Application.