Auto-Generating Configurations for Components to Confluent Cloud

You can connect any Confluent Platform component or Kafka client to Confluent Cloud. Use the following script to autogenerate delta configuration files for Confluent Platform components and clients connecting to Confluent Cloud. These delta configurations can be added to the respective component configuration files or application code.

Running this script will generate delta configurations for:

  • Confluent Platform Components:
    • Schema Registry
    • KSQL Data Generator
    • Confluent KSQL
    • Confluent Replicator
    • Confluent Control Center
    • Kafka Connect
    • Kafka connector
    • Kafka command line tools
  • Kafka Clients:
    • Java (Producer/Consumer)
    • Java (Streams)
    • Python
    • .NET
    • Go
    • Node.js
    • C++
  • OS:
    • ENV file

Get the code

  1. Clone the confluentinc/examples repository from GitHub and check out the 5.3.1-post branch.

    git clone git@github.com:confluentinc/examples.git
cd examples
git checkout 5.3.1-post

  2. Navigate to ccloud directory.

    cd ccloud

Create a Confluent Cloud configuration file

  1. Create a customized Confluent Cloud configuration file with connection details for the Confluent Cloud cluster using the format shown in this example, and save as /tmp/myconfig.properties.

    bootstrap.servers=<BROKER ENDPOINT>
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="<API KEY>" password\="<API SECRET>";

  2. Substitute <BROKER ENDPOINT>, <API KEY>, and <API SECRET> in the file above, to point to your Confluent Cloud cluster using the desired service account’s API key and secret.

  3. If you are using Confluent Cloud Schema Registry, add the following configuration parameters to the same file above.

    basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>

  4. If you are using Confluent Cloud Schema Registry, substitute <SR ENDPOINT>, <SR API KEY>, and <SR API SECRET> in the file above, to point to your Confluent Cloud Schema Registry using the desired service account’s Schema Registry API key and secret (which are different from the cluster API key and secret used earlier).

Generate the Delta Configurations

  1. Run the script, passing in the configuration file you defined above.

    ./ccloud-generate-cp-configs.sh /tmp/myconfig.properties

  2. Verify that your output resembles:

    Confluent Platform Components:
delta_configs/schema-registry-ccloud.delta
delta_configs/replicator-to-ccloud-producer.delta
delta_configs/ksql-server-ccloud.delta
delta_configs/ksql-datagen.delta
delta_configs/control-center-ccloud.delta
delta_configs/connect-ccloud.delta
delta_configs/connector-ccloud.delta
delta_configs/ak-tools-ccloud.delta

Kafka Clients:
delta_configs/java_producer_consumer.delta
delta_configs/java_streams.delta
delta_configs/python.delta
delta_configs/dotnet.delta
delta_configs/go.delta
delta_configs/node.delta
delta_configs/cpp.delta
delta_configs/env.delta

  3. Add the delta configuration output to the respective component’s properties file. Remember that these are the delta configurations, not the complete configurations.

Examples and Demo