Example: Autogenerate Self-Managed Component Configs for Confluent Cloud

You can connect any Confluent Platform component or Kafka client to Confluent Cloud. Use the following script to autogenerate delta configuration files for Confluent Platform components and clients connecting to Confluent Cloud. These delta configurations can be added to the respective component configuration files or application code.

Note

Confluent does not support this script. If you encounter any problems, you can file an issue in GitHub.

Running this script will generate delta configurations for:

  • Confluent Platform Components:
    • Schema Registry
    • ksqlDB Data Generator
    • ksqlDB
    • Confluent Replicator
    • Confluent Control Center
    • Kafka Connect
    • Kafka connector
    • Kafka command line tools
  • Kafka Clients:
    • Java (Producer/Consumer)
    • Java (Streams)
    • Python
    • .NET
    • Go
    • Node.js
    • C++
  • OS:
    • ENV file

Get the code

  1. Clone the confluentinc/examples repository from GitHub.

    git clone git@github.com:confluentinc/examples.git
cd examples
git checkout master

  2. Navigate to ccloud directory.

    cd ccloud

  3. (optional) Download the Confluent Cloud CLI, which you may use to create API keys for the different resources. Alternatively, use the Confluent Cloud Console to create the API keys.

Create a Confluent Cloud configuration file

  1. Create a customized Confluent Cloud configuration file with key=value pairs of connection details for the Confluent Cloud cluster using the format shown in this example, and save as /tmp/myconfig.properties. Note: you cannot use the ~/.ccloud/config.json generated by Confluent Cloud CLI for other Confluent Platform components or clients, which is why you need to manually create your own key=value properties file.

    bootstrap.servers=<BROKER ENDPOINT>
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<API KEY>' password='<API SECRET>';

  2. Substitute <BROKER ENDPOINT>, <API KEY>, and <API SECRET> in the file above, to point to your Confluent Cloud cluster using the desired service account’s cluster API key and secret.

  3. If you are using Confluent Cloud Schema Registry, add the following configuration parameters to the same file above. Substitute <SR ENDPOINT>, <SR API KEY>, and <SR API SECRET> to point to your Confluent Cloud Schema Registry using the desired service account’s Schema Registry API key and secret (which are different from the cluster API key and secret used earlier).

    basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>

  4. If you are using Confluent Cloud ksqlDB, add the following configuration parameters to same file above. Substitute <KSQL ENDPOINT>, <KSQL API KEY>, and <KSQL API SECRET> to point to your Confluent Cloud ksqlDB using the desired service account’s ksqlDB API key and secret (which are different from the cluster API key and secret used earlier).

    ksql.endpoint=<KSQL ENDPOINT>
ksql.basic.auth.user.info=<KSQL API KEY>:<KSQL API SECRET>

  5. Review the /tmp/myconfig.properties file, which may resemble below (with required substitutions):

    bootstrap.servers=<BROKER ENDPOINT>
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<API KEY>' password='<API SECRET>';
basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>
ksql.endpoint=<KSQL ENDPOINT>
ksql.basic.auth.user.info=<KSQL API KEY>:<KSQL API SECRET>

Generate the Delta Configurations

  1. Run the script, passing in the configuration file /tmp/myconfig.properties you defined above. Reminder: you cannot use the ~/.ccloud/config.json generated by Confluent Cloud CLI for other Confluent Platform components or clients, which is why you need to manually create your own key=value properties file in the previous section.

    ./ccloud-generate-cp-configs.sh /tmp/myconfig.properties

  2. Verify that your output resembles:

    Confluent Platform Components:
delta_configs/schema-registry-ccloud.delta
delta_configs/replicator-to-ccloud-producer.delta
delta_configs/ksql-server-ccloud.delta
delta_configs/ksql-datagen.delta
delta_configs/control-center-ccloud.delta
delta_configs/connect-ccloud.delta
delta_configs/connector-ccloud.delta
delta_configs/ak-tools-ccloud.delta

Kafka Clients:
delta_configs/java_producer_consumer.delta
delta_configs/java_streams.delta
delta_configs/python.delta
delta_configs/dotnet.delta
delta_configs/go.delta
delta_configs/node.delta
delta_configs/cpp.delta
delta_configs/env.delta

  3. Add the delta configuration output to the respective component’s properties file. Remember that these are the delta configurations, not the complete configurations.

Examples