Integrate with Confluent Cloud using Confluent for Kubernetes Blueprints¶

Using Confluent for Kubernetes (CFK) Blueprints, you can connect CFK Blueprints-managed Confluent Platform to a Kafka cluster deployed in Confluent Cloud, for example, to run self-managed connectors, Schema Registry, REST Proxy, or ksqlDB against a Kafka cluster in Confluent Cloud.

CFK Blueprints provides a custom resource definition (ConfluentCloudSet CRD) to declaratively configure a connection from a Confluent Platform component to a pre-provisioned Kafka cluster in Confluent Cloud.

To use this feature, the Kafka cluster in Confluent Cloud must be pre-provisioned out of the band and ready to accept client connections.

Use the following custom resources (CRs) to configure and manage Confluent Cloud integration. To get detail information, run kubectl explain <CR name>.

ConfluentCloudSet CR

The ConfluentCloudSet CR represents the client-side configuration to connect to a Kafka cluster and Kafka REST API in Confluent Cloud.
KafkaClusterClass CR

The KafkaClusterClass CR specifies how the Kafka cluster is provisioned and how other components connect to the Kafka cluster. For Confluent Cloud integration, specify Confluent Cloud as the KafkaClusterClass provisioner and reference the ConfluentCloudSet CR in the connection information.
CredentialStoreConfig CR

The CredentialStoreConfig CR contains information about the credential secrets required to authenticate to Confluent Cloud.

The CredentialStoreConfig is referenced in the ConfluentCloudSet CR, and it has to be deployed before the ConfluentCloudSet CR and KafkaClusterClass CR.

A CredentialStoreConfig CR cannot be shared across ConfluentCloudSets.
CertificateStoreConfig CR

The CertificateStoreConfig CR contains information about the certificate authority (CA) and certificates required to communicate with Confluent Cloud if TLS is enabled in the Confluent Cloud cluster.

A CertificateStoreConfig CR has to be deployed before the ConfluentCloudSet CR and KafkaClusterClass CR.

Configure a connection to Confluent Cloud¶

To set up a client-side connection to a Kafka deployed in Confluent Cloud:

Create two .txt files, one for the Kafka listener and one for the REST service, and add the required client-side API keys and secrets for the Kafka and the REST service. Use the following format in both files:
```
ccloud_api_key=<API key>
ccloud_api_secret=<API secret>
```
Create a Kubernetes secret using the expected keys and the value files you created in the previous step.

The default keys are ccloud-kafka-apikey.txt and ccloud-kafkarest-apikey.txt. If using different keys, specify those keys in the next step in blueprint.credentialStoreRef.key.
```
kubectl create secret generic <secret name> \
  --from-file=ccloud-kafka-apikey.txt=<path to the Kafka credential file> \
  --from-file=ccloud-kafkarest-apikey.txt=<path to the Kafka REST credential file> \
  --namespace <namespace>
```
Create a CredentialStoreConfig CR with the reference to the Kubernetes secret you created in the previous step.
Create a CertificateStoreConfig CR if the target Confluent Cloud has TLS enabled.

Create a ConfluentCloudSetSpec CR with the connection information:

kind: ConfluentCloudSet
spec:
  clientConfigs:                    --- [1]
    kafka:
      cluster:                      --- [2]
        authentication:             --- [3]
          type: plain               --- [4]
          plain:                    --- [5]
            providerType: blueprint --- [6]
            blueprint:              --- [7]
              credentialStoreRef:
                name:               --- [8]
                key:                --- [9]
        bootstrapEndpoint:          --- [10]
        clusterID:                  --- [11]
        name:                       --- [12]
      restAPI:                      --- [13]
        authentication:             --- [14]
          type: basic               --- [15]
          basic:                    --- [16]
            providerType: blueprint --- [17]
            blueprint:              --- [18]
              credentialStoreRef:
                name:               --- [19]
                key:                --- [20]
        url:                        --- [21]
  credentialStoreConfigRef:
    name:                           --- [22]
    namespace:                      --- [23]

[1] Required. Configure a client for the Confluent Cloud cluster.
[2] Required. Configure a client for the Confluent Cloud Kafka listeners.
[3] The Confluent Cloud Kafka cluster authentication details.
[4] Required. The type of Confluent Cloud Kafka listener authentication. The valid option is plain.
[5] Configure the plain authentication for Confluent Cloud Kafka.
[6] Required. The provider for the Confluent Cloud Kafka listener credential. The valid option is blueprint.
[7] Required when providerType ([6]) is set to blueprint.
[8] The value of the spec.credentialStoreConfigRefs.name in this CR.
[9] The name of the key in the secret the spec.credentialStoreConfigRefs.name ([8]) refers to. The default value is ccloud-kafka-apikey.txt.
[10] Required. The Confluent Cloud Kafka endpoint.
[11] Required. The ID of the Confluent Cloud Kafka cluster.
[12] Required. The name of the Confluent Cloud Kafka cluster.
[13] Required. Configure a client for the Confluent Cloud Kafka REST service.
[14] The Confluent Cloud Kafka REST service authentication details.
[15] Required. The authentication method for Kafka REST service. The valid option is basic.
[16] Required when [16] is set to basic. Provide the basic authentication configuration details.
[17] Required. The provider for the REST service credential. The valid option is blueprint.
[18] Required when providerType ([17]) is set to blueprint.
[19] The value of the spec.credentialStoreConfigRefs.name in this CR.
[20] The name of the key in the secret credentialStoreConfigRefs.name ([19]) refers to. If omitted, ccloud-kafkarest-apikey.txt is used.
[21] Required. The URL of the Kafka REST endpoint.
[22] Required. The name of the CredentialConfig CR that you created in Step 3. The CR specifies the API keys and secrets for connecting to Confluent Cloud.
[23] The namespace of the CredentialConfig CR. Defaults to the current namespace of this ConfluentCloudSet CR.

The following is a sample ConfluentCloudSet CR:

kind: ConfluentCloudSet
metadata:
  name: ccloud-saslplain-info
  namespace: cpc-system
spec:
  credentialStoreConfigRef:
    name: cp-csc
    namespace: cpc-system
  clientConfigs:
    kafka:
      cluster:
        name: my-name
        clusterID: 65BXHDHWQbiQ-lvL_7GQWA
        bootstrapEndpoint: kafka-cloud-ss.org-andrew.svc.cluster.local:9071
        authentication:
          type: plain
          plain:
            providerType: blueprint
            blueprint:
              credentialStoreRef:
                key: ccloud-kafka-apikey.txt
         tls:
           enabled: true
      restAPI:
        url: https://kafka-cloud-ss.org-andrew.svc.cluster.local:8090
        authentication:
          type: "basic"
          basic:
            providerType: blueprint
            blueprint:
              credentialStoreRef:
                key: ccloud-kafkarest-apikey.txt

Configure the KafkaClusterClass CR to point to the Kafka in Confluent Cloud and to reference the connection information set up in the ConfluentCloudSet CR:

kind: KafkaClusterClass
spec:
  provisioner:
    type: ccloud                      --- [1]
    ccloud:                           --- [2]
      confluentCloudSetRef:           --- [3]
        name:                         --- [4]
        namespace:                    --- [5]

[1] Required. Set to ccloud to specify that the Kafka cluster is provisioned in Confluent Cloud.
[2] Required. Specify how Kafka is provisioned in Confluent Cloud and how to connect to it.
[3] Required. The ConfluentCloudSet CR that contains the client configurations to the cluster in Confluent Cloud.
[4] Required. The name of the referenced ConfluentCloudSet CR.
[5] The namespace of the referenced ConfluentCloudSet CR. If omitted, it defaults to the current namespace of this KafkaClusterClass CR.

The following is a sample KafkaClusterClass CR:

apiVersion: core.cpc.platform.confluent.io/v1beta1
kind: KafkaClusterClass
metadata:
  name: cp-ccloud-cfk-saslplain-kafka-standard
  namespace: cpc-system
spec:
  provisioner:
    type: ccloud
    ccloud:
      confluentCloudSetRef:
        name: ccloud-saslplain-info
        namespace: cpc-system