Quick Start for ksqlDB Cluster API on Confluent Cloud

In addition to managing ksqlDB clusters with the Confluent Cloud Console and Confluent CLI , you can use a collection of REST APIs to perform some basic control operations on ksqlDB clusters.

Like other Confluent Cloud APIs, the APIs have predictable resource-oriented URLs, transport data using JSON, and use standard HTTP verbs, response codes, authentication, and design principles.

You can perform the following operations with this API:

• List ksqlDB clusters
• Create a ksqlDB cluster
• Read a ksqlDB cluster
• Delete a ksqlDB cluster

This topic shows you how to get started with each API and provides example request and responses.

For details about call parameters and examples in multiple languages, see kdqlDB Cluster API reference.

Base URL

The base URL for all ksqlDB cluster management calls is:

https://api.confluent.cloud/

Object model

All Confluent objects share a set of common properties:

• api_version: Indicates the API version for the object. The version for these APIs is ksqldbcm/v2.
• kind: Indicates what kind of object it is.
• id: An identifier for the object.

Confluent Resource Name (CRN) for ksqlDB

The Confluent Resource Name (CRN) for ksqlDB is required for using the REST API or the Confluent Terraform provider to manage ksqlDB resources.

The format for the ksqlDB cluster CRN is:

crn://confluent.cloud/organization=<org_id>/environment=<env-id>/cloud-cluster=<kafka-cluster-id>/ksql=<ksqldb-cluster-name>

The format for the ksqlDB query CRN is:

crn://confluent.cloud/organization=<org_id>/environment=<env-id>/cloud-cluster=<kafka-cluster-id>/ksql=<ksqldb-cluster-name>/query=<query-name>

The following example shows the CRN for a ksqlDB query named “my_clicks”.

crn://confluent.cloud/organization=92b0bf20-f7c2-4a1b-8f62-b1aab73c915c/environment=env-mb0bq/cloud-cluster=lkc-1x7o6/ksql=lksqlc-m87gi/query=my_clicks

Authentication

Each request to the server must include an Authorization: Basic {key} header that contains a Confluent Cloud API key. For more details on creating or accessing your Confluent Cloud API keys, see Manage API Keys in Confluent Cloud. API requests without a correct authentication header will fail. An API key grants access to a single Confluent cluster.

These keys grant API access to many resources, and should not be shared publicly to GitHub, in client-side code, and so forth.

HTTP Basic authorization requires the key to be colon-separated and base64-encoded.

For example, if your API Key is abcp7DEFGH123456789 and the corresponding Secret is XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO, generate a base64-encoded header with the following command on MacOS or Linux:

echo -n "abcp7DEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64

The resulting authorization header will be:

Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=

Alternatively, if testing using Postman, specify Basic Auth and provide the API Key as the Username and the API Secret as the password and Postman will provide the encoding.

Quotas for ksqlDB resources

ksqlDB clusters are subject to limits on how many ksqlDB clusters are allowed per environment, and how many CSUs per cluster. For more information, see Environments and ksqlDB clusters.

List ksqlDB clusters

If you have operator or admin access to an Confluent Cloud environment, you can list all of the ksqlDB clusters with a GET call to the ksqlDB cluster endpoint.

The request:

• Must include the authorization header with your API key and API secret, 64-bit encoded.
• Must specify the environment identifier for the environment that contains the clusters as a query parameter, and optionally specify additional query parameters.
• Does not require a payload.

Request example

The following example shows how to list ksqlDB clusters.

GET /ksqldbcm/v2/clusters?environment={environment_id}&page_size={page_size}&page_token={token}

Your request can include the following query parameters:

Parameter	Required (Yes/No)	Description
environment	Yes	Specifies the identifier for the environment that contains the clusters to list.
page_size	No	Specifies an integer page size for the returned payload. Max value is 100. Default: 10.
page_token	Yes	Contains the opaque pagination token string.

Response example

A successful call returns an HTTP 200 OK and a JSON payload that contains a list of clusters for the environment. The response contains a data array. Each entry in the array describes a ksqlDB cluster in the environment, including the following:

• A spec object with the following info:
  • The ksqlDB identifier and display name.
  • Info about the Kafka cluster associated with the ksqlDB cluster.
  • Info about the environment that contains the ksqlDB cluster.
• The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED).

Following is an example of the response format:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "api_version": "ksqldbcm/v2",
    "data": [
        {
            "api_version": "ksqldbcm/v2",
            "id": "lksqlc-w1234",
            "kind": "Cluster",
            "metadata": {
                "created_at": "2022-07-28T23:45:39.180207Z",
                "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lkc-21012/ksql=lksqlc-m87gi",
                "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-w1234"
            },
            "spec": {
                "credential_identity": {
                    "api_version": "iam/v2",
                    "id": "",
                    "kind": "User",
                    "related": "https://api.confluent.cloud/iam/v2/users",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
                },
                "csu": 4,
                "display_name": "ksqlDB_api",
                "environment": {
                    "api_version": "org/v2",
                    "id": "env-a12b34",
                    "kind": "Environment",
                    "related": "https://api.confluent.cloud/org/v2/environments/env-a12b34",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34"
                },
                "kafka_cluster": {
                    "api_version": "cmk/v2",
                    "environment": "env-a12b34",
                    "id": "lkc-21012",
                    "kind": "Cluster",
                    "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lkc-21012"
                },
                "use_detailed_processing_log": true
            },
            "status": {
                "http_endpoint": "https://pksqlc-nq123.us-central1.gcp.confluent.cloud:443",
                "is_paused": false,
                "phase": "PROVISIONED",
                "storage": 100,
                "topic_prefix": "pksqlc-nq123"
            }
        }
    ],
    "kind": "ClusterList",
    "metadata": {
        "first": "https://api.confluent.cloud/ksqldbcm/v2/clusters",
        "total_size": 1
    }
}

For details on the types of responses you can expect when a call fails, see Failures.

Create a ksqlDB cluster

If you have operator or admin access to a Confluent Cloud environment, you can create a ksqlDB cluster with a POST call to the cluster endpoint. Note that you must adhere to the Service Quotas for Confluent Cloud. The request:

• Must include the authorization header with your API key and API secret.
• Must include a payload that contains a spec object that describes the cluster you want to create.

POST /ksqldbcm/v2/clusters

Request example

When you make the request, include a JSON payload that contains a spec object with the following:

Parameter	Required (Yes/No)	Description
display_name	Yes	The display name of the cluster. The following requirements apply to the cluster name: Cannot exceed 32 characters Can contain Unicode letters, numbers and underscores
csu	Yes	An integer, either 1, 2, 4, 8, 12 that specifies the number of CSUs for the ksqlDB cluster.
kafka_cluster	Yes	The Kafka cluster to associate with the ksqlDB cluster. id: The cluster ID environment: Optional. The environment that contains the cluster, if scoped to an environment.
credential_identity	Yes	The user associated with the ksqlDB cluster. id: The user ID to associate with the ksqlDB cluster environment: Optional. The environment associated with the user, if scoped to an environment.
environment	Yes	Object that contains the environment identifier: id: env-y1234q.

The following example demonstrates the request format:

POST /ksqldbcm/v2/clusters HTTP/1.1
Host: api.confluent.cloud

{
  "spec": {
    "display_name": "ksqlDB_api",
    "csu": 4,
    "kafka_cluster": {
      "id": "lkc-21012",
      "environment": "string"
    },
    "credential_identity": {
      "id": "u-prr8qk"
    },
    "environment": {
      "id": "env-a12b34"
    }
  }
}

Response example

Success returns an HTTP 202 ACCEPTED with a JSON payload that describes the ksqlDB cluster. The cluster description includes the following:

• A spec object that describes the ksqlDB cluster, including the following:
  • The ksqlDB identifier and display name
  • Information about the Kafka cluster associated with the ksqlDB cluster
  • Information about the environment that contains the ksqlDB cluster
• The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED)

Following is an example of the response format:

HTTP/1.1 202 ACCEPTED
Content-Type: application/json

{
 "api_version": "ksqldbcm/v2",
 "id": "lksqlc-012345",
 "kind": "Cluster",
 "metadata": {
     "created_at": "2022-07-29T17:42:53.018344Z",
     "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lksqlc-012345",
     "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-012345"
 },
 "spec": {
     "credential_identity": {
         "api_version": "iam/v2",
         "id": "",
         "kind": "User",
         "related": "https://api.confluent.cloud/iam/v2/users",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
     },
     "csu": 4,
     "display_name": "ksqlDB_api",
     "environment": {
         "api_version": "org/v2",
         "id": "env-a12b34",
         "kind": "Environment",
         "related": "https://api.confluent.cloud/org/v2/environments/env-a12b34",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34"
     },
     "kafka_cluster": {
         "api_version": "cmk/v2",
         "environment": "env-a12b34",
         "id": "lkc-21012",
         "kind": "Cluster",
         "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lkc-21012"
     },
     "use_detailed_processing_log": true
 },
 "status": {
     "http_endpoint": "",
     "is_paused": false,
     "phase": "PROVISIONING",
     "storage": 100,
     "topic_prefix": "pksqlc-12wko"
 }
}

For details on the types of responses you can expect when a call fails, see Failures.

Read a ksqlDB cluster

If you have operator or admin access to a Confluent Cloud environment, you can make a GET call to the ksqlDB cluster endpoint to retrieve details about a specified cluster. The request:

• Does not require a payload.
• Must include an authorization header with your API key and API secret.
• Must specify the cluster and environment identifiers as path and query parameters, respectively.

Request example

GET /ksqldbcm/v2/clusters/{id}?environment={environment_id}

Your request should include the following parameters:

Parameter	Required (Yes/No)	Description
id	Yes	Path parameter specifying the identifier for the cluster to read.
environment	Yes	Query parameter specifying the identifier for the environment that contains the cluster to read.

Response example

Success returns an HTTP 200 OK and a JSON payload that describes the cluster, including the following:

• A spec object that describes the ksqlDB cluster, including the following:
  • The ksqlDB identifier and display name
  • Info about the Kafka cluster associated with the ksqlDB cluster
  • Info about the environment that contains the ksqlDB cluster
• The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED)

Following is an example of the response format:

HTTP/1.1 200 OK
Content-Type: application/json

{
 "api_version": "ksqldbcm/v2",
 "id": "lksqlc-012345",
 "kind": "Cluster",
 "metadata": {
     "created_at": "2022-07-29T17:42:53.018344Z",
     "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lksqlc-012345",
     "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-012345"
 },
 "spec": {
     "credential_identity": {
         "api_version": "iam/v2",
         "id": "",
         "kind": "User",
         "related": "https://api.confluent.cloud/iam/v2/users",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
     },
     "csu": 4,
     "display_name": "ksqlDB_api",
     "environment": {
         "api_version": "org/v2",
         "id": "env-a12b34",
         "kind": "Environment",
         "related": "https://api.confluent.cloud/org/v2/environments/env-a12b34",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34"
     },
     "kafka_cluster": {
         "api_version": "cmk/v2",
         "environment": "env-a12b34",
         "id": "lkc-21012",
         "kind": "Cluster",
         "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-a12b34/cloud-cluster=lkc-21012"
     },
     "use_detailed_processing_log": true
 },
 "status": {
     "http_endpoint": "https://pksqlc-12wko.us-central1.gcp.confluent.cloud:443",
     "is_paused": false,
     "phase": "PROVISIONING",
     "storage": 100,
     "topic_prefix": "pksqlc-12wko"
 }
}

For details on the types of responses you can expect when a call fails, see Failures.

Delete a ksqlDB cluster

If you have operator or admin access to Confluent Cloud environment, you can delete a ksqlDB cluster with a DELETE call to the cluster endpoint. The request:

• Must include the cluster and environment identifiers in the request as path and query parameters, respectively.
• Must include an authorization header with your API key and API secret.
• Does not require a payload.

Request example

DELETE /ksqldbcm/v2/clusters/{id}?environment={environment_id}

Parameter	Required (Yes/No)	Description
id	Yes	Path parameter specifying the identifier for the cluster to delete.
environment	Yes	Query parameter specifying the identifier for the environment that contains the cluster to delete.

Response example

Success returns an HTTP 204 No Content.

For details on the types of responses you can expect when a call fails, see Failures.

Failures

When a failure occurs, you should receive one of the following statuses:

400 Bad Request

Typically a malformed or missing parameter or a parameter value that is not allowed.

401 Unauthorized

Typically invalid or missing authentication credentials.

403 Forbidden

Indicates the credentials provided are not valid for the resource. Make sure your cluster and environment identifiers are correct and match the credentials provided.

404 Not Found

Indicates the resource could not be found at the specified URL.

429 Rate Limit Exceeded

You have sent too many requests.

500 Oops something went wrong

An issue not covered by other errors.

The failure message body will provide more detail about the failure reason and an identifier to use with Confluent support if you need help solving the problem. The following example shows a 400 Bad Request error with details about why the call failed.

HTTP/1.1 400 BAD REQUEST
Content-Type: application/json

{
    "errors": [
      {
           "id": "c9cb03c2878ca172bbd0072bd99a9aaa",
            "status": "400",
            "code": "parameter_missing",
            "detail": "Missing Parameter",
            "source": {
               "parameter": "environment"
             }
      }
   ]
 }

Related content

• kdqlDB Cluster API Reference (control plane)
• ksqlDB REST APIs (data plane)
• Service Quotas for Confluent Cloud