ksqlDB Cluster API Quick Start

Important

The ksqlDB cluster APIs are available as a part of an Early Access Program. An Early Access Feature is a component of Confluent Cloud introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent, particularly as it becomes more widely available in follow-on preview editions. If you would like to participate in the Early Access Program, email ksql-api-access@confluent.io.

Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. The warranty, SLA, and Support Services provisions of your agreement with Confluent do not apply to Early Access Program features. Early Access Program features are considered to be a Proof of Concept as defined in the Confluent Cloud Terms of Service. Confluent may discontinue providing preview releases of the Early Access Program features at any time in Confluent’s sole discretion.

In addition to managing ksqlDB clusters with the Confluent Cloud Console and Confluent CLI , you can use a collection of REST APIs to perform some basic control operations on ksqlDB clusters.

Like other Confluent Cloud APIs, the APIs have predictable resource-oriented URLs, transport data using JSON, and use standard HTTP verbs, response codes, authentication, and design principles.

You can perform the following operations with this API:

This topic shows you how to get started with each API and provides example request and responses.

For details about call parameters and examples in multiple languages, see kdqlDB Cluster API reference.

Base URL

The base URL for all ksqlDB cluster management calls is:

https://api.confluent.cloud/

Object model

All Confluent objects share a set of common properties:

  • api_version: Indicates the API version for the object. The version for these APIs is ksqldbcm/v2.
  • kind: Indicates what kind of object it is.
  • id: An identifier for the object.

Authentication

Each request to the server must include an Authorization: Basic {key} header that contains a Confluent Cloud API key. For more details on creating or accessing your Confluent Cloud API keys, see Cloud API keys. API requests without a correct authentication header will fail. An API key grants access to a single Confluent cluster.

These keys grant API access to many resources, and should not be shared publicly to GitHub, in client-side code, and so forth.

HTTP Basic authorization requires the key to be colon-separated and base64-encoded.

For example, if your API Key is abcp7DEFGH123456789 and the corresponding Secret is XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO, generate a base64-encoded header with the following command on MacOS or Linux:

echo -n "abcp7DEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64

The resulting authorization header will be:

Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=

Alternatively, if testing using Postman, specify Basic Auth and provide the API Key as the Username and the API Secret as the password and Postman will provide the encoding.

Quotas for ksqlDB resources

ksqlDB clusters are subject to limits on how many ksqlDB clusters are allowed per environment, and how many CSUs per cluster. For more information, see Environment quotas and ksqlDB cluster quotas.

List ksqlDB clusters

If you have operator or admin access to an Confluent Cloud environment, you can list all of the ksqlDB clusters with a GET call to the ksqlDB cluster endpoint.

The request:

  • Must include the authorization header with your API key and API secret, 64-bit encoded.
  • Must specify the environment identifier for the environment that contains the clusters as a query parameter, and optionally specify additional query parameters.
  • Does not require a payload.

Request example

The following example shows how to list ksqlDB clusters.

GET /ksqldbcm/v2/clusters?environment={environment_id}&page_size={page_size}&page_token={token}

Your request can include the following query parameters:

Parameter Required (Yes/No) Description
environment Yes Specifies the identifier for the environment that contains the clusters to list.
page_size No Specifies an integer page size for the returned payload. Max value is 100. Default: 10.
page_token Yes Contains the opaque pagination token string.

Response example

A successful call returns an HTTP 200 OK and a JSON payload that contains a list of clusters for the environment. The response contains a data array. Each entry in the array describes a ksqlDB cluster in the environment, including the following:

  • A spec object with the following info:
    • The ksqlDB identifier and display name.
    • Info about the Kafka cluster associated with the ksqlDB cluster.
    • Info about the environment that contains the ksqlDB cluster.
  • The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED).

Following is an example of the response format:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "api_version": "ksqldbcm/v2",
    "data": [
        {
            "api_version": "ksqldbcm/v2",
            "id": "lksqlc-w1234",
            "kind": "Cluster",
            "metadata": {
                "created_at": "2022-07-28T23:45:39.180207Z",
                "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lksqlc-w1234",
                "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-w1234"
            },
            "spec": {
                "credential_identity": {
                    "api_version": "iam/v2",
                    "id": "",
                    "kind": "User",
                    "related": "https://api.confluent.cloud/iam/v2/users",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
                },
                "csu": 4,
                "display_name": "ksqlDB_api",
                "environment": {
                    "api_version": "org/v2",
                    "id": "env-012345",
                    "kind": "Environment",
                    "related": "https://api.confluent.cloud/org/v2/environments/env-012345",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345"
                },
                "kafka_cluster": {
                    "api_version": "cmk/v2",
                    "environment": "env-012345",
                    "id": "lkc-21012",
                    "kind": "Cluster",
                    "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
                    "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lkc-21012"
                },
                "use_detailed_processing_log": true
            },
            "status": {
                "http_endpoint": "https://pksqlc-nq123.us-central1.gcp.confluent.cloud:443",
                "is_paused": false,
                "phase": "PROVISIONED",
                "storage": 100,
                "topic_prefix": "pksqlc-nq123"
            }
        }
    ],
    "kind": "ClusterList",
    "metadata": {
        "first": "https://api.confluent.cloud/ksqldbcm/v2/clusters",
        "total_size": 1
    }
}

For details on the types of responses you can expect when a call fails, see Failures.

Create a ksqlDB cluster

If you have operator or admin access to a Confluent Cloud environment, you can create a ksqlDB cluster with a POST call to the cluster endpoint. Note that you must adhere to the Service Quotas for Confluent Cloud. The request:

  • Must include the authorization header with your API key and API secret.
  • Must include a payload that contains a spec object that describes the cluster you want to create.
POST /ksqldbcm/v2/clusters

Request example

When you make the request, include a JSON payload that contains a spec object with the following:

Parameter Required (Yes/No) Description
display_name Yes

The display name of the cluster. The following requirements apply to the cluster name:

  • Cannot exceed 32 characters
  • Can contain Unicode letters, numbers and underscores
csu Yes An integer, either 1, 2, 4, 8, 12 that specifies the number of CSUs for the ksqlDB cluster.
kafka_cluster Yes

The Kafka cluster to associate with the ksqlDB cluster.

  • id: The cluster ID
  • environment: Optional. The environment that contains the cluster, if scoped to an environment.
credential_identity Yes

The user associated with the ksqlDB cluster.

  • id: The user ID to associate with the ksqlDB cluster
  • environment: Optional. The environment associated with the user, if scoped to an environment.
environment Yes Object that contains the environment identifier: id: env-1234.

The following example demonstrates the request format:

POST /ksqldbcm/v2/clusters HTTP/1.1
Host: api.confluent.cloud

{
  "spec": {
    "display_name": "ksqlDB_api",
    "csu": 4,
    "kafka_cluster": {
      "id": "lkc-21012",
      "environment": "string"
    },
    "credential_identity": {
      "id": "u-prr8qk"
    },
    "environment": {
      "id": "env-012345"
    }
  }
}

Response example

Success returns an HTTP 202 ACCEPTED with a JSON payload that describes the ksqlDB cluster. The cluster description includes the following:

  • A spec object that describes the ksqlDB cluster, including the following:
    • The ksqlDB identifier and display name
    • Information about the Kafka cluster associated with the ksqlDB cluster
    • Information about the environment that contains the ksqlDB cluster
  • The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED)

Following is an example of the response format:

HTTP/1.1 202 ACCEPTED
Content-Type: application/json

{
 "api_version": "ksqldbcm/v2",
 "id": "lksqlc-012345",
 "kind": "Cluster",
 "metadata": {
     "created_at": "2022-07-29T17:42:53.018344Z",
     "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lksqlc-012345",
     "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-012345"
 },
 "spec": {
     "credential_identity": {
         "api_version": "iam/v2",
         "id": "",
         "kind": "User",
         "related": "https://api.confluent.cloud/iam/v2/users",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
     },
     "csu": 4,
     "display_name": "ksqlDB_api",
     "environment": {
         "api_version": "org/v2",
         "id": "env-012345",
         "kind": "Environment",
         "related": "https://api.confluent.cloud/org/v2/environments/env-012345",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345"
     },
     "kafka_cluster": {
         "api_version": "cmk/v2",
         "environment": "env-012345",
         "id": "lkc-21012",
         "kind": "Cluster",
         "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lkc-21012"
     },
     "use_detailed_processing_log": true
 },
 "status": {
     "http_endpoint": "",
     "is_paused": false,
     "phase": "PROVISIONING",
     "storage": 100,
     "topic_prefix": "pksqlc-12wko"
 }
}

For details on the types of responses you can expect when a call fails, see Failures.

Read a ksqlDB cluster

If you have operator or admin access to a Confluent Cloud environment, you can make a GET call to the ksqlDB cluster endpoint to retrieve details about a specified cluster. The request:

  • Does not require a payload.
  • Must include an authorization header with your API key and API secret.
  • Must specify the cluster and environment identifiers as path and query parameters, respectively.

Request example

GET /ksqldbcm/v2/clusters/{id}?environment={environment_id}

Your request should include the following parameters:

Parameter Required (Yes/No) Description
id Yes Path parameter specifying the identifier for the cluster to read.
environment Yes Query parameter specifying the identifier for the environment that contains the cluster to read.

Response example

Success returns an HTTP 200 OK and a JSON payload that describes the cluster, including the following:

  • A spec object that describes the ksqlDB cluster, including the following:
    • The ksqlDB identifier and display name
    • Info about the Kafka cluster associated with the ksqlDB cluster
    • Info about the environment that contains the ksqlDB cluster
  • The ksqlDB cluster status (PROVISIONED, PROVISIONING, FAILED)

Following is an example of the response format:

HTTP/1.1 200 OK
Content-Type: application/json

{
 "api_version": "ksqldbcm/v2",
 "id": "lksqlc-012345",
 "kind": "Cluster",
 "metadata": {
     "created_at": "2022-07-29T17:42:53.018344Z",
     "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lksqlc-012345",
     "self": "https://api.confluent.cloud/ksqldbcm/v2/clusters/lksqlc-012345"
 },
 "spec": {
     "credential_identity": {
         "api_version": "iam/v2",
         "id": "",
         "kind": "User",
         "related": "https://api.confluent.cloud/iam/v2/users",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/user="
     },
     "csu": 4,
     "display_name": "ksqlDB_api",
     "environment": {
         "api_version": "org/v2",
         "id": "env-012345",
         "kind": "Environment",
         "related": "https://api.confluent.cloud/org/v2/environments/env-012345",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345"
     },
     "kafka_cluster": {
         "api_version": "cmk/v2",
         "environment": "env-012345",
         "id": "lkc-21012",
         "kind": "Cluster",
         "related": "https://api.confluent.cloud/cmk/v2/clusters/lkc-21012",
         "resource_name": "crn://confluent.cloud/organization=guid-1234/environment=env-012345/cloud-cluster=lkc-21012"
     },
     "use_detailed_processing_log": true
 },
 "status": {
     "http_endpoint": "https://pksqlc-12wko.us-central1.gcp.confluent.cloud:443",
     "is_paused": false,
     "phase": "PROVISIONING",
     "storage": 100,
     "topic_prefix": "pksqlc-12wko"
 }
}

For details on the types of responses you can expect when a call fails, see Failures.

Delete a ksqlDB cluster

If you have operator or admin access to Confluent Cloud environment, you can delete a ksqlDB cluster with a DELETE call to the cluster endpoint. The request:

  • Must include the cluster and environment identifiers in the request as path and query parameters, respectively.
  • Must include an authorization header with your API key and API secret.
  • Does not require a payload.

Request example

DELETE /ksqldbcm/v2/clusters/{id}?environment={environment_id}
Parameter Required (Yes/No) Description
id Yes Path parameter specifying the identifier for the cluster to delete.
environment Yes Query parameter specifying the identifier for the environment that contains the cluster to delete.

Response example

Success returns an HTTP 204 No Content.

For details on the types of responses you can expect when a call fails, see Failures.

Failures

When a failure occurs, you should receive one of the following statuses:

400 Bad Request
Typically a malformed or missing parameter or a parameter value that is not allowed.
401 Unauthorized
Typically invalid or missing authentication credentials.
403 Forbidden
Indicates the credentials provided are not valid for the resource. Make sure your cluster and environment identifiers are correct and match the credentials provided.
404 Not Found
Indicates the resource could not be found at the specified URL.
429 Rate Limit Exceeded
You have sent too many requests.
500 Oops something went wrong
An issue not covered by other errors.

The failure message body will provide more detail about the failure reason and an identifier to use with Confluent support if you need help solving the problem. The following example shows a 400 Bad Request error with details about why the call failed.

HTTP/1.1 400 BAD REQUEST
Content-Type: application/json

{
    "errors": [
      {
           "id": "c9cb03c2878ca172bbd0072bd99a9aaa",
            "status": "400",
            "code": "parameter_missing",
            "detail": "Missing Parameter",
            "source": {
               "parameter": "environment"
             }
      }
   ]
 }