ksqlDB Stream Processing

Confluent Cloud ksqlDB

You can provision ksqlDB applications by using the Confluent Cloud UI or the ccloud CLI.

For more information, see Create streaming queries in Confluent Cloud ksqlDB.

Supported Features for ksqlDB in Confluent Cloud

  • Web interface for managing your ksqlDB cloud environment directly from your browser that exposes all critical ksqlDB information.
  • SQL editor to write, develop, and execute SQL queries with auto completion directly from the Web interface.
  • Integration with Confluent Cloud Schema Registry to leverage your existing schemas to use within your SQL queries.
  • Available in AWS, GCP, and Azure in all regions.

New Features for ksqlDB in Confluent Cloud

  • New built-in functions:

    • COUNT_DISTINCT(v) - aggregate function, computes the number of distinct values of v.
    • CUBE_EXPLODE(a) - tabular function, outputs a row for each unique combination of elements in array a.
  • Expression support for joins:

    SELECT * FROM s1 JOIN s2 ON s1.str = SUBSTRING(s2.str, 3) EMIT CHANGES;
    
  • Native Map and Array constructors:

    SELECT ARRAY[1, 2] FROM s1 EMIT CHANGES;
    SELECT MAP(k1:=v1, k2:=v1*2) FROM s1 EMIT CHANGES;
    
  • Expression support for PARTITION BY:

    SELECT col0, col2, col3 FROM s1 PARTITION BY SUBSTRING(col2, 1, 4) EMIT CHANGES;
    
  • Improved error messages: ACL-related errors link to relevant ACL documentation.

Limitations for ksqlDB in Confluent Cloud

  • Currently, user-defined functions (UDFs, UDAFs, and UDTFs) aren’t supported.
  • You can have a maximum of 20 continuous queries per application.
  • You can have a maximum of three ksqlDB applications.
  • Pull queries are in preview. For more information, see Pull queries preview.
  • Connect integration isn’t supported.
  • ccloud CLI version 0.198.0 or higher is required to create ksqlDB API keys for Confluent Cloud.

Pricing for ksqlDB in Confluent Cloud

The unit of pricing in Confluent Cloud ksqlDB is the Confluent Streaming Unit. A Confluent Streaming Unit is an abstract unit that represents the linearity of performance. For example, if a workload gets a certain level of throughput with four CSUs, you can expect about three times the throughput with 12 CSUs.

Confluent charges you in CSUs per hour.

You select the number of CSUs for your application at provisioning time. You can configure CSUs in blocks of four.

  • 4 CSUs is the minimum.
  • 12 CSUs is the initial maximum.
  • Your application can have 4, 8, or 12 CSUs.

Note

Confluent Streaming Unit pricing varies slightly depending on cloud provider and region. Pricing is shown in the web interface as part of ksqlDB provisioning so that you can see exact pricing for your cloud and region of choice.

Scaling CSUs for ksqlDB in Confluent Cloud

Scaling your application’s CSUs after initial provisioning is not supported. If you determine that your application requires more CSUs, you must provision a new application with the desired number of CSUs, and migrate to your new one.

Sizing guidelines for ksqlDB in Confluent Cloud

Four CSUs are sufficient for many workloads. In general, start with four CSUs and scale out if more capacity is needed. To identify when more CSUs are needed, check ksqlDB consumer lag.

After your ksqlDB application is provisioned, you can only change the Confluent Streaming Unit allocation by migrating to a new one one with the desired number of CSUs.