Confluent Cloud Features and Limits by Cluster Type

Some features are supported by all Confluent clusters, regardless of type. However, the Confluent Cloud cluster type you choose determines the full set of features, capabilities, and the price of the cluster.

Cluster types include:

Important

The capabilities provided in this topic are for planning purposes, and are not a guarantee of performance, which varies depending on each unique configuration.

For per-topic settings and limits, see Confluent Cloud Cluster and Topic Configuration Settings. For quotas that apply to organizations, environments, clusters, and accounts, see Service Quotas for Confluent Cloud. To monitor the performance of your clusters, see Confluent Cloud Metrics.

All Confluent Cloud cluster types support the following features:

Feature Details
Kafka ACLs For details, see Use Access Control Lists (ACLs) for Confluent Cloud.
Encryption at rest Confluent Cloud uses encrypted volumes for all data storage at rest.
Encryption in motion Confluent Cloud requires SSL/TLS 1.2 for all connections.
Exactly Once Semantics For details, see Processing Guarantees.
Fully-managed replica placement Confluent Cloud manages replica placement, ensuring that replicas are spread across different brokers. For multi-zone clusters, Confluent Cloud ensures replicas are also spread across different Availability Zones.
Key based compacted storage For details, see Log Compaction.
UI: Consumer lag For details, see Monitor Consumer Lag.
Topic management For details, see Create, Edit, and Delete Topics.
Uptime SLA For details, see the Confluent Cloud Service Level Agreement .

Basic clusters

Basic clusters are designed for development use-cases. Basic clusters support the following:

  • 99.5% uptime SLA
  • Up to 100 MB/s of throughput and 5 TB of storage.
  • You only pay for the ingress, egress, storage, and partitions. There is no base cluster price.
  • Can be upgraded from a single-zone Basic cluster to a single-zone Standard cluster at any time using the Confluent Cloud Console.

Basic cluster limits per cluster

Dimension Capability Additional details
Ingress Max 100 MBps None
Egress Max 100 MBps None
Storage (pre-replication) Max 5 TB None
Partitions (pre-replication) Max 2048 Confluent Cloud clusters apply a limit to the number of partitions for a cluster. The partition limit is determined by the cluster type and size, and refers to the pre-replicated number of partitions. All topics that the customer creates as well as internal topics that are automatically created by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster partition limit. The automatically created topics are prefixed with an underscore (_), created using the Cloud Console, Confluent Cloud CLI, Kafka AdminAPI, Confluent Control Center, Kafka Streams applications, and ksqlDB. However, topics that are internal to Kafka itself (e.g., consumer offsets) are not visible in the Cloud Console, and do not count against partition limits or toward partition billing.
Total client connections Max 1000 None
Connection attempts Max 80 per second Connection attempts are equal to the number of successful authentications plus unsuccessful authentication attempts. You can use the Metrics API to query for successful connection attempts. For more information, see Confluent Cloud Metrics.
Requests Max ~15000 per second None
Message size Max 8 MB For message size defaults at the topic level see Custom topic settings for all cluster types.
Client version Minimum 0.11.0 None
Request size Max 100 MB None
Fetch bytes Max 55 MB None
API keys Max 50 None
Partition creation and deletion Max 250 per 5 minute period

The following occurs when partition creation and deletion rate limit is reached:

  • For clients < Kafka 2.7:

    The cluster always accepts and processes all the partition creates and deletes within a request, and then throttles the connection of the client until the rate of changes is below the quota.

  • For clients >= Kafka 2.7:

    The cluster only accepts and processes partition creates and deletes up to the quota. All other partition creates and deletes in the request are rejected with a THROTTLING_QUOTA_EXCEEDED error. By default, the admin client will automatically retry on that error until default.api.timeout.ms is reached. When the automatic retry is disabled by the client, the THROTTLING_QUOTA_EXCEEDED error is immediately returned to the client.

Connector tasks per Kafka cluster Max 250 1 task per connector
ACLs Max 1000 None

Basic cluster limits per partition

The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.

Dimension Capability
Ingress per Partition ~5 MBps
Egress per Partition ~15 MBps
Storage per Partition Max 250 GB

Cluster Linking capabilities on a Basic Cluster

  • A Standard cluster can be a source cluster of a cluster link (can send data).
  • A Standard cluster cannot be a destination cluster (cannot receive data).

To learn more, see Supported Cluster Types in the Cluster Linking documentation.

Standard clusters

Standard clusters are designed for production-ready features and functionality. Standard clusters support the following:

Standard cluster limits per cluster

Dimension Capability Additional details
Ingress Max 100 MBps None
Egress Max 100 MBps None
Storage (pre-replication) Infinite

Standard Confluent Cloud clusters support infinite storage. This means there is no maximum size limit for the amount of data that can be stored on the cluster.

The storage policy settings retention.bytes and retention.ms are still configurable at a topic level so you can control exactly how much and how long to retain data in a way that makes sense for your applications and controls your costs.

Partitions (pre-replication) Max 2048 Confluent Cloud clusters apply a limit to the number of partitions for a cluster. The partition limit is determined by the cluster type and size, and refers to the pre-replicated number of partitions. All topics that the customer creates as well as internal topics that are automatically created by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster partition limit. The automatically created topics are prefixed with an underscore (_), created using the Cloud Console, Confluent Cloud CLI, Kafka AdminAPI, Confluent Control Center, Kafka Streams applications, and ksqlDB. However, topics that are internal to Kafka itself (e.g., consumer offsets) are not visible in the Cloud Console, and do not count against partition limits or toward partition billing.
Total client connections Max 1000 None
Connection attempts Max 80 per second Connection attempts are equal to the number of successful authentications plus unsuccessful authentication attempts. You can use the Metrics API to query for successful connection attempts. For more information, see Confluent Cloud Metrics.
Requests Max ~15000 per second None
Message size Max 8 MB For message size defaults at the topic level, see Custom topic settings for all cluster types.
Client version Minimum 0.11.0 None
Request size Max 100 MB None
Fetch bytes Max 55 MB None
API keys Max 100 None
Partition creation and deletion Max 500 per 5 minute period

The following occurs when partition creation and deletion rate limit is reached:

  • For clients < Kafka 2.7:

    The cluster always accepts and processes all the partition creates and deletes within a request, and then throttles the connection of the client until the rate of changes is below the quota.

  • For clients >= Kafka 2.7:

    The cluster only accepts and processes partition creates and deletes up to the quota. All other partition creates and deletes in the request are rejected with a THROTTLING_QUOTA_EXCEEDED error. By default, the admin client will automatically retry on that error until default.api.timeout.ms is reached. When the automatic retry is disabled by the client, the THROTTLING_QUOTA_EXCEEDED error is immediately returned to the client.

Connector tasks per Kafka cluster Max 250 None
ACLs Max 1000 None

Standard cluster limits per partition

The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.

Dimension Capability
Ingress per Partition ~5 MBps
Egress per Partition ~15 MBps
Storage per Partition Max 250 GB

Cluster Linking capabilities on a Standard Cluster

  • A Standard cluster can be a source cluster of a cluster link (can send data).
  • A Standard cluster cannot be a destination cluster (cannot receive data).

To learn more, see Supported Cluster Types in the Cluster Linking documentation.

Dedicated clusters

Dedicated clusters are designed for critical production workloads with high traffic or private networking requirements. Dedicated clusters support the following:

Dedicated clusters are provisioned and billed in terms of Confluent Unit for Kafka (CKU). CKUs are a unit of horizontal scalability in Confluent Cloud that provide a pre-allocated amount of resources. How much you can ingest and stream per CKU depends on a variety of factors including client application design and partitioning strategy.

CKU limits per Dedicated cluster

Dedicated clusters can be purchased in any whole number of CKUs up to a limit.

  • For organizations with credit card billing, the upper limit is 4 CKUs per dedicated cluster.
  • For organizations with integrated cloud provider billing or payment using an invoice, the upper limit is 24 CKUs per dedicated cluster. Clusters up to 100 CKUs are available by request.

To provision a cluster with more CKUs than the upper limit, contact Confluent Support.

Single-zone clusters can have 1 or more CKUs, whereas multi-zone clusters require a minimum of 2 CKUs and provide high availability. Zone availability cannot be changed after the cluster is created.

Limits per CKU

CKUs determine the capacity of your cluster. For a Confluent Cloud cluster, the expected performance for any given workload is dependent on a variety of dimensions, such as message size and number of partitions.

There are two categories of CKU dimensions:

  • Dimensions with a fixed limit that cannot be exceeded.
  • Dimensions with a more flexible guideline that may be exceeded depending on the overall cluster load.

The recommended guideline for a dimension is calculated for a workload optimized across the dimensions, enabling high levels of CKU utilization as measured by the cluster load metric. You may exceed the recommended guideline for a dimension, and achieve higher performance for that dimension, usually only if your usage of other dimensions is less than the recommended guideline or fixed limit.

Also note that usage patterns across all dimensions affect the workload and you may not achieve the suggested guideline for a particular dimension. For example, if you reach the partition limit, you will not likely reach the maximum CKU throughput guideline.

You should monitor the cluster load metric for your cluster to see how your usage pattern correlates with cluster utilization.

When a cluster’s load metric is high, the cluster may delay new connections and/or throttle clients in an attempt to ensure the cluster remains available. This throttling would register as non-zero values for the producer client produce-throttle-time-max and produce-throttle-time-avg metrics and consumer client fetch-throttle-time-max and fetch-throttle-time-avg metrics.

Use the information in the following tables to determine the minimum number of CKUs to use for a given workload, how to monitor a dimension and suggestions to reduce your use of a particular dimension.

Dimensions with fixed limits

The following table lists dimensions that have a fixed maximum limit that cannot be exceeded.

Dimension Maximum per CKU Details
Storage (pre-replication) Infinite

Dedicated clusters have infinite storage, which means there is no maximum size limit for the amount of data that can be stored on the cluster.

Retrieve using the Confluent Cloud Metrics as retained_bytes (convert from bytes to TB). The API value is post-replication, so divide by the replication factor of three to get pre-replication storage usage.

You can configure policy settings retention.bytes and retention.ms at a topic level so you can control exactly how much and how long to retain data in a way that makes sense for your applications and helps control your costs. If you exceed the maximum retention values, the producers will be throttled to prevent additional writes, which will register as non-zero values for the producer client produce-throttle-time-max and produce-throttle-time-avg metrics.

If you are self-managing Kafka, you can look at how much disk space your cluster is using to understand your storage needs.

To reduce usage on this dimension, you can compress your messages and reduce your retention settings. lz4 is recommended for compression. gzip is not recommended because it incurs high overhead on the cluster.

Partitions (pre-replication) 4,500 partitions (100,000 partitions maximum across all CKUs)

Maximum number of partitions that can exist on the cluster at one time, before replication.

Retrieve using the Confluent Cloud Metrics as partition_count.

Attempts to create additional partitions beyond this limit will fail with an error message. See Notes about cluster limits for more about partition limits.

If you are self-managing Kafka, you can look at the kafka.controller:type=KafkaController,name=GlobalPartitionCount metric to understand your partition usage. Find details in the Broker section.

To reduce usage on this dimension, you can delete unused topics and create new topics with fewer partitions. You can use the Kafka Admin interface to increase the partition count of an existing topic if the initial partition count is too low.

Connection attempts 250 connection attempts per second

Maximum number of new TCP connections to the cluster that can be created in one second. This means successful authentications plus unsuccessful authentication attempts.

If you exceed the maximum, connection attempts may be refused.

If you are self-managing Kafka, you can look at the rate of change for the kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metric and the Consumer connection-creation-rate metric to understand how many new connections you are creating. For details, see Broker Metrics and Global Connection Metrics. . To reduce usage on this dimension, you can use longer-lived connections to the cluster.

Dedicated cluster limits per cluster

You can add CKUs to a dedicated cluster to meet the capacity for your high traffic workloads. However, the limits shown in this table will not change as you increase the number of CKUs. For more information about quotas and limits, see Service Quotas for Confluent Cloud and Custom topic settings for all cluster types.

Dimension Capability
Message size Max 20 MB (for message size defaults at the topic level, see Custom topic settings for all cluster types)
Client version Minimum 0.11.0
Request size Max 100 MB
Fetch bytes Max 55 MB
Partitions Depends on number of CKUs, absolute max 100,000
API keys Max 2000
Partition creation and deletion Max 5000 per 5 minute period [2]
Connector tasks per Kafka cluster Max 250
ACLs Max 10000

Notes about cluster limits

  1. Confluent Cloud clusters apply a limit to the number of partitions for a cluster. The partition limit is determined by the cluster type and size, and refers to the pre-replicated number of partitions. All topics that the customer creates as well as internal topics that are automatically created by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster partition limit. The automatically created topics are prefixed with an underscore (_), created using the Cloud Console, Confluent Cloud CLI, Kafka AdminAPI, Confluent Control Center, Kafka Streams applications, and ksqlDB. However, topics that are internal to Kafka itself (e.g., consumer offsets) are not visible in the Cloud Console, and do not count against partition limits or toward partition billing.

  2. The following occurs when partition creation and deletion rate limit is reached:

    • For clients < Kafka 2.7:

      The cluster always accepts and processes all the partition creates and deletes within a request, and then throttles the connection of the client until the rate of changes is below the quota.

    • For clients >= Kafka 2.7:

      The cluster only accepts and processes partition creates and deletes up to the quota. All other partition creates and deletes in the request are rejected with a THROTTLING_QUOTA_EXCEEDED error. By default, the admin client will automatically retry on that error until default.api.timeout.ms is reached. When the automatic retry is disabled by the client, the THROTTLING_QUOTA_EXCEEDED error is immediately returned to the client.

Dedicated cluster limits per partition

You can add CKUs to a dedicated cluster to meet the capacity for your high traffic workloads. However, the limits shown in this table will not change as you increase the number of CKUs.

The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.

Dimension Capability
Ingress per Partition Max 10 MBps (aggregate producer throughput)
Egress per Partition Max 30 MBps (aggregate consumer throughput)
Storage per Partition Max 250 GB

Dedicated cluster provisioning and resizing time

Provisioning time

The estimated time to initially provision a dedicated cluster depends on the cluster’s size and your choice of cloud provider. Regardless of the cloud provider, sometimes provisioning can take 24 hours or more. Contact Confluent Support if provisioning takes longer than 24 hours.

Note that provisioning time is excluded from the Confluent SLA.

  • AWS: the expected provisioning times is one hour per CKU. A dedicated 2 CKU cluster on AWS is estimated to complete provisioning in about two hours.
  • GCP: the expected provisioning time is one hour per CKU. A dedicated 2 CKU cluster on GCP is estimated to complete provisioning in two hours.
  • Azure: the expected provisioning times is two and a half hours per CKU. A dedicated 2 CKU cluster on Azure is estimated to complete in five hours.

You will receive an email when provisioning is complete.

Resizing time

Resizing a dedicated cluster depends on the cluster’s size, underlying cloud provider, and the current workload on the cluster.

When a cluster is not heavily loaded, expansion times will generally be similar to expected provisioning times (single-digit hours). When a cluster is under heavy load, expansion can take longer, up to a few days to complete resizing and data rebalancing.

When shrinking a cluster by 1 CKU, you can expect a shrink time of 1-2 hours.

Regardless of the load, when a cluster has infinite storage enabled, the resize should generally complete more quickly.

During a resize operation, your applications may see leader elections, but otherwise performance will not suffer. Supported Kafka clients will gracefully handle these changes. You will receive an email when the resize operation is complete.

Cluster Linking capabilities on a Dedicated cluster

A Dedicated cluster can be a source or destination of a cluster link, dependent on the networking type and the other cluster involved. To learn more, see Supported Cluster Types in the Cluster Linking documentation.