Kafka Cluster Types in Confluent Cloud¶
Confluent offers different types of Kafka clusters in Confluent Cloud. The cluster type you choose determines the features, capabilities, and price of the cluster. Use the information in this topic to find the cluster with the features and capabilities that best meets your needs. Use Basic clusters for experimentation and early development. For a production cluster, choose from Standard, Enterprise, or Dedicated.
The table below offers a high-level comparison of features across Kafka cluster types.
Feature | Basic | Standard | Enterprise | Dedicated |
---|---|---|---|---|
Kafka ACLs | Yes | Yes | Yes | Yes |
Encryption-at-rest | Yes | Yes | Yes | Yes |
Encryption-in-transit | Yes | Yes | Yes | Yes |
Exactly Once Semantics | Yes | Yes | Yes | Yes |
Fully-managed replica placement | Yes | Yes | Yes | Yes |
Key based compacted storage | Yes | Yes | Yes | Yes |
User interface to manage consumer lag | Yes | Yes | Yes | Yes |
Topic management | Yes | Yes | Yes | Yes |
Uptime Service Level Agreement (SLA) | Yes | Yes | Yes | Yes |
Role-based Access Control (RBAC) | Yes [*] | Yes | Yes | Yes |
OAuth | No | Yes | Yes | Yes |
Audit logs | No | Yes | Yes | Yes |
Public networking | Yes | Yes | No | Yes |
Private networking | No | No | Yes | Yes |
Self-managed encryption keys | No | No | No | Yes |
[*] | RBAC roles for resources within the Kafka cluster (DeveloperRead, DeveloperWrite, DeveloperManage, and ResourceOwner) are not available on Basic clusters. |
Important
The capabilities provided in this topic are for planning purposes, and are not a guarantee of performance, which varies depending on each unique configuration.
Basic clusters¶
Basic Kafka clusters are designed for experimentation, early development, and basic use cases. Basic clusters support the following:
- 99.5% uptime SLA
- Up to 1 GB/s of throughput and 5 TB of storage.
- You only pay for the ingress, egress, storage, and partitions. There is no base cluster price.
- Can be upgraded to a single-zone Standard cluster at any time using the Confluent Cloud Console.
Note
You can view connector events in Confluent Cloud Console with a Basic cluster, but you can’t consume events from a topic using Confluent CLI, Java, or C/C++. For more information, see View Connector Events.
Basic limits per cluster¶
Dimension | Capability | Additional details |
---|---|---|
Ingress | Max 250 MBps | Number of bytes that can be produced to the cluster in one second. Available in the Metrics API as If you are self-managing Kafka, you can look at the producer outgoing-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec metrics to understand your throughput. To reduce usage on this dimension, you can compress your messages.
|
Egress | Max 750 MBps | Number of bytes that can be consumed from the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the consumer incoming-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec to understand your throughput. To reduce usage on this dimension, you can compress your messages
and ensure each consumer is only consuming from the topics it requires. |
Storage (pre-replication) | Max 5 TB | Number of bytes retained on the cluster, pre-replication. Available in the Metrics API as You can configure policy settings If you are self-managing Kafka, you can look at how much disk space your cluster is using to understand your storage needs. To reduce usage on this dimension, you can
compress your messages
and reduce your retention settings.
|
Partitions (pre-replication) | Max 4096 | Maximum number of partitions that can exist on the cluster at one time, before replication.
All topics that the customer creates as well as internal topics that are automatically created
by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster
partition limit. The automatically created topics are prefixed with an underscore ( Available in the Metrics API as Attempts to create additional partitions beyond this limit will fail with an error message. If you are self-managing Kafka, you can look at the To reduce usage on this dimension, you can delete unused topics and create new topics with fewer partitions. You can use the Kafka Admin interface to increase the partition count of an existing topic if the initial partition count is too low. |
Total client connections | Max 1000 | Number of TCP connections to the cluster that can be open at one time. Available
in the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metrics to understand how many connections you are using. This value may not have a 1:1 ratio to connections in Confluent Cloud, depending on the number of brokers, partitions, and applications in your self-managed cluster. How many connections a cluster supports can vary widely based on several factors, including number of producer clients, number of consumer clients, partition keying strategy, produce patterns per client, and consume patterns per client. |
Connection attempts | Max 80 per second | Maximum number of new TCP connections to the cluster that can be created in one second. This means successful authentications plus unsuccessful authentication attempts. Available in the Metrics API as If you are self-managing Kafka, you can look at the rate of change for the
kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metric and
the Consumer To reduce usage on this dimension, you can use longer-lived connections to the cluster. |
Requests | Max ~15000 per second | Number of client requests to the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce FetchConsumer FetchFollower} metrics and client request-rate metrics to understand your request volume. To reduce usage on this dimension, you can adjust producer batching configurations, consumer client batching configurations, and shut down otherwise inactive clients. |
Message size | Max 8 MB | For message size defaults at the topic level see Kafka topic configurations for all Confluent Cloud cluster types. |
Client version | Minimum 0.11.0 | None |
Request size | Max 100 MB | None |
Fetch bytes | Max 55 MB | None |
API keys | Default limit 50 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Partition creation and deletion | Max 250 per 5 minute period | The following occurs when partition creation and deletion rate limit is reached:
|
Connector tasks per Kafka cluster | Max 250 | 1 task per connector |
ACLs | Default limit 1000 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Kafka REST Produce v3 | Max throughput: 10 MBps Max connection requests: 25 connection requests per second Max streamed requests: 1000 requests per second Max message size for Kafka REST Produce API: 1 MB |
Client applications can connect over the REST API to Produce records directly to the Confluent Cloud cluster. To learn more, see the Kafka REST concepts overview. To learn more about the Kafka REST Produce API streaming mode, see the examples and explanation of streaming mode in the concept docs, and the Produce example in the quick start. A streamed request is a single record to be produced to Kafka. Concatenate the records to produce multiple records in the same request. Delivery reports are concatenated in the same order as the records are sent. For more information, see Produce Records. |
Basic limits per partition¶
The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.
Dimension | Capability |
---|---|
Ingress per Partition | ~5 MBps |
Egress per Partition | ~15 MBps |
Storage per Partition | Unlimited |
Storage per Partition for Compacted Topics | Unlimited |
Cluster Linking capabilities¶
- A Basic cluster can be a source cluster of a cluster link (can send data).
- A Basic cluster cannot be a destination cluster (cannot receive data).
To learn more, see Supported cluster types in the Cluster Linking documentation.
Standard clusters¶
Standard clusters are designed for production-ready features and functionality. Standard clusters support the following:
- Uptime SLA: 99.95% for Single-Zone, 99.99% for Multi-Zone
- Up to 1 GB/s of throughput and unlimited storage.
- Multi-zone high availability (optional). A multi-zone cluster is spread across three availability zones for better resiliency.
- Charged an hourly base price in addition to the ingress, egress, storage, and partitions.
Standard limits per cluster¶
Dimension | Capability | Additional details |
---|---|---|
Ingress | Max 250 MBps | Number of bytes that can be produced to the cluster in one second. Available in the Metrics API as If you are self-managing Kafka, you can look at the producer outgoing-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec metrics to understand your throughput. To reduce usage on this dimension, you can compress your messages.
|
Egress | Max 750 MBps | Number of bytes that can be consumed from the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the consumer incoming-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec to understand your throughput. To reduce usage on this dimension, you can compress your messages
and ensure each consumer is only consuming from the topics it requires. |
Storage (pre-replication) | Infinite | Number of bytes retained on the cluster, pre-replication. Standard Confluent Cloud clusters support infinite storage. This means there is no maximum size limit for the amount of data that can be stored on the cluster. Available in the Metrics API as You can configure policy settings If you are self-managing Kafka, you can look at how much disk space your cluster is using to understand your storage needs. To reduce usage on this dimension, you can
compress your messages
and reduce your retention settings.
|
Partitions (pre-replication) | Max 4096 | Maximum number of partitions that can exist on the cluster at one time, before replication.
All topics that the customer creates as well as internal topics that are automatically created
by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster
partition limit. The automatically created topics are prefixed with an underscore ( Available in the Metrics API as Attempts to create additional partitions beyond this limit will fail with an error message. If you are self-managing Kafka, you can look at the To reduce usage on this dimension, you can delete unused topics and create new topics with fewer partitions. You can use the Kafka Admin interface to increase the partition count of an existing topic if the initial partition count is too low. |
Total client connections | Max 1000 | Number of TCP connections to the cluster that can be open at one time. Available
in the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metrics to understand how many connections you are using. This value may not have a 1:1 ratio to connections in Confluent Cloud, depending on the number of brokers, partitions, and applications in your self-managed cluster. How many connections a cluster supports can vary widely based on several factors, including number of producer clients, number of consumer clients, partition keying strategy, produce patterns per client, and consume patterns per client. |
Connection attempts | Max 80 per second | Maximum number of new TCP connections to the cluster that can be created in one second. This means successful authentications plus unsuccessful authentication attempts. Available in the Metrics API as If you are self-managing Kafka, you can look at the rate of change for the
kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metric and
the Consumer To reduce usage on this dimension, you can use longer-lived connections to the cluster. |
Requests | Max ~15000 per second | Number of client requests to the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce FetchConsumer FetchFollower} metrics and client request-rate metrics to understand your request volume. To reduce usage on this dimension, you can adjust producer batching configurations, consumer client batching configurations, and shut down otherwise inactive clients. |
Message size | Max 8 MB | For message size defaults at the topic level, see Kafka topic configurations for all Confluent Cloud cluster types. |
Client version | Minimum 0.11.0 | None |
Request size | Max 100 MB | None |
Fetch bytes | Max 55 MB | None |
API keys | Default limit 100 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Partition creation and deletion | Max 500 per 5 minute period | The following occurs when partition creation and deletion rate limit is reached:
|
Connector tasks per Kafka cluster | Max 250 | None |
ACLs | Default limit 1000 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Kafka REST Produce v3 | Max throughput: 10 MBps Max connection requests: 25 connection requests per second Max streamed requests: 1000 requests per second Max message size for Kafka REST Produce API: 1 MB |
Client applications can connect over the REST API to Produce records directly to the Confluent Cloud cluster. To learn more, see the Kafka REST concepts overview. To learn more about the Kafka REST Produce API streaming mode, see the examples and explanation of streaming mode in the concept docs, and the Produce example in the quick start. A streamed request is a single record to be produced to Kafka. Concatenate the records to produce multiple records in the same request. Delivery reports are concatenated in the same order as the records are sent. For more information, see Produce Records. |
Standard limits per partition¶
The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.
Dimension | Capability |
---|---|
Ingress per Partition | ~5 MBps |
Egress per Partition | ~15 MBps |
Storage per Partition | Unlimited |
Storage per Partition for Compacted Topics | Unlimited |
Cluster Linking capabilities¶
- A Standard cluster can be a source cluster of a cluster link (can send data).
- A Standard cluster cannot be a destination cluster (cannot receive data).
To learn more, see Supported cluster types in the Cluster Linking documentation.
Enterprise clusters¶
Enterprise clusters are designed for production-ready functionality that requires private endpoint networking capabilities. Enterprise clusters support the following:
- Uptime SLA: 99.99%
- Up to 1 GB/s of throughput and unlimited storage
- Multi-zone high availability cluster spread across three availability zones for better resiliency
- Automatic scaling based on load
Confluent uses Confluent Unit for Kafka for Enterprise clusters (E-CKU) to provision and bill for Enterprise Kafka clusters. Sometimes E-CKU is referred to as CKU for Enterprise.
E-CKU limits per cluster¶
Enterprise clusters are elastic, shrinking and expanding automatically based on load. You don’t resize your cluster. When you need more capacity, your Enterprise cluster expands up to the fixed ceiling. If you’re not using capacity above the minimum, you’re not paying for it.
Enterprise cluster capacity:
- Minimum: two E-CKU
- Fixed ceiling: five E-CKU
E-CKU capacity guidance¶
The dimensions in the following table describe the capacity of a single E-CKU. For more information about E-CKU, see Confluent Unit for Kafka for Enterprise clusters and Compare CKU for Enterprise and Dedicated Kafka clusters.
Dimension | E-CKU capacity |
---|---|
Storage (pre-replication) | Infinite |
Partitions (pre-replication) | 1000 partitions for non-compacted topics, 120 for compacted topics |
Connection attempts | 250 connection attempts per second |
Ingress | ~50 megabytes per second (MBps) |
Egress | ~150 megabytes per second (MBps) |
Total client connections | ~4,500 connections |
Requests | ~7,500 requests per second |
Enterprise limits per cluster¶
Enterprise clusters have a maximum capacity of five E-CKUs. For any Confluent Cloud cluster, the expected performance for any given workload is dependent on a variety of dimensions, such as message size and number of partitions.
Use the information in the following tables to determine if a given workload fits within the fixed ceiling, how to monitor dimensions, and suggestions to reduce your use of a particular dimension.
Dimension | Limits per cluster |
---|---|
Ingress | 250 megabytes per second (MBps) |
Egress | 750 megabytes per second (MBps) |
Total client connections | 22,500 connections |
Requests | 37,500 requests per second |
Storage (pre-replication) | Infinite |
Partitions (pre-replication) | 5,000 partitions for non-compacted topics, 600 for compacted topics |
Connection attempts | 1,250 connection attempts per second |
Kafka REST Produce v3 | Max throughput: 10 MBps Max connection requests: 25 connection requests per second Max streamed requests: 1000 requests per second Max message size for Kafka REST Produce API: 1 MB |
Message size | Max 8 MB |
Client version | Minimum 0.11.0 |
Request size | Max 100 MB |
Fetch bytes | Max 55 MB |
API keys | Default limit 500 (You can request an increase. For more information, see Service Quotas for Confluent Cloud.) |
Partition creation and deletion | Max 5000 per 5 minute period |
Connector tasks per Kafka cluster | Max 250 |
ACLs | Default limit 4,000 (You can request an increase. For more information, see Service Quotas for Confluent Cloud.) |
Enterprise limits per partition¶
The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.
Dimension | Capability |
---|---|
Ingress per Partition | ~5 MBps |
Egress per Partition | ~15 MBps |
Storage per Partition | Unlimited |
Storage per Partition for Compacted Topics | Unlimited |
Cluster Linking capabilities¶
Enterprise clusters can be a source of a cluster link, dependent on the networking type and the other cluster involved. To learn more, see Supported cluster types in the Cluster Linking documentation.
Dedicated clusters¶
Dedicated clusters are designed for critical production workloads with high traffic or private networking requirements. Dedicated clusters support the following:
- Single-tenant deployments with a 99.95% uptime SLA for Single-Zone, and 99.99% for Multi-Zone
- Private networking options including VPC peering, AWS Transit Gateway, AWS PrivateLink, and Azure PrivateLink.
- Self-managed keys when AWS, Azure, or Google Cloud is the cloud service provider.
- Multi-zone high availability (optional). A multi-zone cluster is spread across three availability zones for better resiliency.
- Can be scaled to achieve gigabytes per second of ingress.
- Simple scaling in terms of CKUs.
- Cluster expansion, and Cluster shrinking.
- Cluster Linking for fully-managed replication (multiregion, multicloud, hybrid cloud, and inter-organization).
Dedicated clusters are provisioned and billed in terms of Confluent Unit for Kafka (CKU). CKUs are a unit of horizontal scalability in Confluent Cloud that provide a pre-allocated amount of resources. How much you can ingest and stream per CKU depends on a variety of factors including client application design and partitioning strategy.
CKU limits per cluster¶
Dedicated clusters can be purchased in any whole number of CKUs up to a limit.
- For organizations with credit card billing, the upper limit is 4 CKUs per Dedicated cluster. Clusters up to 100 CKUs are available by request.
- For organizations with integrated cloud provider billing or payment using an invoice, the upper limit is 24 CKUs per Dedicated cluster. Clusters up to 100 CKUs are available by request.
For clusters that can scale to 100 CKU, contact Confluent Support to discuss the onboarding process and product considerations.
Single-zone clusters can have 1 or more CKUs, whereas multi-zone clusters, which are spread across three availability zones, require a minimum of 2 CKUs. Zone availability cannot be changed after the cluster is created.
Limits per CKU¶
CKUs determine the capacity of your cluster. For a Confluent Cloud cluster, the expected performance for any given workload is dependent on a variety of dimensions, such as message size and number of partitions.
There are two categories of CKU dimensions:
- Dimensions with a fixed limit that cannot be exceeded.
- Dimensions with a more flexible guideline that may be exceeded depending on the overall cluster load.
The recommended guideline for a dimension is calculated for a workload optimized across the dimensions, enabling high levels of CKU utilization as measured by the cluster load metric. You may exceed the recommended guideline for a dimension, and achieve higher performance for that dimension, usually only if your usage of other dimensions is less than the recommended guideline or fixed limit.
Also note that usage patterns across all dimensions affect the workload and you may not achieve the suggested guideline for a particular dimension. For example, if you reach the partition limit, you will not likely reach the maximum CKU throughput guideline.
You should monitor the cluster load metric for your cluster to see how your usage pattern correlates with cluster utilization.
When a cluster’s load metric is high, the cluster may delay new connections and/or throttle clients in an attempt to ensure the cluster remains available. This throttling would register as non-zero values for the producer client produce-throttle-time-max and produce-throttle-time-avg metrics and consumer client fetch-throttle-time-max and fetch-throttle-time-avg metrics.
Use the information in the following tables to determine the minimum number of CKUs to use for a given workload, how to monitor a dimension and suggestions to reduce your use of a particular dimension.
Dimensions with fixed limits¶
The following table lists dimensions that have a fixed maximum limit that cannot be exceeded.
Dimension | Maximum per CKU | Details |
---|---|---|
Storage (pre-replication) | Infinite | Number of bytes retained on the cluster, pre-replication. Dedicated clusters have infinite storage, which means there is no maximum size limit for the amount of data that can be stored on the cluster. Available in the Metrics API as You can configure policy settings If you are self-managing Kafka, you can look at how much disk space your cluster is using to understand your storage needs. To reduce usage on this dimension, you can
compress your messages
and reduce your retention settings.
|
Partitions (pre-replication) | 4,500 partitions (100,000 partitions maximum across all CKUs) | Maximum number of partitions that can exist on the cluster at one time, before replication.
All topics that the customer creates as well as internal topics that are automatically created
by Confluent Platform components–such as ksqlDB, Kafka Streams, Connect, and Control Center–count towards the cluster
partition limit. The automatically created topics are prefixed with an underscore ( Available in the Metrics API as Attempts to create additional partitions beyond this limit will fail with an error message. If you are self-managing Kafka, you can look at the To reduce usage on this dimension, you can delete unused topics and create new topics with fewer partitions. You can use the Kafka Admin interface to increase the partition count of an existing topic if the initial partition count is too low. |
Connection attempts | 500 connection attempts per second | Maximum number of new TCP connections to the cluster that can be created in one second. This means successful authentications plus unsuccessful authentication attempts. If you exceed the maximum, connection attempts may be refused. If you are self-managing Kafka, you can look at the rate of change for the
kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metric and
the Consumer |
Kafka REST Produce v3 | Max throughput: 50 MBps Max active connections: 300 Max connection requests: 300 connection requests per second Max streamed requests: 3000 requests per second |
Client applications can connect over the REST API to Produce records directly to the Confluent Cloud cluster. To learn more, see the Kafka REST concepts overview. To learn more about the Kafka REST Produce API streaming mode, see the examples and explanation of streaming mode in the concept docs, and the Produce example in the quick start. A streamed request is a single record to be produced to Kafka. Concatenate the records to produce multiple records in the same request. Delivery reports are concatenated in the same order as the records are sent. For more information, see Produce Records. |
Dimensions with a recommended guideline¶
The following table lists dimensions with a recommended guideline.
These dimensions provide guidelines for capacity planning. The ability to fully utilize these dimensions depend on the workload and utilization of other dimensions. See more about measuring load in cluster load metric and for the maximum bandwidth for each cloud provider (AWS, GCP, Azure), are available in Benchmark Your Dedicated Apache Kafka Cluster on Confluent Cloud.
Dimension | Guideline per CKU | Details |
---|---|---|
Ingress | ~50 megabytes per second (MBps) | Number of bytes that can be produced to the cluster in one second. Available in the Metrics API as If you are self-managing Kafka, you can look at the producer outgoing-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec metrics to understand your throughput. To reduce usage on this dimension, you can compress your messages.
|
Egress | ~150 megabytes per second (MBps) | Number of bytes that can be consumed from the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the consumer incoming-byte-rate metrics and broker kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec to understand your throughput. To reduce usage on this dimension, you can compress your messages
and ensure each consumer is only consuming from the topics it requires. |
Total client connections | ~9,000 connections | Number of TCP connections to the cluster that can be open at one time. Available
in the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.server:type=socket-server-metrics,listener={listener_name},networkProcessor={#},name=connection-count metrics to understand how many connections you are using. This value may not have a 1:1 ratio to connections in Confluent Cloud, depending on the number of brokers, partitions, and applications in your self-managed cluster. How many connections a cluster supports can vary widely based on several factors, including number of producer clients, number of consumer clients, partition keying strategy, produce patterns per client, and consume patterns per client. Confluent derives this guideline from benchmarking that indicates exceeding this number of connections increases produce latency for test clients. However, this doesn’t apply to all workloads. This is why total client connections are a guideline, not a hard limit for Dedicated Kafka clusters. Monitor the impact on cluster load as connection count increases, as this is the final representation of the impact of a given workload or CKU dimension on the cluster’s underlying resources. Consider this guideline a per-CKU guideline. The number of connections tends to increase when you add brokers. In other words, if you significantly exceed the per-CKU guideline, cluster expansion won’t always give your cluster more connection count headroom. |
Requests | ~15,000 requests per second | Number of client requests to the cluster in one second. Available in the the Metrics API as If you are self-managing Kafka, you can look at the broker kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce FetchConsumer FetchFollower} metrics and client request-rate metrics to understand your request volume. To reduce usage on this dimension, you can adjust producer batching configurations, consumer client batching configurations, and shut down otherwise inactive clients. A high number of requests per second results in increased load on the cluster. |
Dedicated limits per cluster¶
You can add CKUs to a Dedicated cluster to meet the capacity for your high traffic workloads. However, the limits shown in this table will not change as you increase the number of CKUs. For more information about quotas and limits, see Service Quotas for Confluent Cloud and Kafka topic configurations for all Confluent Cloud cluster types.
Dimension | Capability | Additional details |
---|---|---|
Message size | Max 20 MB | For message size defaults at the topic level, see Kafka topic configurations for all Confluent Cloud cluster types. |
Client version | Minimum 0.11.0 | None |
Request size | Max 100 MB | None |
Fetch bytes | Max 55 MB | None |
Partitions | Depends on number of CKUs, absolute max 100,000 | None |
API keys | Default limit 2000 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Partition creation and deletion | Max 5000 per 5 minute period | The following occurs when partition creation and deletion rate limit is reached:
|
Connector tasks per Kafka cluster | Max 250 | None |
ACLs | Default limit 10000 | You can request an increase. For more information, see Service Quotas for Confluent Cloud. |
Dedicated limits per partition¶
You can add CKUs to a Dedicated cluster to meet the capacity for your high traffic workloads. However, the limits shown in this table will not change as you increase the number of CKUs.
The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary depending on your individual configuration, and these benchmarks do not guarantee performance.
Dimension | Capability |
---|---|
Ingress per Partition | Max 10 MBps (aggregate producer throughput) |
Egress per Partition | Max 30 MBps (aggregate consumer throughput) |
Storage per Partition | Unlimited |
Storage per Partition for Compacted Topics | Unlimited |
Dedicated provisioning time¶
The estimated time to initially provision a Dedicated cluster depends on the cluster’s size and your choice of cloud provider. Regardless of the cloud provider, sometimes provisioning can take 24 hours or more. Contact Confluent Support if provisioning takes longer than 24 hours.
Note that provisioning time is excluded from the Confluent SLA.
- AWS: the expected provisioning time is one hour per CKU. A dedicated 2 CKU cluster on AWS is estimated to complete provisioning in about two hours.
- GCP: the expected provisioning time is one hour per CKU. A dedicated 2 CKU cluster on GCP is estimated to complete provisioning in two hours.
- Azure: the expected provisioning time is two and a half hours per CKU. A dedicated 2 CKU cluster on Azure is estimated to complete provisioning in five hours.
You will receive an email when provisioning is complete.
Dedicated resizing time¶
When a cluster is not heavily loaded, expansion and shrink times are generally similar to expected provisioning times (single-digit hours). When a cluster is under heavy load, resizing can take longer.
Resizing a cluster should take about 1-2 hours per CKU.
During a resize operation, your applications may see leader elections, but otherwise performance will not suffer. Supported Kafka clients will gracefully handle these changes. You will receive an email when the resize operation is complete.
Cluster Linking capabilities¶
Dedicated clusters can be a source or destination of a cluster link, dependent on the networking type and the other cluster involved. To learn more, see Supported cluster types in the Cluster Linking documentation.