Freight Clients for Confluent Cloud

Freight clients need configurations slightly different than the optimizations for other cluster types. This topic contains several example client configurations for optimizing client performance with Freight clusters. These configurations are designed to help you minimize latency, maximize throughput, meet Freight cluster requirements, and minimize costs.

Java clients

Use these examples for Java clients.

Producer

Requirement:

  • Disable idempotent producer: enable.idempotence=false

Recommendations:

  • Increase linger for larger batches: linger.ms=100

  • Use larger batch size (1MB): batch.size=1048576

  • Increase buffer memory size (500MB): buffer.memory=524288000

  • Disable adaptive partitioning: partitioner.adaptive.partitioning.enable=false

  • Increase metadata refresh rate: metadata.max.age.ms=60000

Consumer

  • Set max fetch bytes: max.partition.fetch.bytes=8388608

  • Increase metadata refresh rate: metadata.max.age.ms=60000

librdkafka-based clients

Use these examples for librdkafka-based clients.

Producer

Requirement:

  • Disable idempotent producer: enable.idempotence=false

Recommendations:

  • Increase linger for larger batches: linger.ms=100

  • Increase in-flight request: max.in.flight.requests.per.connection=5

Consumer

  • Set max fetch bytes: max.partition.fetch.bytes=8388608

Zone alignment

Freight clusters support producer and consumer zone alignment to minimize cross-zone network charges. To ensure Kafka clients produce to and consume from Confluent Cloud endpoints within the same zone, you must specify a zone identifier as part of the client identifier settings. Zone identifiers must match the zone of the network to which the client connects. You can find zone identifiers in the Cloud Console as well as in the Confluent Cloud Networking API. For more information, see Networks-(networkingv1).

Specify zone identifiers

This section shows you how to use the client.id string to specify zone identifiers.

Considerations:

  • The Availability Zone ID in your client configuration must match exactly the Availability Zone ID in the Confluent Cloud Console

  • Availability Zone ID (use2-az1) is different from an Availability Zone name (us-east-1a)

To specify zone identifiers, include a confluent_zone parameter as part of your client.id string. Add the confluent_zone parameter as a prefix to your existing client identifier separated by a comma.

client.id=confluent_zone=<Zone-ID>,existing-client-id

For example, if your existing client identifier is producer-1 and the zone identifier is use2-az1, your client identifier would look like this:

client.id=confluent_zone=use2-az1,producer-1

Availability Zone IDs vs. Availability Zone names

Use the Availability Zone ID (use2-az1) to identify your Availability Zone and not the Availability Zone name (us-east-1a). AWS generates Availability Zone names randomly for each AWS account. Availability Zone ID are constant across all accounts. For more information, see Availability Zone IDs for your AWS resources.

Follower fetching vs. zone alignment

For consumer traffic in AWS, Freight clusters support follower fetching. For more information, see Optimize Egress Costs with Follower Fetching on Confluent Cloud using AWS VPC Peering.

Follower fetching requires a similar client configuration for identifying the zone to which a consumer belongs. If you configure a consumer for both follower fetching and zone alignment, the follower fetching configuration takes precedence.

Note that client.rack must be configured for consumers.

Related content