Optimize Egress Costs with Follower Fetching on Confluent Cloud using AWS VPC Peering¶
Important
Limited Availability
Support for /27
CIDR blocks is in Limited Availability to a subset
of Confluent customers. To be considered for access before General
Availability, contact Confluent Support.
In Confluent Cloud, the cross-zone traffic costs are split between Confluent and you. You can reduce your AWS egress bandwidth costs by configuring your clients to consume from the nearest follower, instead of the leader.
This “fetch from follower” feature is available in Confluent Cloud based on the Kafka Improvement Proposal (KIP) 392. If you configure clients to use this feature, clients consume from replicas in the same Availability Zone (AZ) and reduce cross-zone egress charges on your AWS networking.
How it works¶
In AWS, data transfer over a virtual private cloud (VPC) peering connection that stays within an Availability Zone is free. However, by default, clients connect to the leader Kafka replica. In a multi-zone cluster that means that clients make expensive cross-zone calls.
To get the benefit of fetch from followers, you do the following:
- Distribute clients across all Availability Zones
- Configure clients to fetch from followers in the local Availability Zone
Use Confluent Cloud Metrics API to identify clients not configured to fetch from follower.
Prerequisites¶
- A Confluent Cloud network of the VPC PEERING type using /27 CIDR blocks.
- A VPC peering connection between AWS and the above Confluent Cloud network.
Configure clients¶
Some things to consider about client configuration:
- Distribute your clients across all the Availability Zones for your cluster
- The Availability Zone ID in your client configuration must match exactly the Availability Zone ID in the Confluent Cloud Console
- Availability Zone ID (
use2-az1
) is different from an Availability Zone name (us-east-1a
) - After you configure and deploy clients, use the metrics API to ensure your clients are making calls in the same zones as followers
- Clients should have unique principal IDs to simplify tracking
To configure your clients to use this optimization, set the client.rack property to use one of the AWS zones listed for your network.
client.rack=<AWS-Zone-ID>
Availability Zone IDs vs. Availability Zone names¶
Use the Availability Zone ID (use2-az1
) to identify your Availability Zone and not the Availability Zone name (us-east-1a
).
AWS generates Availability Zone names randomly for each AWS account. Availability Zone ID are constant across all
accounts. For more information, see Availability Zone IDs for your AWS resources.
Use the Metrics API to find clients not yet configured¶
Use Confluent Cloud Metrics API to query io.confluent.kafka.server/response_bytes
. Group the query with
the metric.zone_alignment
and metric.principal_id
labels to determine how clients connected to your
cluster are utilizing bandwidth.
Possible metric.zone_alignment
values:
- CROSS_ZONE - some cross zone traffic might be necessary to contact the leader
- SAME_ZONE - the client is using the same zone to communicate with a replica
- UNKNOWN - the
client.rack
property is not configured for this client
For more information on the Confluent Cloud Metrics API, see API Reference.
How to query¶
Group the metric.zone_alignment
and metric.principal_id
labels to filter io.confluent.kafka.server/response_bytes
.
Create a file named
metric_zone_alignment.json
using the following template. Be sure to changelkc-XXXXX
and the timestamp values to match your needs.{ "aggregations": [ { "metric": "io.confluent.kafka.server/response_bytes" } ], "filter": { "op": "AND", "filters": [ { "field": "resource.kafka.id", "op": "EQ", "value": "lkc-XXXXX" }, { "field": "metric.type", "op": "EQ", "value": "Fetch" } ] }, "group_by": [ "metric.zone_alignment", "metric.principal_id" ], "granularity": "PT30M", "intervals": [ "2023-01-13T23:30:00Z/PT1H" ], "limit": 1000 }
Submit the query as a
POST
using the following command. Be sure to changeAPI_KEY
andSECRET
to match your environments.http 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/query' --auth '<API_KEY>:<SECRET>' < metric_zone_alignment.json
Your output should resemble:
{ "timestamp": "2023-01-01T00:00:00Z", "value": 76, "metric.principal_id": "sa-abc123", "metric.zone_alignment": "CROSS_ZONE" }
{ "timestamp": "2023-01-01T00:00:00Z", "value": 11535077, "metric.principal_id": "sa-456xxx", "metric.zone_alignment": "SAME_ZONE" }
{ "timestamp": "2023-01-01T00:00:00Z", "value": 7340578, "metric.principal_id": "sa-xyz123", "metric.zone_alignment": "UNKNOWN" }