How to Use an HTTPS Proxy with the Confluent CLI

This topic describes how to use an HTTPS Proxy with the Confluent CLI. An HTTPS Proxy can be used if your workstation and network environment require a proxy server for HTTP/HTTPS traffic sent by the CLI. An HTTPS Proxy can also be leveraged if you are using the CLI to manage Confluent Cloud private network clusters that require public and private connections to endpoints.

The use cases presented in this topic assume you have the following:

Use case 1: Confluent Cloud cluster with public networking

For users with a simple setup–with Confluent CLI installed and with internet connectivity –you want to do two tasks:

  1. List the API keys:

    confluent api-key list

  2. List the topics on the cluster:

    confluent kafka topic list --environment env-639yqq --cluster lkc-nykw7z

In this use case, lkc-nykw7z is a basic cluster with public/internet endpoints. Both of the previous commands will egress using your workstation’s default gateway to the internet, but to two different internet endpoints. The API Key request goes to an api.confluent.cloud endpoint (Control Plane), and the topic list request goes to the broker’s Kafka REST API endpoint (Data Plane).

Here are the two requests:

GET https://api.confluent.cloud/iam/v2/api-keys?page_size=100&spec.owner=&spec.resource= HTTP/2.0

GET https://pkc-ldvj1.ap-southeast-2.aws.confluent.cloud/kafka/v3/clusters/lkc-nykw7z/topics

If network administration and firewall rules prevent direct outbound internet connections and an outbound forward proxy server is required, you must configure the Confluent CLI to use the proxy server. The workstation user must add the following to the .bashrc file. Note that you must supply the proxy server host and port, for example, http://<PROXYHOST>:<PROXYPORT>:

export HTTPS_PROXY=http://localhost:8080

Both of the previous connections will be routed using the proxy.

Use case 2: Confluent Cloud cluster with private networking

If your Confluent CLI workstation requires a proxy to connect to the internet, and is also connected to a VPC (through AWS Direct Connect, Microsoft Azure, Google Cloud, or a VPN, for example) and the VPC accesses a Confluent Cloud cluster with private networking (VPC Peering, Transit Gateway, PrivateLink), you must do the following:

  1. Send the topic list to an endpoint that is no longer on the internet–that is, it’s on a private IP routed, for example, through the AWS Direct Connect to the VPC. The forward HTTPS Proxy might not even have access or a route.

  2. Export NO_PROXY in order to prevent the Confluent CLI from sending certain connections using the proxy. Confluent CLI needs to connect directly to VPC:

    export NO_PROXY=pkc-ldvj1.ap-southeast-2.aws.confluent.cloud

    The api.confluent.cloud endpoint will be routed through the HTTPS proxy on localhost (using HTTP CONNECT). The Data Plane list topics API call goes directly to the broker’s REST endpoint following DNS lookup and routing. This use case does not use the proxy server.

Use case 3: Confluent Cloud cluster with private networking running in a VPC

A more common setup is where there is no workstation and the CLI is running on a Virtual Machine (VM) in the VPC with the Peering, AWS Transit Gateway (TGW), or PrivateLink to the Confluent Cloud cluster. In this use case, the Confluent CLI requires the following to work:

  • Internet egress routed through a proxy for Control Plane operations
  • Direct non-proxied access for Cluster Data Plane (HTTPS/REST) operations, such as list topics

Note that if a proxy server is required for Control Plane, that can also access a route to Kafka REST API endpoints, then a special NO_PROXY would not be required. This is not a typical configuration.