Configure and Manage Confluent REST Proxy for Confluent Platform Using Confluent for Kubernetes

The Confluent REST Proxy provides a RESTful interface to a Kafka cluster. The Standalone REST Proxy is a component of Confluent Platform, and the API supports many interactions with your cluster, including producing and consuming messages and accessing cluster metadata such as the set of topics and mapping of partitions to brokers.

With Confluent for Kubernetes (CFK), you can configure and deploy two RESTful interfaces to Kafka:

Embedded Admin REST API
This is built into each broker and makes available RESTful API for administrative functionality for Kafka and MDS.
Standalone REST Proxy
This is a separate component service that makes available RESTful API for producing to, consuming from, and administering Kafka.

For more information on REST Proxy, see Confluent REST APIs.

Like with other Confluent Platform components, CFK provides a custom resource definition (CRD) for the REST Proxy component. This supports:

  • Configuration overrides for server (in the kafka-rest.properties file), JVM, and log4j
  • Specification of Kubernetes scheduling mechanisms
  • Pod annotations
  • Node labels
  • Tolerations

REST Proxy in CFK works with Confluent Platform 6.2.0 and later.

Configure REST Proxy

The following is a REST Proxy custom resource (CR) example:

apiVersion: platform.confluent.io/v1beta1
kind: KafkaRestProxy
metadata:
  name: kafkarestproxy
  namespace: confluent
spec:
  authentication:
    basic:
      secretRef: kafkarestproxy-users
    type: basic
  dependencies:
    schemaRegistry:
      authentication:
        basic:
          secretRef: sr-client-basic
        type: basic
      tls:
        enabled: true
      url: https://schemaregistry.operator.svc.cluster.local:8081
  image:
    application: confluentinc/cp-kafka-rest:7.7.1
    init: confluentinc/confluent-init-container:2.9.3
    pullSecretRef:
      - confluent-registry
  replicas: 3
  tls:
    autoGeneratedCerts: true

Configure dependencies

REST Proxy has a direct dependency on the following Confluent Platform components:

  • Kafka
  • Metadata Service (MDS) if RBAC is enabled
  • Schema Registry
  • Monitoring interceptors if monitoring production and consumption in Confluent Control Center

Explore the REST Proxy custom resource definition (CRD) with the following command for the dependency properties you need to set. See Use kubectl to examine Confluent Platform CRDs for more information about the command.

kubectl explain kafkarestproxy.spec

Kafka

Run the following command iteratively to see the available dependency properties in the REST Proxy CR:

kubectl explain kafkarestproxy.spec.dependencies.kafka

MDS

If RBAC is enabled, MDS needs to be configured for REST Proxy.

Run the following command iteratively to see the available dependency properties in the REST Proxy CR:

kubectl explain kafkarestproxy.spec.dependencies.mds

Schema Registry

REST Proxy has a dependency on Schema Registry if it’s being used to manage schemas.

Run the following command iteratively to see the available dependency properties in the REST Proxy CR:

kubectl explain kafkarestproxy.spec.dependencies.schemaRegistry

Set the following Schema Registry URL explicitly in the REST Proxy CR to be able to access Schema Registry:

dependencies:
  schemaRegistry:
    url:

Interceptor

REST Proxy supports interceptor configurations as part of Java new producer and consumer settings. This can be configured in the interceptor dependencies spec.

Run the following command iteratively to see the available dependency properties in the REST Proxy CR:

kubectl explain kafkarestproxy.spec.dependencies.interceptor

Configure external access

CFK supports the following external access mechanisms to the REST Proxy endpoint:

Session affinity for consumers

Kafka consumers are stateful, and requests from each consumer instance must be routed to the same rest proxy instance. When you configure external access to REST Proxy for a consumer, enable session affinity in the REST Proxy custom resource (CR).

See Load balancer, Nodeports, and Openshift Routes for configuration details of session affinity.

Configure security

Authentication

Authenticate clients with REST Proxy

To authenticate clients with REST Proxy, you can configure one of the following in the REST Proxy CR:

Authenticate REST Proxy with Kafka

To authenticate REST Proxy with Kafka brokers, you can configure one of the following in the REST Proxy CR in the Kafka dependency section:

Authenticate REST Proxy with MDS

For REST Proxy to authenticate with MDS when RBAC is enabled, configure authentication of REST Proxy with MDS as described in:

Authenticate REST Proxy with Schema Registry

To authenticate REST Proxy with Schema Registry to configure and manage schemas when Schema Registry is secured, configure authentication in the REST Proxy CR in the Schema Registry dependency section.

See Configure Authentication for Confluent Platform Components Using Confluent for Kubernetes for the client-side authentication mechanism you want to use.

Authorization

REST Proxy supports the following authorization methods:

Network encryption

REST Proxy supports the following network encryption methods:

Configure Confluent license

REST Proxy is under a subcription license.

Configure the license in the REST Proxy CR as described in Update Confluent Platform License in Confluent for Kubernetes.

Manage REST Proxy

You can scale your REST Proxy deployment up or down based on your needs with multiple instances of REST Proxy. See REST Proxy Deployment and Load Balancing for more information.

For other post-deployment tasks, such as upgrades or a rolling restart of the cluster and how those actions affect associated producers and consumers, refer to Rest Proxy Post Deployment.