Confluent REST Proxy for Apache Kafka

The Confluent REST Proxy provides a RESTful interface to an Apache Kafka® cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.

Some example use cases are:

  • Reporting data to Kafka from any frontend app built in any language not supported by official Confluent clients
  • Ingesting messages into a stream processing framework that doesn’t yet support Kafka
  • Scripting administrative actions

There is a plugin available for REST Proxy that helps authenticate incoming requests and propagates the authenticated principal to requests to Kafka. This enables REST Proxy clients to utilize the multi-tenant security features of the Kafka broker. For more information, see Deploy Secure Standalone REST Proxy in Confluent Platform and REST Proxy Security Plugins in Confluent Platform.

The Admin REST APIs allow you to create and manage topics, manage MDS, and produce and consume to topics. The Admin REST APIs are available in these forms:

  • You can deploy Confluent Server, which exposes Admin REST APIs directly on the brokers by default. Confluent Server is shipped with Confluent Enterprise.
  • Admin REST APIs are being incrementally added to Confluent Cloud, as documented at Confluent Cloud.
  • You can deploy standalone Kafka REST Proxy nodes, which in addition to Produce and Consume APIs, also offer Admin REST APIs as of API v3.

Ready to get started?

API availability

There is a difference between the REST APIs available on Kafka brokers deployed with Confluent Server and the REST APIs available with Standalone REST Proxy. Confluent Server provides several REST APIs that are not available in the open-source Apache Kafka® distribution provided with Standalone Kafka REST Proxy. The following REST API endpoints are only available with a Confluent Server deployment:

Configs
/clusters/{cluster_id}/topics/{topic_name}/default-configs
Broker
/clusters/{cluster_id}/brokers:delete

/clusters/{cluster_id}/brokers/{broker_id}:unregister

The GET version of the following endpoint is available in open-source Kafka REST APis. The DELETE option is available only with REST that runs with a Confluent Server deployment.

/clusters/{cluster_id}/brokers/{broker_id}
Cluster Linking
/clusters/{cluster_id}/links
/clusters/{cluster_id}/links/{link_name}
/clusters/{cluster_id}/links/{link_name}/configs
/clusters/{cluster_id}/links/{link_name}/configs/{config_name}
/clusters/{cluster_id}/links/{link_name}/configs:alter
/clusters/{cluster_id}/links/{link_name}/mirrors
/clusters/{cluster_id}/links/-/mirrors
/clusters/{cluster_id}/links/{link_name}/mirrors/{mirror_topic_name}
/clusters/{cluster_id}/links/{link_name}/mirrors:promote
/clusters/{cluster_id}/links/{link_name}/mirrors:failover
/clusters/{cluster_id}/links/{link_name}/mirrors:pause
/clusters/{cluster_id}/links/{link_name}/mirrors:resume
BalancerStatus
/clusters/{cluster_id}/balancer
/clusters/{cluster_id}/balancer/any-uneven-load
BrokerTask
/clusters/{cluster_id}/brokers/-/tasks
/clusters/{cluster_id}/brokers/{broker_id}/tasks
/clusters/{cluster_id}/brokers/-/tasks/{task_type}
/clusters/{cluster_id}/brokers/{broker_id}/tasks/{task_type}
BrokerReplicaExclusion
/clusters/{cluster_id}/broker-replica-exclusions
/clusters/{cluster_id}/broker-replica-exclusions/{broker_id}
/clusters/{cluster_id}/broker-replica-exclusions:create
/clusters/{cluster_id}/broker-replica-exclusions:delete
RemoveBrokerTask
/clusters/{cluster_id}/remove-broker-tasks
/clusters/{cluster_id}/remove-broker-tasks/{broker_id}
ReplicaStatus
/clusters/{cluster_id}/topics/-/partitions/-/replica-status
/clusters/{cluster_id}/topics/{topic_name}/partitions/-/replica-status
/clusters/{cluster_id}/topics/{topic_name}/partitions/{partition_id}/replica-status

REST that runs with a Confluent Server deployment provides the full set of REST APIs. REST that runs in a Standalone deployment consists of the open-source Kafka REST APIs only. For more information about the open-source Kafka REST APIs available, see Kafka REST Proxy and the openapi yaml.

Features

The following functionality is currently exposed and available through Confluent REST APIs.

  • Metadata - Most metadata about the cluster – brokers, topics, partitions, and configs – can be read using GET requests for the corresponding URLs.
  • Producers - Instead of exposing producer objects, the API accepts produce requests targeted at specific topics or partitions and routes them all through a small pool of producers.
    • Producer configuration - Producer instances are shared, so configs cannot be set on a per-request basis. However, you can adjust settings globally by passing new producer settings in the REST Proxy configuration. For example, you might pass in the compression.type option to enable site-wide compression to reduce storage and network overhead.
  • Consumers - Consumers are stateful and therefore tied to specific REST Proxy instances. Offset commit can be either automatic or explicitly requested by the user. Currently limited to one thread per consumer; use multiple consumers for higher throughput. The REST Proxy uses either the high level consumer (v1 api) or the new 0.9 consumer (v2 api) to implement consumer-groups that can read from topics. Note: the v1 API has been marked for deprecation.
    • Consumer configuration - Although consumer instances are not shared, they do share the underlying server resources. Therefore, limited configuration options are exposed via the API. However, you can adjust settings globally by passing consumer settings in the REST Proxy configuration.
  • Data Formats - The REST Proxy can read and write data using JSON, raw bytes encoded with base64 or using JSON-encoded Avro, Protobuf, or JSON Schema. With Avro, Protobuf, or JSON Schema, schemas are registered and validated against Schema Registry.
  • REST Proxy Clusters and Load Balancing - The REST Proxy is designed to support multiple instances running together to spread load and can safely be run behind various load balancing mechanisms (e.g. round robin DNS, discovery services, load balancers) as long as instances are configured correctly.
  • Simple Consumer - The high-level consumer should generally be preferred. However, it is occasionally useful to use low-level read operations, for example to retrieve messages at specific offsets.

See also

For an example that shows how to set Docker environment variables for Confluent Platform running in ZooKeeper mode, see the Confluent Platform demo. Refer to the demo’s docker-compose.yml file for a configuration reference.

Just as important, here’s a list of features that aren’t yet supported:

  • Multi-topic Produce Requests - Currently each produce request may only address a single topic or topic-partition. Most use cases do not require multi-topic produce requests, they introduce additional complexity into the API, and clients can easily split data across multiple requests if necessary
  • Most Producer/Consumer Overrides in Requests - Only a few key overrides are exposed in the API (but global overrides can be set by the administrator). The reason is two-fold. First, proxies are multi-tenant and therefore most user-requested overrides need additional restrictions to ensure they do not impact other users. Second, tying the API too much to the implementation restricts future API improvements; this is especially important with the new upcoming consumer implementation.
  • Allow different serializer for key and value - Currently, REST Proxy chooses the serializer based on the Content-Type header; as a result, the serializer for key and value must be the same in this design.

Installation

Before starting the REST Proxy you must start Kafka and Schema Registry. For installation instructions, see Install Confluent Platform On-Premises.

Deployment

Starting the Confluent REST Proxy service is simple once its dependencies are running:

# Start the REST Proxy. The default settings automatically work with the
# default settings for local Kafka nodes.
<path-to-confluent>/bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties

If you installed Debian or RPM packages, you can simply run kafka-rest-start as it will be on your PATH. The kafka-rest.properties file contains configuration settings. The default configuration included with the REST Proxy has convenient defaults for a local testing setup and should be modified for a production deployment. By default, the server starts bound to port 8082 and does not specify a unique instance ID (required to safely run multiple proxies concurrently).

If you started the service in the background, you can use the following command to stop it:

bin/kafka-rest-stop

Development

To build a development version, you may need a development versions of common, rest-utils, and schema-registry. After installing these, you can build the Confluent REST Proxy with Maven. All the standard lifecycle phases work. During development, use

mvn -f kafka-rest/pom.xml compile

to build,

mvn -f kafka-rest/pom.xml test

to run the unit and integration tests, and

mvn exec:java

to run an instance of the proxy against a local Kafka cluster (using the default configuration included with Kafka).

To create a packaged version, optionally skipping the tests:

mvn -f kafka-rest/pom.xml package [-DskipTests]

This will produce a version ready for production in target/kafka-rest-$VERSION-package containing a directory layout similar to the packaged binary versions. You can also produce a standalone fat jar using the standalone profile:

mvn -f kafka-rest/pom.xml package -P standalone [-DskipTests]

generating target/kafka-rest-$VERSION-standalone.jar, which includes all the dependencies as well.

To run a local Kafka and REST Proxy cluster, for testing:

./testing/environments/minimal/run.sh