Connecting REST Proxy to Confluent Cloud¶
You may configure a local Confluent REST Proxy to produce to and consume from a Apache Kafka® topic in a Kafka cluster in Confluent Cloud.
To connect REST Proxy to Confluent Cloud, you must download the Confluent Platform tarball and then start REST Proxy by using a customized properties file.
- Prerequisites
- Access to Confluent Cloud.
- Confluent Cloud CLI installed and configured.
Download Confluent Platform and extract the contents.
Create a topic named
rest-proxy-test
by using the Confluent Cloud CLI:ccloud kafka topic create --partitions 4 rest-proxy-test
Create a properties file.
Find the client settings for your cluster by clicking CLI & client configuration from the Confluent Cloud web interface and copy the configuration.
They should look like the following:
ssl.endpoint.identification.algorithm=https sasl.mechanism=PLAIN request.timeout.ms=20000 bootstrap.servers=<BOOTSTRAP_SERVERS> retry.backoff.ms=500 sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api-key>" password="<api-secret>"; security.protocol=SASL_SSL
Create a properties file named
ccloud-kafka-rest.properties
.cd <path-to-confluent> cat ccloud-kafka-rest.properties
Paste the client config contents and add the
client.
prefix to each of security properties. For example:id=kafka-rest-with-ccloud bootstrap.servers=<BOOTSTRAP_SERVERS> client.sasl.mechanism=PLAIN client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="<api-key>" password="<api-secret>"; client.security.protocol=SASL_SSL client.ssl.endpoint.identification.algorithm=https # consumer only properties must be prefixed with consumer. consumer.retry.backoff.ms=600 consumer.request.timeout.ms=25000 # producer only properties must be prefixed with producer. producer.acks=1 # admin client only properties must be prefixed with admin. admin.request.timeout.ms=50000 # uncomment and set correct value if using with schema registry #schema.registry.url=http://localhost:8081
Producers, consumers, and admin client will share the
client.
properties. To specify properties only for the producer or consumer, use the following table:Component Prefix Example Admin Client admin.
admin.request.timeout.ms Consumer consumer.
consumer.request.timeout.ms Producer producer.
producer.acks
(Optional) Add configs for Confluent Cloud Schema Registry per the examples on GitHub at ccloud/examples/template_delta_configs.
Start the REST Proxy.
./bin/kafka-rest-start ccloud-kafka-rest.properties
Make REST calls using v2 of the REST API (e.g.
application/vnd.kafka.v2+json
) because v2 has no ZooKeeper dependency. Using v1 of the API (e.g.application/vnd.kafka.v1+json
) will not work because v1 has a ZooKeeper dependency that does not work with Confluent Cloud.