Connecting Kafka REST Proxy to Confluent Cloud

You may configure a local Kafka REST proxy to produce to and consume from a Kafka topic in a Kafka cluster in Confluent Cloud.

To connect Kafka REST to Confluent Cloud, you must download the Confluent Platform tarball and then start Kafka REST by using a customized properties file.

Prerequisites

• Access to Confluent Cloud
• Confluent Cloud CLI installed

1. Download Confluent Platform and extract the contents.

2. Create a topic named rest-proxy-test by using the Confluent Cloud CLI:

   $ ccloud topic create --partitions 4 rest-proxy-test
   

3. Create a properties file.

   1. Find the client settings for your cluster by clicking Client config from the Confluent Cloud web interface and copy the configuration.

      

      They should look like the following:

      ssl.endpoint.identification.algorithm=https
      sasl.mechanism=PLAIN
      request.timeout.ms=20000
      bootstrap.servers=<BOOTSTRAP_SERVERS>
      retry.backoff.ms=500
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<API_KEY>" password="<API_SECRET>";
      security.protocol=SASL_SSL
      

   2. Create a properties file named ccloud-kafka-rest.properties.

      $ cd <path-to-confluent>
      $ cat ccloud-kafka-rest.properties
      

   3. Paste the client config contents and add the client. prefix to each of security properties. For example:

      id=kafka-rest-with-ccloud
      bootstrap.servers=<BOOTSTRAP_SERVERS>
      client.sasl.mechanism=PLAIN
      client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="<API_KEY>" password="<API_SECRET>";
      client.security.protocol=SASL_SSL
      client.ssl.endpoint.identification.algorithm=https
      
      # consumer only properties must be prefixed with consumer.
      consumer.retry.backoff.ms=600
      consumer.request.timeout.ms=25000
      
      # producer only properties must be prefixed with producer.
      producer.acks=1
      
      # admin client only properties must be prefixed with admin.
      admin.request.timeout.ms=50000
      
      # uncomment and set correct value if using with schema registry
      #schema.registry.url=http://localhost:8081
      

      Producers, consumers, and admin client will share the client. properties. To specify properties only for the producer or consumer, use the following table:

4. Start the Kafka REST proxy.

   $ ./bin/kafka-rest-start ccloud-kafka-rest.properties
   

5. Make REST calls using v2 of the REST API (e.g. application/vnd.kafka.v2+json) because v2 has no ZooKeeper dependency. Using v1 of the API (e.g. application/vnd.kafka.v1+json) will not work because v1 has a ZooKeeper dependency that does not work with Confluent Cloud.