Connecting Kafka REST Proxy to Confluent Cloud

You may configure a local Kafka REST proxy to produce to and consume from a Kafka topic in a Kafka cluster in Confluent Cloud.

To connect Kafka REST to Confluent Cloud, you must download the Confluent Platform tarball and then start Kafka REST by using a customized properties file.

Prerequisites
  • Access to Confluent Cloud
  • Confluent Cloud CLI installed
  1. Download Confluent Platform and extract the contents.

  2. Create a topic named rest-proxy-test by using the Confluent Cloud CLI:

    $ ccloud topic create --partitions 4 rest-proxy-test
    
  3. Create a properties file.

    1. Find the client settings for your cluster by clicking Client config from the Confluent Cloud web interface and copy the configuration.

      ../../_images/cloud-client-config.png

      They should look like the following:

      ssl.endpoint.identification.algorithm=https
      sasl.mechanism=PLAIN
      request.timeout.ms=20000
      bootstrap.servers=<BOOTSTRAP_SERVERS>
      retry.backoff.ms=500
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<API_KEY>" password="<API_SECRET>";
      security.protocol=SASL_SSL
      
    2. Create a properties file named ccloud-kafka-rest.properties.

      $ cd <path-to-confluent>
      $ cat ccloud-kafka-rest.properties
      
    3. Paste the client config contents and add the client. prefix to each of security properties. For example:

      id=kafka-rest-with-ccloud
      bootstrap.servers=<BOOTSTRAP_SERVERS>
      client.sasl.mechanism=PLAIN
      client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="<API_KEY>" password="<API_SECRET>";
      client.security.protocol=SASL_SSL
      client.ssl.endpoint.identification.algorithm=https
      
      # consumer only properties must be prefixed with consumer.
      consumer.retry.backoff.ms=600
      consumer.request.timeout.ms=25000
      
      # producer only properties must be prefixed with producer.
      producer.acks=1
      
      # admin client only properties must be prefixed with admin.
      admin.request.timeout.ms=50000
      
      # uncomment and set correct value if using with schema registry
      #schema.registry.url=http://localhost:8081
      

      Producers, consumers, and admin client will share the client. properties. To specify properties only for the producer or consumer, use the following table:

  4. Start the Kafka REST proxy.

    $ ./bin/kafka-rest-start ccloud-kafka-rest.properties
    
  5. Make REST calls using v2 of the REST API (e.g. application/vnd.kafka.v2+json) because v2 has no ZooKeeper dependency. Using v1 of the API (e.g. application/vnd.kafka.v1+json) will not work because v1 has a ZooKeeper dependency that does not work with Confluent Cloud.