Connect Self-Managed MQTT Proxy to Confluent Cloud

The Confluent Platform MQTT Proxy component enables MQTT clients to use the MQTT 3.1.1 protocol to publish data directly to Apache Kafka®. You can configure the Confluent MQTT Proxy to produce to a topic in a Kafka cluster in Confluent Cloud.

The following steps walk you through configuring the MQTT Proxy with Confluent Cloud.

Prerequisites

Set up the cluster

In the following steps, you install the Confluent CLI and use it to sign in to Confluent Cloud, get the cluster endpoint, and create a topic and an API key that you will use to configure the MQTT proxy.

  1. Install Confluent CLI as described in the Confluent CLI installation guide. For a list of all of the Confluent CLI commands, see Confluent CLI Command Reference.

  2. Sign in to your Confluent Cloud cluster.

    confluent login

    Your output should resemble:

    Enter your Confluent credentials:
Email: jdoe@myemail.io
Password: ***********************

Logged in as "jdoe@myemail.io"
Using environment "t118" ("default")

  3. Run the confluent kafka cluster list command to get the Kafka cluster ID.

    confluent kafka cluster list

    Your output should resemble:

         Id      |        Name         |   Type    | Provider |  Region  | Availability | Status
-------------+---------------------+-----------+----------+----------+--------------+---------
  lkc-m1234  | Dev                 | BASIC     | gcp      | us-west4 | single-zone  | UP
  lkc-r1234  | Test                | BASIC     | gcp      | us-east4 | single-zone  | UP
  lkc-g1234  | Prod                | DEDICATED | gcp      | us-west4 | single-zone  | UP

  4. Set the active Kafka cluster. In this example, the cluster ID is lkc-m1234.

    confluent kafka cluster use lkc-m1234

  5. Run the confluent kafka cluster describe command to get the endpoint for your Confluent Cloud cluster.

    confluent kafka cluster describe

    Your output should resemble:

     +--------------+--------------------------------------------------------+
 | Id           | lkc-m1234                                              |
 | Name         | mqtt-proxy-quickstart                                  |
 | Type         | BASIC                                                  |
 | Ingress      |                                                    100 |
 | Egress       |                                                    100 |
 | Storage      |                                                   5000 |
 | Provider     | gcp                                                    |
 | Availability | single-zone                                            |
 | Region       | us-west2                                               |
 | Status       | UP                                                     |
 | Endpoint     | SASL_SSL://pkc-12345.us-west2.gcp.confluent.cloud:9092 |
 | ApiEndpoint  | https://pkac-12345.us-west2.gcp.confluent.cloud        |
 +--------------+--------------------------------------------------------+

    Save the Endpoint value, which you’ll use to configure the bootstrap server for the MQTT Proxy.

  6. Create a Kafka topic that the MQTT proxy will produce to. Use the Confluent CLI to create a topic named “temperature”.

    confluent kafka topic create temperature

  1. Create a Kafka API key and secret that the MQTT proxy can use to access Confluent Cloud. You must specify the cluster with the resource flag for this step.

    confluent api-key create --resource lkc-m1234

    Your output should resemble:

    It may take a couple of minutes for the API key to be ready.
Save the API key and secret. The secret is not retrievable later.
+---------+------------------------------------------------------------------+
| API Key | ABCXQHYDZXMMUDEF                                                 |
| Secret  | aBCde3s54+4Xv36YKPLDKy2aklGr6x/ShUrEX5D1Te4AzRlphFlr6eghmPX81HTF |
+---------+------------------------------------------------------------------+

    Important

    Save the API key and secret. You need this information to configure your applications that communicate with Confluent Cloud. This is the only time that you can access, view, and save the key and secret.

Configure and start the MQTT Proxy

You can either configure and start the MQTT proxy by downloading Confluent Platform locally, or using a Docker container.

  1. Install Confluent Platform.

  2. Customize the /etc/confluent-kafka-mqtt/kafka-mqtt-dev.properties properties file, specifying:

    • The Confluent Cloud Endpoint that you saved earlier for the bootstrap server.
    • Security information including the Confluent Cloud API key and secret you created in the previous section.
    • Topic information.
    # add bootstrap server
bootstrap.servers=pkc-12345.us-west2.gcp.confluent.cloud:9092

#configure connection to Confluent Cloud
producer.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api-key>" password="<secret>";

# configure topic settings
topic.regex.list=temperature:.*temperature
confluent.topic.replication.factor=3

  3. Start the MQTT proxy, specifying the properties file:

    bin/kafka-mqtt-start etc/confluent-kafka-mqtt/kafka-mqtt-dev.properties

Send messages with the MQTT client

You can now send messages to the MQTT proxy using your client.

  • Use the previously configured MQTT client to publish messages to the MQTT proxy.

    mosquitto_pub -h 0.0.0.0 -p 1883 -t car/engine/temperature -q 2 -m "190F"
mosquitto_pub -h 0.0.0.0 -p 1883 -t car/engine/temperature -q 2 -m "200F"
mosquitto_pub -h 0.0.0.0 -p 1883 -t car/engine/temperature -q 2 -m "210F"

View MQTT messages in Confluent Cloud

Sign in to the Cloud Console, and access the temperatures topic to see the MQTT messages.

Click the Messages tab, and you should see the messages sent from your client to the MQTT Proxy and then to Confluent Cloud.

../_images/ccloud-mqtt-messages.png