Confluent Platform Quick Start (Docker)

This quick start shows you how to get up and running with Confluent Platform and its main components using Docker containers. This quick start will show you the basic and most powerful capabilities of Confluent Platform, including using Control Center for topic management and stream processing by using KSQL. In this quick start, you create Kafka topics, use Kafka Connect to generate mock data to these topics, and create KSQL streaming queries on these topics. You will then go to Control Center to monitor and analyze the streaming queries.

You can also run an automated version of this quick start designed for Confluent Platform local installs.

Prerequisites
  • Docker
    • Docker version 1.11 or later is installed and running.
    • Docker Compose is installed. It is installed by default with Docker for Mac and Windows.
    • Docker memory resource is allocated minimally at 8 GB.
  • Git
  • Internet connectivity

Step 1: Download and Start Confluent Platform Using Docker

  1. Clone the Confluent Platform Docker Images GitHub Repository and checkout the 5.1.0-post branch.

    git clone https://github.com/confluentinc/cp-docker-images
    git checkout 5.1.0-post
    
  2. Navigate to cp-all-in-one examples directory.

    cd cp-docker-images/examples/cp-all-in-one/
    
  3. Start Confluent Platform specifying two options: (-d) to run in detached mode and (--build) to build the Kafka Connect image with the source connector kafka-connect-datagen from Confluent Hub .

    Important

    You must allocate a minimum of 8 GB of Docker memory resource.

    docker-compose up -d --build
    

    This starts the Confluent Platform with separate containers for all Confluent Platform components. Your output should resemble the following:

    Creating network "cp-all-in-one_default" with the default driver
    Creating zookeeper ... done
    Creating broker    ... done
    Creating schema-registry ... done
    Creating rest-proxy      ... done
    Creating connect         ... done
    Creating ksql-datagen    ... done
    Creating ksql-server     ... done
    Creating control-center  ... done
    Creating ksql-cli        ... done
    
  4. Optional: Run this command to verify that the services are up and running.

    docker-compose ps
    

    You should see the following:

         Name                    Command               State                Ports
    ------------------------------------------------------------------------------------------
    broker            /etc/confluent/docker/run        Up      0.0.0.0:29092->29092/tcp,
                                                               0.0.0.0:9092->9092/tcp
    connect           /etc/confluent/docker/run        Up      0.0.0.0:8083->8083/tcp,
                                                               9092/tcp
    control-center    /etc/confluent/docker/run        Up      0.0.0.0:9021->9021/tcp
    ksql-cli          ksql http://localhost:8088       Up
    ksql-datagen      bash -c echo Waiting for K ...   Up
    ksql-server       /etc/confluent/docker/run        Up      0.0.0.0:8088->8088/tcp
    rest-proxy        /etc/confluent/docker/run        Up      0.0.0.0:8082->8082/tcp
    schema-registry   /etc/confluent/docker/run        Up      0.0.0.0:8081->8081/tcp
    zookeeper         /etc/confluent/docker/run        Up      0.0.0.0:2181->2181/tcp,
                                                               2888/tcp, 3888/tcp
    

    If the state is not Up, rerun the docker-compose up -d command.

Step 2: Create Kafka Topics

In this step you create Kafka topics by using the Confluent Control Center. Confluent Control Center provides the functionality for building and monitoring production data pipelines and streaming applications.

  1. Navigate to the Control Center web interface at http://localhost:9021/.

    ../_images/c3-landing-page.png
  2. Select Management -> Topics and click Create topic.

    ../_images/c3-create-topic.png
  3. Create a topic named pageviews and click Create with defaults.

    ../_images/c3-create-topic-name.png
  4. Create a topic named users and click Create with defaults.

    ../_images/c3-create-topic-name-2.png

Step 3: Install a Kafka Connector and Generate Sample Data

In this step, you use Kafka Connect to run a demo source connector called kafka-connect-datagen that creates sample data for the Kafka topics pageviews and users.

  1. Run one instance of the kafka-connect-datagen connector to produce Kafka data to the pageviews topic in JSON format.

    1. From the Control Center interface, click Management -> Kafka Connect.

      ../_images/connect-page-gui-11.png
    2. Click the Add connector button and scroll down to select DatagenConnector.

      Tip

      The DatagenConnector was installed manually in step 1.

      ../_images/connect-page-new-source1.png
    3. Name the connector datagen-pageviews. After you name the connector, new fields will appear. Scroll down to General and configure this connector.

      • In the kafka.topic field, type pageviews
      • In the max.interval field, type 100
      • In the iterations field, type 1000000
      • In the quickstart field, type pageviews
      ../_images/connect-configure-pageviews.png
    4. Click Continue.

    5. Review the connector configuration and click Launch.

      ../_images/connect-review-pageviews1.png
  2. Run another instance of the kafka-connect-datagen connector to produce Kafka data to the users topic in JSON format.

    1. Click the Add connector button from the upper right-hand corner.

      ../_images/connect-page-gui.png
    2. Click the Add connector button and scroll down to select DatagenConnector.

      Tip

      The DatagenConnector was installed manually in step 1.

      ../_images/connect-page-new-source1.png
    3. Name the connector datagen-users. After you name the connector, new fields will appear. Scroll down to General and configure this connector.

      • In the kafka.topic field, type users
      • In the max.interval field, type 1000
      • In the iterations field, type 1000000
      • In the quickstart field, type users
      ../_images/connect-configure-users.png
    4. Click Continue.

    5. Review the connector configuration and click Launch.

      ../_images/connect-review-users.png

Step 4: Create and Write to a Stream and Table using KSQL

In this step, KSQL queries are run on the pageviews and users topics that were created in the previous step. The KSQL commands are run using the KSQL tab in Control Center.

Tip

You can also run these commands using the KSQL CLI from your Docker container with this command: docker-compose exec ksql-cli ksql http://ksql-server:8088.

Create Streams and Tables

In this step, KSQL is used to create streams and tables for the pageviews and users topics.

  1. From the Control Center interface, click Development -> KSQL. By default your are on the KSQL EDITOR page.

    ../_images/ksql-interface-create-stream.png
  2. Click the STREAMS tab and Add a stream and select the pageviews topic.

    ../_images/c3-ksql-create-stream-pageview.png
  3. Choose your stream options and click Save STREAM.

    • In the How are your messages encoded? field, select JSON.
    • In the Field(s) you’d like to include in your STREAM field, add additional fields for:
      • viewtime with type BIGINT
      • userid with type VARCHAR
      • pageid with type VARCHAR
    ../_images/c3-ksql-create-stream-pageview-2.png
  4. Click the TABLES tab and Add a table and then select the users topic.

    ../_images/c3-ksql-create-stream-users.png
  5. Choose your table options and click Save TABLE.

    • In the How are your messages encoded? field, select JSON.
    • In the Key field, select userid.
    • In the Field(s) you’d like to include in your TABLE field, add additional fields for:
      • registertime with type BIGINT
      • gender with type VARCHAR
      • regionid with type VARCHAR
      • userid with type VARCHAR
      • interests with type ARRAY<VARCHAR>
      • contact_info with type MAP<VARCHAR, VARCHAR>
    ../_images/c3-ksql-create-table-users.png

Write Queries

These examples write queries using the KSQL tab in Control Center.

  1. From the Control Center interface, click Development -> KSQL. By default your are on the KSQL EDITOR page. Add the custom query property earliest for the auto.offset.reset parameter. This instructs KSQL queries to read all available topic data from the beginning. This configuration is used for each subsequent query. For more information, see the KSQL Configuration Parameter Reference.

    ../_images/ksql-interface-create-stream.png
  2. Run the following queries.

    1. Create a query that returns data from a stream with the results limited to three rows.

    2. Create a persistent query that filters for female users. The results from this query are written to the Kafka PAGEVIEWS_FEMALE topic. This query enriches the pageviews STREAM by doing a LEFT JOIN with the users TABLE on the user ID, where a condition (gender = 'FEMALE') is met.

      CREATE STREAM pageviews_female AS SELECT users.userid AS userid, pageid, regionid, gender FROM pageviews LEFT JOIN users ON pageviews.userid = users.userid WHERE gender = 'FEMALE';
      
    3. Create a persistent query where a condition (regionid) is met, using LIKE. Results from this query are written to a Kafka topic named pageviews_enriched_r8_r9.

      CREATE STREAM pageviews_female_like_89 WITH (kafka_topic='pageviews_enriched_r8_r9', value_format='JSON') AS SELECT * FROM pageviews_female WHERE regionid LIKE '%_8' OR regionid LIKE '%_9';
      
    4. Create a persistent query that counts the pageviews for each region and gender combination in a tumbling window of 30 seconds when the count is greater than 1. Because the procedure is grouping and counting, the result is now a table, rather than a stream. Results from this query are written to a Kafka topic called PAGEVIEWS_REGIONS.

      CREATE TABLE pageviews_regions AS SELECT gender, regionid , COUNT(*) AS numusers FROM pageviews_female WINDOW TUMBLING (size 30 second) GROUP BY gender, regionid HAVING COUNT(*) > 1;
      
    5. Click RUNNING QUERIES and you should see the following:

      ../_images/c3-ksql-persistent-query.png

Step 5: View Your Stream in Control Center

From the Control Center interface you can view all of your streaming KSQL queries.

Navigate to the Control Center web interface Monitoring -> Data streams tab at http://localhost:9021/monitoring/streams. The monitoring page shows the total number of messages produced and consumed on the cluster. You can scroll down to see more details on the consumer groups for your queries.

Tip

Depending on your machine, these charts may take a few minutes to populate and you might need to refresh your browser.

../_images/c3-monitor-ksql1.png

Now that your streams are running you can monitor them.

  • View the consumers that have been created by KSQL

    Click the CONSUMER GROUPS tab and then the View Details button for the _confluent-ksql-default_query_CSAS_PAGEVIEWS_FEMALE_0 consumer group.

    ../_images/ksql-interface-monitor-cnsmgp.png

    This graph shows the messages being consumed by the stream query.

    ../_images/ksql-interface-monitor-pgvw-f.png

Next Steps

Learn more about the components shown in this quick start: