Tutorial: User Management in Confluent Cloud

This tutorial provides an end-to-end workflow for Confluent Cloud user and service account management. In this example, a Confluent Cloud user account is created. The user then creates topics, a service account for applications, and then access control lists (ACLs) to authorize access.

Step 1: Invite User

A Confluent Cloud administrator invites a user by email.

../_images/cloud-invite-user-email.png

Step 2: Login to Confluent Cloud Web Browser and CLI

  1. User accepts the invitation from email and logs in using web browser.

  2. Install the Confluent Cloud CLI.

  3. Log in to the Confluent Cloud CLI using the ccloud login command with the cluster URL specified.

    ccloud login
    

    Specify your credentials.

    Enter your Confluent Cloud credentials:
    Email: jane.smith@big-data.com
    Password: ************
    

Step 3: Configure CLI and Connect to Cluster

  1. List the available clusters using the ccloud kafka cluster list command.

    ccloud kafka cluster list
    

    The output should resemble:

         ID     |          NAME          | PROVIDER |   REGION    | DURABILITY | STATUS
    +-----------+------------------------+----------+-------------+------------+---------+
      lkc-43npm | Cluster1               | aws      | us-west-2   | LOW        | UP
      lkc-lq8dd | Cluster2               | aws      | us-west-2   | LOW        | UP
      lkc-43nkw | Cluster3               | aws      | us-west-2   | LOW        | DELETED
      lkc-4xrp1 | Cluster4               | gcp      | us-central1 | LOW        | UP
    
  2. Connect to Cluster4 (lkc-4xrp1) using the ccloud kafka cluster use command.

    ccloud kafka cluster use lkc-4xrp1
    

Step 4: Configure Access to Kafka

  1. Log in to your cluster using the ccloud login command with the cluster URL specified.

    ccloud login
    
    Enter your Confluent Cloud credentials:
    Email: susan@myemail.com
    Password:
    
  2. Set the Confluent Cloud environment.

    1. Get the environment ID.

      ccloud environment list
      

      Your output should resemble:

           Id    |      Name
      +----------+----------------+
        * a-542  | dev
          a-4985 | prod
          a-2345 | jdoe-gcp-env
          a-9012 | jdoe-aws-env
      
    2. Set the environment using the ID (<env-id>).

      ccloud environment use <env-id>
      

      Your output should resemble:

      Now using a-4985 as the default (active) environment.
      
  3. Set the cluster to use.

    1. Get the cluster ID.

      ccloud kafka cluster list
      

      Your output should resemble:

            Id      |       Name        | Provider |   Region    | Durability | Status
      +-------------+-------------------+----------+-------------+------------+--------+
          ekg-rr8v7 | dev-aws-oregon    | aws      | us-west-2   | LOW        | UP
          ekg-q2j96 | prod              | gcp      | us-central1 | LOW        | UP
      
    2. Set the cluster using the ID (<cluster-id>). This is the cluster where the commands are run.

      ccloud kafka cluster use <cluster-id>
      
  4. Create an API key and secret, and save them. This is required to produce or consume to your topic.

    You can generate the API key from the Confluent Cloud web UI or on the Confluent Cloud CLI. Be sure to save the API key and secret.

    • On the web UI, click the Kafka API keys tab and click Create key. Save the key and secret, then click the checkbox next to I have saved my API key and secret and am ready to continue.

      ../_images/cloud-api-key-confirm.png
    • Or, from the Confluent Cloud CLI, type the following command:

      ccloud api-key create --resource <resource-id>
      

      Your output should resemble:

      Save the API key and secret. The secret is not retrievable later.
      +---------+------------------------------------------------------------------+
      | API Key | LD35EM2YJTCTRQRM                                                 |
      | Secret  | 67JImN+9vk+Hj3eaj2/UcwUlbDNlGGC3KAIOy5JNRVSnweumPBUpW31JWZSBeawz |
      +---------+------------------------------------------------------------------+
      
  5. Optional: Add the API secret with ccloud api-key store <key> <secret>. When you create an API key with the CLI, it is automatically stored locally. However, when you create an API key using the UI, API, or with the CLI on another machine, the secret is not available for CLI use until you store it. This is required because secrets cannot be retrieved after creation.

    ccloud api-key store <api-key> <api-secret> --resource <resource-id>
    
  6. Set the API key to use for Confluent Cloud CLI commands with the command ccloud api-key use <key> --resource <resource-id>.

    ccloud api-key use <api-key> --resource <resource-id>
    

Step 5: Create and Manage Topics

  1. Create a topic with all the default values using the ccloud kafka topic create command.

    ccloud kafka topic create myTopic1
    
  2. Create a topic with six partitions and a replication factor of three.

    ccloud kafka topic create myTopic2 --partitions 6
    
  3. List topics using the ccloud kafka topic list command.

    ccloud kafka topic list
    

    The output should resemble:

        NAME
    +----------+
      myTopic1
      myTopic2
    
  4. Delete a topic named myTopic1 using the ccloud kafka topic delete command.

    ccloud kafka topic delete myTopic1
    
  5. Describe a topic using the ccloud kafka topic describe command.

    ccloud kafka topic describe myTopic2
    

    The output should resemble:

    Topic: myTopic2 PartitionCount: 6 ReplicationFactor: 3
       TOPIC   | PARTITION | LEADER | REPLICAS |   ISR
    +----------+-----------+--------+----------+---------+
      myTopic2 |         0 |      2 | [2 1 3]  | [2 1 3]
      myTopic2 |         1 |      3 | [3 2 0]  | [3 2 0]
      myTopic2 |         2 |      0 | [0 3 1]  | [0 3 1]
      myTopic2 |         3 |      1 | [1 0 2]  | [1 0 2]
      myTopic2 |         4 |      2 | [2 3 0]  | [2 3 0]
      myTopic2 |         5 |      3 | [3 0 1]  | [3 0 1]
    
  6. Modify the myTopic2 configuration to set cleanup.policy using the ccloud kafka topic update command.

    ccloud kafka topic update myTopic2 --config cleanup.policy=compact
    

Step 6: Produce and consume

  1. Produce messages to a topic using the ccloud kafka topic produce when you are done.

    ccloud kafka topic produce myTopic2
    
  2. Consume messages from a topic using the ccloud kafka topic consume when you are done.

    ccloud kafka topic consume myTopic2
    

Step 7: Create Service Accounts and API Key/Secret Pairs

  1. Create a service account named dev-apps using the ccloud service-account create command.

    ccloud service-account create "dev-apps" \
    --description "Service account for dev apps"
    

    The output should resemble:

    +----------------+--------------------------------+
    | Id             |                           1629 |
    | Name           | dev-apps                       |
    | Description    | Service account for dev apps   |
    | OrganizationId |                            857 |
    +----------------+--------------------------------+
    

    Note the Id associated to this service account, in this case 1629.

  2. Create an API key/secret pair for this service account using the Cloud API key command. It also needs the cluster ID, which is available from the output of ccloud kafka cluster list.

    ccloud api-key create --service-account 1629 --resource lkc-4xrp1
    
  3. Take note of the API key and secret, this is the only time you will be able to see it.

  4. Client applications that will connect to this cluster will need to configure at least these three identifying parameters:

    • API key: available when you create the API key/secret pair the first time
    • API secret: available when you create the API key/secret pair the first time
    • bootstrap.servers: set to the Endpoint in the output of ccloud kafka cluster describe

Step 8: Manage Access with ACLs

  1. Grant the dev-apps service account the ability to produce to a particular topic using the ccloud kafka acl create command.

    ccloud kafka acl create --allow --service-account 1629 --operation WRITE --topic myTopic2
    
  2. If the service also needs to create topics, grant the dev-apps service account the ability to create new topics.

    ccloud kafka acl create --allow --service-account 1629 --operation CREATE --topic myTopic2
    
  3. Grant the dev-apps service account the ability to consume from a particular topic using the ccloud kafka acl create command. Note that it requires two commands: one to specify the topic and one to specify the consumer group.

    ccloud kafka acl create --allow --service-account 1629 --operation READ --topic myTopic2
    ccloud kafka acl create --allow --service-account 1629 --operation READ --consumer-group java_example_group_1
    
  4. List all ACLs for the dev-apps service account using the ccloud kafka acl list command.

    ccloud kafka acl list --service-account 1629
    

    The output should resemble:

      SERVICEACCOUNTID | PERMISSION | OPERATION | RESOURCE |        NAME
    +------------------+------------+-----------+----------+----------------------+
      User:1629        | ALLOW      | WRITE     | TOPIC    | myTopic2
      User:1629        | ALLOW      | CREATE    | TOPIC    | myTopic2
      User:1629        | ALLOW      | READ      | TOPIC    | myTopic2
      User:1629        | ALLOW      | READ      | GROUP    | java_example_group_1
    
  5. You can add ACLs on prefixed resource patterns. For example, you can add an ACL for any topic whose name starts with demo

    ccloud kafka acl create --allow --service-account 1629 --operation WRITE --topic demo --prefix
    
  6. You can add ACLs using a wildcard which matches any name for that resource. For example, you can add an ACL to allow a topic of any name.

    ccloud kafka acl create --allow --service-account 1629 --operation WRITE --topic '*'
    
  7. Remove an ACL from the dev-apps service account using the ccloud kafka acl delete command.

    ccloud kafka acl delete --allow --service-account 1629 --operation WRITE --topic myTopic2
    

Step 9: Logout

Log out using the ccloud logout command.

ccloud logout