Get Started with Confluent Cloud for Government¶
Confluent Cloud for Government is a data streaming service based on Apache Kafka® and delivered as a fully-managed, cloud-native service. Use Confluent Cloud for Government to collect real-time data from multiple sources and put it in motion. Confluent Cloud for Government provides multiple interfaces to manage your data streams, including a web interface, a command-line interface, and APIs.
Use this quick start to create a Kafka cluster and a topic. Connect a self-managed data source to the cluster and produce data to the topic. Then use the Cloud Console to review the incoming data, similar to how a self-managed client would consume the data you’re producing. This quick start lists all the interfaces available to complete a step, but you should only use the interface that best fits your needs.
Prerequisites¶
You must contact Confluent to setup a Confluent Cloud for Government organization before you can use the service.
Step 1: Create a Kafka cluster in Confluent Cloud for Government¶
In this step, you create an environment , select a region, and then create and launch a dedicated Kafka cluster inside your new environment.
- Log into the Confluent Cloud for Government Console. For more information, see Log in to Confluent Cloud for Government.
- From Home, click View environments, and select the environment where you want to create a cluster, or click Add cloud environment and enter a name to create a new environment.
- Click Add cluster. If this is your fist cluster, click Create cluster.
- In Dedicated, leave Price as sized set to 1 CKU and click Begin configuration.
- Select the Region for the cluster, leave Availability set to Single zone, and click Continue.
- Select Internet and click Continue
- Specify Automatic security and click Continue.
- Specify Cluster name, review the details of your configuration, then click Launch cluster.
Log in to Confluent Cloud for Government:
Log in:
confluent login --us-gov
Create a Kafka cluster:
confluent kafka cluster create <name> [flags]
For example:
confluent kafka cluster create quickstart_cluster --cloud "aws" --region "us-gov-west-1" --type dedicated --cku 1
Create a Kafka cluster.
Request:
POST /cmk/v2/clusters
Host: api.confluentgov.com
{
"spec": {
"display_name": "quickstart_cluster",
"availability": "SINGLE_ZONE",
"cloud": "AWS",
"region": "us-gov-west-1",
"config": {
"kind": "Dedicated",
"cku": 1
},
"environment": {
"id": "env-000000"
}
}
}
Response:
{
"api_version": "cmk/v2",
"id": "lkc-12345",
"kind": "Cluster",
"metadata": {
"created_at": "2023-06-02T20:59:25.078112Z",
"resource_name": "crn://confluent.cloud/organization=11cd2222-a214-4b44-8fae-d4cc6example/environment=env-00000/cloud-cluster=lkc-12345/kafka=lkc-12345",
"self": "https://api.confluentgov.com/cmk/v2/clusters/lkc-12345",
"updated_at": "2023-06-02T20:59:25.077444Z"
},
"spec": {
"api_endpoint": "",
"availability": "SINGLE_ZONE",
"cloud": "AWS",
"config": {
"cku": 1,
"kind": "Dedicated",
"zones": [
"usgw1-az3"
]
},
"display_name": "quickstart_cluster",
"environment": {
"api_version": "org/v2",
"id": "env-00000",
"kind": "Environment",
"related": "https://api.confluentgov.com/org/v2/environments/env-00000",
"resource_name": "crn://confluent.cloud/organization=11cd2222-a214-4b44-8fae-d4cc6example/environment=env-12345"
},
"http_endpoint": "",
"kafka_bootstrap_endpoint": "",
"region": "us-gov-west-1"
},
"status": {
"cku": 1,
"phase": "PROVISIONING"
}
}
Tip
For more information about private networking and self-managed encryption keys, see links in Related content.
Step 2: Create a Kafka topic¶
In this step, you create a topic. A Kafka topic is a unit of organization for a cluster, and is essentially an append-only log. For more about topics, see What is Apache Kafka.
- From the cluster you just created, click Topics, and then click Create topic. If you already have a topic, click Add topic.
- Type a name in Topic name and click Create with defaults.
The topic is created on the Kafka cluster and is available for use by producers and consumers.
Create a topic:
confluent kafka topic create <name> [flags]
For example:
confluent kafka topic create my-topic --cluster lkc-12345
Create a topic:
Request:
POST /kafka/v3/clusters/{cluster_id}/topics
Host: pkc-{00000}.{region}.aws.confluentgov.com
{
"topic_name": "my-topic",
"partitions_count": 6,
"replication_factor": 3,
"configs": [{
"name": "cleanup.policy",
"value": "delete"
},
{
"name": "compression.type",
"value": "gzip"
}
]
}
Response:
{
"kind": "KafkaTopic",
"metadata": {
"self": "https://pkc-abc45.us-gov-west-1.aws.confluentgov.com/kafka/v3/clusters/lkc-12345/topics/users",
"resource_name": "crn:///kafka=lkc-12345/topic=my-topic"
},
"cluster_id": "lkc-12345",
"topic_name": "my-topic",
"is_internal": false,
"replication_factor": 3,
"partitions_count": 6,
"partitions": {
"related": "https://pkc-abc45.us-gov-west-1.aws.confluentgov.com/kafka/v3/clusters/lkc-12345/topics/my-topic/partitions"
},
"configs": {
"related": "https://pkc-abc45.us-gov-west-1.aws.confluentgov.com/kafka/v3/clusters/lkc-12345/topics/my-topic/configs"
},
"partition_reassignments": {
"related": "https://pkc-abc45.us-gov-west-1.aws.confluentgov.com/kafka/v3/clusters/lkc-12345/topics/my-topic/partitions/-/reassignment"
},
"authorized_operations": []
}
Step 3: Create a self-managed producer¶
You write Kafka client applications to connect to Confluent Cloud for Government in any supported language. Configure the clients using the Confluent Cloud for Government cluster credentials.
Federal Information Processing Standard (FIPS) is a U.S. government computer security standard used to validate and certify software cryptographic modules. Confluent has tested implementations of the following clients for FIPS-compliance:
- Select Clients from the navigation menu.
- Select the language you are using for your client application.
- After you select a language, create a Kafka cluster API key.
- Copy and paste the displayed configuration into your client application source code.
- Create a producer and produce to the topic you created in the previous step.
Run the following command to create the API key and secret, using the ID (
<cluster-id>).confluent api-key create --resource <cluster-id>
Your output should resemble:
It may take a couple of minutes for the API key to be ready. Save the API key and secret. The secret is not retrievable later. +---------+------------------------------------------------------------------+ | API Key | ABC123xyz | | Secret | 123xyzABC123xyzABC123xyzABC123xyzABC123xyzABC123xyzABC123xyzABCx | +---------+------------------------------------------------------------------+
Set the API key to use for Confluent CLI commands, using the ID (
<cluster-id>).confluent api-key use <api-key> --resource <cluster-id>
Run the following command to create a client configuration file for the language of your choice:
confluent kafka client-config create <language-id> \ --environment <env-id> \ --cluster <cluster-id> \ --api-key <api-key> --api-secret <api-secret>
Use a
<language-id>from the table below to specify a language:Language Language ID Examples Java javaPython pythonconfluent-kafka-python//examples Copy and paste the displayed configuration into your client application source code.
For more information, see confluent kafka client-config create.
Step 4: Consume messages¶
Your new topic is now consuming messages. Use Confluent Cloud Console to see the data.
- From the navigation menu, select Topics to show the list of topics in your cluster.
- Select the topic you created.
- In the topic detail page, select Messages to view the messages being produced to the topic.
- Use the producer you created to produce messages to the topic. The message viewer only shows messages produced since you loaded the page.
Step 5: Clean up¶
Delete the topic you created to avoid unexpected charges to your account.
- From the navigation menu, click Topics, select the users topic, and then choose the Configuration tab.
- Click Delete topic, enter the topic name (
users), and select Continue.
Delete the topic:
confluent kafka topic delete <topic name> [flags]
For example:
confluent kafka topic delete users --cluster lkc-12345
Delete a topic:
DELETE /kafka/v3/clusters/{kafka_cluster_id}/topics/{topic_name}
Host: pkc-{0000}.{region}.aws.confluentgov.com