Amazon Kinesis Source Connector for Confluent Cloud

Note

If you are installing the connector locally for Confluent Platform, see Amazon Kinesis Source Connector for Confluent Platform.

The Kafka Connect Amazon Kinesis Source connector is used to pull data from Amazon Kinesis and persist the data to an Apache Kafka® topic.

Features

The Amazon Kinesis Source connector for Confluent Cloud provides the following features:

  • Fetches records from all shards in one Kinesis stream.
  • Select configuration properties:
    • Offset position:
      • AT_SEQUENCE_NUMBER
      • AFTER_SEQUENCE_NUMBER
      • TRIM_HORIZON
      • LATEST
      • AT_TIMESTAMP
    • Other properties:
      • kinesis.region
      • kinesis.record.limit
      • kinesis.throughput.exceeded.backoff.ms

Configuration properties that are not shown in the Confluent Cloud UI use the default values. For default values and property definitions, see Kinesis Source Connector Configuration Properties.

For more information, see the Confluent Cloud connector limitations.

Quick Start

Use this quick start to get up and running with the Confluent Cloud Kinesis source connector. The quick start shows how to select the connector and configure it to pull data from Amazon Kinesis and persist the data to an Apache Kafka® topic. It then monitors and records all subsequent row-level changes.

Prerequisites
  • Authorized access to a Confluent Cloud cluster on Amazon Web Services (AWS).
  • At least one topic must exist before creating the connector.
  • The Confluent Cloud CLI installed and configured for the cluster. See Install and Configure the Confluent Cloud CLI.
  • An AWS account configured with Access Keys. You use these access keys when setting up the connector.
  • An available Amazon Kinesis Data Stream.
  • Use one of the following for the Kafka cluster credentials fields:
    • A Confluent Cloud API key and secret. After you have created your cluster, go to Cluster settings > API access > Create Key.
    • A Confluent Cloud service account.

Step 1: Launch your Confluent Cloud cluster.

See the Confluent Cloud Quick Start for installation instructions.

Step 2: Add a connector.

Click Connectors > Add connector.

../../_images/ccloud-add-connector.png

Step 3: Select your connector.

Click the Kinesis Source connector icon.

Step 4: Set up the connection.

Complete the following and click Continue.

Note

Make sure you have all your prerequisites completed.

  1. Enter a connector name.
  2. Enter your Kafka Cluster credentials. The credentials are either the API key and secret or the service account API key and secret.
  3. Enter the topic name or topic names where you want to send data.
  4. Enter your AWS credentials.
  5. Add the Kinesis details. The stream offset position is where messages start being consumed from the Kinesis stream.
  6. Add the Connection details for your connection to the stream.
  7. Enter the number of tasks in use by the connector. Refer to Confluent Cloud connector limitations for additional information.

Step 5: Launch the connector.

Verify the connection details and click Launch.

../../_images/ccloud-kinesis-source-launch-connector.png

Step 6: Check the connector status.

The status for the connector should go from Provisioning to Running. It may take a few minutes.

../../_images/ccloud-kinesis-source-status.png

Step 7: Check the Kafka topic.

After the connector is running, verify that messages are populating your Kafka topic.

For more information about this connector see, Amazon Kinesis Source Connector for Confluent Platform. Note that not all Confluent Platform connector features are provided in the Confluent Cloud connector.

Next Steps

Try out a Confluent Cloud demo.