# Amazon Kinesis Source Connector for Confluent Cloud The fully managed Amazon Kinesis Source connector for Confluent Cloud pulls data from Amazon Kinesis and writes the data to Apache Kafka® topics. #### NOTE * This Quick Start is for the fully managed Confluent Cloud connector. If you are installing the connector locally for Confluent Platform, see [Amazon Kinesis Source Connector for Confluent Platform](https://docs.confluent.io/kafka-connectors/kinesis/current/). * If you require private networking for fully managed connectors, make sure to set up the proper networking beforehand. For more information, see [Manage Networking for Confluent Cloud Connectors](networking/internet-resource.md#clusters-connect-cloud). ## Features The Amazon Kinesis Source connector provides the following features: * **Topics created automatically**: The connector can automatically create Kafka topics. * **Fetches records from all shards** in one Kinesis stream. * **Select configuration properties**: - Offset position: - `AT_TIMESTAMP` - `LATEST` - `TRIM_HORIZON` - `kinesis.shard.timestamp.ms` - Other properties: - `kinesis.region` - `kinesis.record.limit` - `kinesis.throughput.exceeded.backoff.ms` * **Offset management capabilities**: Supports offset management. For more information, see [Manage custom offsets](#cc-kinesis-source-custom-offsets). * **Provider integration support**: The connector supports IAM role-based authorization using Confluent Provider Integration. For more information about provider integration setup, see the [IAM roles authentication](#cc-kinesis-source-setup-connection). For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## Limitations Be sure to review the following information. * For connector limitations, see [Amazon Kinesis Source Connector](limits.md#cc-kinesis-source-limits) limitations. * If you plan to use one or more Single Message Transformations (SMTs), see [SMT Limitations](single-message-transforms.md#cc-single-message-transforms-limitations). ## Manage custom offsets You can manage the offsets for this connector. Offsets provide information on the point in the system from which the connector is accessing data. For more information, see [Manage Offsets for Fully Managed Connectors in Confluent Cloud](offsets.md#connect-custom-offsets). **To manage offsets**: - Manage offsets using Confluent Cloud APIs. For more information, see [Connect offsets API reference](https://docs.confluent.io/cloud/current/ccloud/offsets-connect-v-1/). ### Get the current offset To get the current offset, make a `GET` request that specifies the environment, Kafka cluster, and connector name. ```bash GET /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets Host: https://api.confluent.cloud ``` **Response:** Successful calls return HTTP `200` with a JSON payload that describes the offset. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [ { "partition": { "kinesis.shard.id": "shardId-123400000000", "kinesis.stream.name": "my-kinesis-stream123" }, "offset": { "kinesis.sequence.number": "4965198826755595916031282174506905389407012937123456789", "kinesis.subsequence.number": 0 } } ], "metadata": { "observed_at": "2024-03-28T17:57:48.139635200Z" } } ``` Responses include the following information: - The position of latest offset. - The observed time of the offset in the metadata portion of the payload. The `observed_at` time indicates a snapshot in time for when the API retrieved the offset. A running connector is always updating its offsets. Use `observed_at` to get a sense for the gap between real time and the time at which the request was made. By default, offsets are observed every minute. Calling `GET` repeatedly will fetch more recently observed offsets. - Information about the connector. - In these examples, the curly braces around “{connector_name}” indicate a replaceable value. ### Update the offset To update the offset, make a `POST` request that specifies the environment, Kafka cluster, and connector name. Include a JSON payload that specifies new offset and a patch type. ```bash POST /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request Host: https://api.confluent.cloud { "type": "PATCH", "offsets": [ { "partition": { "kinesis.shard.id": "shardId-123400000000", "kinesis.stream.name": "my-kinesis-stream123" }, "offset": { "kinesis.sequence.number": "49651988267555959160312821747517128678789842239469125634", "kinesis.subsequence.number": 0 } } ] } ``` **Considerations:** - You can only make one offset change at a time for a given connector. - This is an asynchronous request. To check the status of this request, you must use the check offset status API. For more information, see **Get the status of an offset request**. - For source connectors, the connector attempts to read from the position defined by the requested offsets. **Response:** Successful calls return HTTP `202 Accepted` with a JSON payload that describes the offset. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [ { "partition": { "kinesis.shard.id": "shardId-123400000000", "kinesis.stream.name": "my-kinesis-stream123" }, "offset": { "kinesis.sequence.number": "49651988267555959160312821747517128678789842239469125634", "kinesis.subsequence.number": 0 } } ], "requested_at": "2024-03-28T17:58:45.606796307Z", "type": "PATCH" } ``` Responses include the following information: - The requested position of the offsets in the source. - The time of the request to update the offset. - Information about the connector. ### Delete the offset To delete the offset, make a `POST` request that specifies the environment, Kafka cluster, and connector name. Include a JSON payload that specifies the delete type. ```bash POST /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request Host: https://api.confluent.cloud { "type": "DELETE" } ``` **Considerations:** - Delete requests delete the offset for the provided partition and reset to the base state. A delete request is as if you created a fresh new connector. - This is an asynchronous request. To check the status of this request, you must use the check offset status API. For more information, see **Get the status of an offset request**. - Do not issue delete and patch requests at the same time. - For source connectors, the connector attempts to read from the position defined in the base state. **Response**: Successful calls return HTTP `202 Accepted` with a JSON payload that describes the result. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [], "requested_at": "2024-03-28T17:59:45.606796307Z", "type": "DELETE" } ``` Responses include the following information: - Empty offsets. - The time of the request to delete the offset. - Information about the Kafka cluster and connector. - The type of request. ### Get the status of an offset request To get the status of a previous offset request, make a `GET` request that specifies the environment, Kafka cluster, and connector name. ```bash GET /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request/status Host: https://api.confluent.cloud ``` **Considerations:** - The status endpoint always shows the status of the most recent PATCH/DELETE operation. **Response**: Successful calls return HTTP `200` with a JSON payload that describes the result. The following is an example of an applied patch. ```bash { "request": { "id": "lcc-example123", "name": "{connector_name}", "offsets": [], "requested_at": "2024-03-28T17:58:45.606796307Z", "type": "PATCH" }, "status": { "phase": "APPLIED", "message": "The Connect framework-managed offsets for this connector have been altered successfully. However, if this connector manages offsets externally, they will need to be manually altered in the system that the connector uses." }, "previous_offsets": [ { "partition": { "kinesis.shard.id": "shardId-123400000000", "kinesis.stream.name": "my-kinesis-stream123" }, "offset": { "kinesis.sequence.number": "49651988267555959160312821747521964382068303779824926722", "kinesis.subsequence.number": 0 } } ], "applied_at": "2024-03-28T17:58:48.079141883Z" } ``` Responses include the following information: - The original request, including the time it was made. - The status of the request: applied, pending, or failed. - The time you issued the status request. - The previous offsets. These are the offsets that the connector last updated prior to updating the offsets. Use these to try to restore the state of your connector if a patch update causes your connector to fail or to return a connector to its previous state after rolling back. ### JSON payload The table below offers a description of the unique fields in the JSON payload for managing offsets of the TODO: {NAME} connector. | Field | Definition | Required/Optional | |------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------| | `kinesis.shard.id` | The ID of the Kinesis shard. | Required | | `kinesis.stream.name` | The name of the Kinesis stream. | Required | | `kinesis.sequence.number` | This is the sequence identifier that AWS assigns to the record when it gets persisted in AWS Kinesis data shard. | Required | | `kinesis.subsequence.number` | At times, multiple user generated records get combined into a single Kinesis record due to aggregation by AWS Kinesis Library. This identifier is used to distinguish between such individual user records that have been pushed as a single record in Kinesis Data Stream. | Optional | ## Quick Start Use this quick start to get up and running with the Confluent Cloud Kinesis source connector. The quick start shows how to select the connector and configure it to pull data from Amazon Kinesis and persist the data to an Apache Kafka® topic. It then monitors and records all subsequent row-level changes. Prerequisites : - Authorized access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster on Amazon Web Services (AWS), Microsoft Azure (Azure), or Google Cloud. - The Confluent CLI installed and configured for the cluster. See [Install and Configure the Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html). - For networking considerations, see [Networking and DNS](overview.md#connect-internet-access-resources). To use a set of public egress IP addresses, see [Public Egress IP Addresses for Confluent Cloud Connectors](static-egress-ip.md#cc-static-egress-ips). - An AWS account configured with [Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). You use these access keys when setting up the connector. - An available [Amazon Kinesis Data Stream](https://docs.aws.amazon.com/streams/latest/dev/getting-started.html).
- Kafka cluster credentials. The following lists the different ways you can provide credentials. - Enter an existing [service account](service-account.md#s3-cloud-service-account) resource ID. - Create a Confluent Cloud [service account](service-account.md#s3-cloud-service-account) for the connector. Make sure to review the ACL entries required in the [service account documentation](service-account.md#s3-cloud-service-account). Some connectors have specific ACL requirements. - Create a Confluent Cloud API key and secret. To create a key and secret, you can use [confluent api-key create](https://docs.confluent.io/confluent-cli/current/command-reference/api-key/confluent_api-key_create.html) *or* you can autogenerate the API key and secret directly in the Cloud Console when setting up the connector. ### Using the Confluent Cloud Console #### Step 1: Launch your Confluent Cloud cluster To create and launch a Kafka cluster in Confluent Cloud, see [Create a kafka cluster in Confluent Cloud](../get-started/index.md#cloud-create-kafka-cluster). #### Step 2: Add a connector In the left navigation menu, click **Connectors**. If you already have connectors in your cluster, click **+ Add connector**. #### Step 3: Select your connector Click the **Amazon Kinesis Source** connector card. ![Amazon Kinesis Source Connector Card](images/ccloud-kinesis-source-icon.png) #### Step 4: Enter the connector details #### NOTE * Ensure you have all your [prerequisites](#cc-kinesis-source-prereqs) completed. * An asterisk ( \* ) designates a required entry. At the **Add Amazon Kinesis Source Connector** screen, complete the following: ### Topic selection Select the topic you want to send data to from the **Topics** list. To create a new topic, click **+Add new topic**. ### Kafka access 1. Select the way you want to provide **Kafka Cluster credentials**. You can choose one of the following options: - **My account**: This setting allows your connector to globally access everything that you have access to. With a user account, the connector uses an API key and secret to access the Kafka cluster. This option is not recommended for production. - **Service account**: This setting limits the access for your connector by using a [service account](service-account.md#s3-cloud-service-account). This option is recommended for production. - **Use an existing API key**: This setting allows you to specify an API key and a secret pair. You can use an existing pair or create a new one. This method is not recommended for production environments. #### NOTE Freight clusters support only service accounts for Kafka authentication. 2. Click **Continue**. ### Authentication 1. Configure the authentication properties: - **Kinesis stream name**: In the **Kinesis stream name** field, enter the Kinesis stream name. - **Kinesis region**: In the **Kinesis region** field, enter the AWS region for the Kinesis stream. **AWS credentials** - **Authentication method**: Select how you want to authenticate with AWS: - If you select **Access Keys**, enter your AWS credentials in the **Amazon Access Key ID** and **Amazon Secret Access Key** fields to connect to Amazon Kinesis.For information about how to set these up, see [Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). - If you select **IAM Roles**, choose an existing integration name under **Provider integration name** dropdown that has access to your resource. For more information, see [Manage Provider Integration for Fully Managed Connectors in Confluent Cloud](provider-integration.md#cloud-pi-quickstart). - **Provider Integration**: Select an existing integration that has access to your resource if you choose **IAM Roles** as your authentication method. - **AWS Access Key ID**: Enter the Amazon Access Key that lets the connector access your Kinesis resource if you select **Access Keys** as your authentication method. - **AWS Secret Key**: Enter the Amazon Secret Key that lets the connector access your Kinesis resou rce if you select **Access Keys** as your authentication method. 2. Click **Continue**. ### Configuration #### NOTE * The connector does not convert Kinesis base64-encoded data before storing the data in Kafka. * For all property values and definitions, see [Configuration Properties](#cc-aws-kinesis-source-config-properties). ### **Show advanced configurations** - **Stream offset position**: The position in the stream to reset to if no offsets are stored. - **Kinesis shard timestamp**: Timestamp (the Unix epoch date with precision in milliseconds) after which to start reading records from. To be used only in combination with `kinesis.shard.position=AT_TIMESTAMP`. - **De-aggregate KPL aggregated records**: Set this value as `true` if you want to de-aggregate individual Kinesis Record (aggregated using KPL) into separate Source Record(s). - **Number of records to read per poll**: The number of records to read in each poll of the Kinesis shard. - **Number of ms to backoff when throughput is exceeded**: The number of milliseconds to backoff when a throughput exceeded exception is thrown. - **Number of ms to backoff when stream is empty**: The number of milliseconds to backoff when the stream is empty. **Auto-restart policy** - **Enable Connector Auto-restart**: Enables the auto-restart behavior of the connector and its task in the event of user-actionable errors. Defaults to `true`, enabling the connector to automatically restart in case of user-actionable errors. Set this property to `false` to disable auto-restart for failed connectors. If disabled, you must manually restart the connector. **Additional Configs** - **Value Converter Decimal Format**: Specifies the `JSON` or `JSON_SR` serialization format for Connect `DECIMAL` logical type values with two allowed literals: `BASE64` to serialize `DECIMAL` logical types as base64 encoded binary data, and `NUMERIC` to serialize `DECIMAL` logical type values in `JSON` or `JSON_SR` as a number representing the decimal value. - **Key Converter Schema ID Serializer**: The class name of the schema ID serializer for keys. This is used to serialize schema IDs in the message headers. - **Value Converter Reference Subject Name Strategy**: Sets the subject reference name strategy for values. Valid entries are `DefaultReferenceSubjectNameStrategy` or `QualifiedReferenceSubjectNameStrategy`. You can use this strategy only with `PROTOBUF` format; the default strategy is `DefaultReferenceSubjectNameStrategy`. - **Value Converter Connect Meta Data**: Enables the Connect converter to add its metadata to the output schema. Applies to Avro converters. - **Value Converter Value Subject Name Strategy**: Determines how to construct the subject name under which the value schema is registered with Schema Registry. - **Key Converter Key Subject Name Strategy**: Determines how to construct the subject name for key schema registration. - **Value Converter Schema ID Serializer**: The class name of the schema ID serializer for values. This is used to serialize schema IDs in the message headers. **Transforms** - **Single Message Transformations**: To add a new SMT, see [Add transforms](single-message-transforms.md#cc-single-message-transforms-ui). For more information about unsupported SMTs, see [Unsupported transformations](single-message-transforms.md#cc-single-message-transforms-unsupported-transforms). See [Configuration Properties](#cc-aws-kinesis-source-config-properties) for all property values and definitions. - Click **Continue**. ### Sizing Based on the number of topic partitions you select, you will be provided with a recommended number of tasks. 1. To change the number of tasks, use the Range Slider to select the desired number of tasks. 2. Click **Continue**. ### Review and Launch 1. Verify the connection details by previewing the running configuration. 2. Once you’ve validated that the properties are configured to your satisfaction, click **Launch**. ![Launch the connector](images/ccloud-kinesis-source-launch-connector.png) The status for the connector should go from **Provisioning** to **Running**. ![Check the connector status](images/ccloud-kinesis-source-status.png) #### Step 5: Check the Kafka topic After the connector is running, verify that messages are populating your Kafka topic. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ### Using the Confluent CLI Complete the following steps to set up and run the connector using the Confluent CLI. #### NOTE Make sure you have all your [prerequisites](#cc-kinesis-source-prereqs) completed. #### IMPORTANT You must create topic names before before creating and launching this connector. For this Quick Start example, the database table being sourced is named `kinesis-testing`. Before starting these steps, make sure you create a Kafka topic named `kinesis-testing` using the command below: ```none confluent kafka topic create kinesis-testing ``` #### Step 1: List the available connectors Enter the following command to list available connectors: ```none confluent connect plugin list ``` #### Step 2: List the connector configuration properties Enter the following command to show the connector configuration properties: ```none confluent connect plugin describe ``` The command output shows the required and optional configuration properties. #### Step 3: Create the connector configuration file Create a JSON file that contains the connector configuration properties. The following example shows the required connector properties. ```none { "name" : "confluent-kinesis-source", "connector.class": "KinesisSource", "kafka.auth.mode": "KAFKA_API_KEY", "kafka.api.key": "", "kafka.api.secret" : "", "kafka.topic" : "kinesis-testing", "aws.access.key.id" : "", "aws.secret.key.id": "", "kinesis.stream": "my-kinesis-stream", "kinesis.region" : "us-west-2", "kinesis.position": "AT_TIMESTAMP", "kinesis.shard.timestamp.ms": "1590692978237" "tasks.max" : "1" } ``` Note the following property definitions: * `"name"`: Sets a name for your new connector. * `"connector.class"`: Identifies the connector plugin name. * `"kafka.auth.mode"`: Identifies the connector authentication mode you want to use. There are two options: `SERVICE_ACCOUNT` or `KAFKA_API_KEY` (the default). To use an API key and secret, specify the configuration properties `kafka.api.key` and `kafka.api.secret`, as shown in the example configuration (above). To use a [service account](service-account.md#s3-cloud-service-account), specify the **Resource ID** in the property `kafka.service.account.id=`. To list the available service account resource IDs, use the following command: ```bash confluent iam service-account list ``` For example: ```bash confluent iam service-account list Id | Resource ID | Name | Description +---------+-------------+-------------------+------------------- 123456 | sa-l1r23m | sa-1 | Service account 1 789101 | sa-l4d56p | sa-2 | Service account 2 ``` * `"kinesis.region"`: Identifies the AWS region where the Kinesis data stream is located. Examples are `us-west-2`, `us-east-2`, `ap-northeast-1`, `eu-central-1`, and so on. * (Optional) `"kinesis.position"`: Identifies the stream offset position. This is where messages start being consumed from the Kinesis stream. Available offset positions are: - `AT_TIMESTAMP`: Get records starting at a point in time. Used with the timestamp format below. - `LATEST`: Start with the most recent record. - `TRIM_HORIZON` (default): Start with the last untrimmed record (the oldest record). * (Optional) `"kinesis.shard.timestamp.ms"`: The timestamp format to use when `AT_TIMESTAMP` is selected. Allowed formats are the simple date-time format `yyyy-MM-dd’T’HH:mm:ss.SSSXXX` or epoch time in milliseconds. * `"tasks.max"`: The maximum number of connector [tasks](/platform/current/connect/concepts.html#tasks). **SMTs**: For details about adding SMTs, see the [Single Message Transformations](single-message-transforms.md#cc-single-message-transforms) documentation. For a list of SMTs that are not supported with this connector, see [Unsupported transformations](single-message-transforms.md#cc-single-message-transforms-unsupported-transforms). See [Configuration Properties](#cc-aws-kinesis-source-config-properties) for all property values and definitions. #### Step 4: Load the properties file and create the connector Enter the following command to load the configuration and start the connector: ```none confluent connect cluster create --config-file .json ``` For example: ```none confluent connect cluster create --config-file kinesis-source.json ``` Example output: ```none Created connector confluent-kinesis-source lcc-ix4dl ``` #### Step 5: Check the connector status Enter the following command to check the connector status: ```none confluent connect cluster list ``` Example output: ```none ID | Name | Status | Type +-----------+--------------------------+---------+--------+ lcc-ix4dl | confluent-kinesis-source | RUNNING | source ``` #### Step 6: Check the Kafka topic. After the connector is running, verify that messages are populating your Kafka topic. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## Configuration Properties Use the following configuration properties with the fully managed connector. For self-managed connector property definitions and other details, see the connector docs in [Self-managed connectors for Confluent Platform](/platform/current/connect/kafka_connectors.html). ### How should we connect to your data? `name` : Sets a name for your connector.
* Type: string * Valid Values: A string at most 64 characters long * Importance: high ### Kafka Cluster credentials `kafka.auth.mode` : Kafka Authentication mode. It can be one of KAFKA_API_KEY or SERVICE_ACCOUNT. It defaults to KAFKA_API_KEY mode, whenever possible.
* Type: string * Valid Values: SERVICE_ACCOUNT, KAFKA_API_KEY * Importance: high `kafka.api.key` : Kafka API Key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high `kafka.service.account.id` : The Service Account that will be used to generate the API keys to communicate with Kafka Cluster.
* Type: string * Importance: high `kafka.api.secret` : Secret associated with Kafka API key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high ### Which topic do you want to send data to? `kafka.topic` : Identifies the topic name to write the data to.
* Type: string * Importance: high ### AWS credentials `authentication.method` : Select how you want to authenticate with AWS.
* Type: string * Default: Access Keys * Importance: high `aws.access.key.id` : The Amazon Access Key used to connect to Kinesis.
* Type: password * Importance: high `provider.integration.id` : Select an existing integration that has access to your resource. In case you need to integrate a new IAM role, use provider integration
* Type: string * Importance: high `aws.secret.key.id` : The Amazon Secret Key used to connect to Kinesis.
* Type: password * Importance: high ### Kinesis details `kinesis.region` : The AWS region for the Kinesis stream.
* Type: string * Default: us-west-2 * Importance: high `kinesis.stream` : The Kinesis stream to read from.
* Type: string * Importance: high `kinesis.shard.timestamp` : Timestamp (the Unix epoch date with precision in milliseconds) after which to start reading records from. To be used only in combination with kinesis.shard.position=AT_TIMESTAMP. Allowed formats: yyyy-MM-dd’T’HH:mm:ss.SSSXXX or epoch time in ms. Note: this will apply to every specified shard in the stream.
* Type: string * Importance: low `kinesis.position` : The position in the stream to reset to if no offsets are stored.
* Type: string * Default: TRIM_HORIZON * Importance: low `kinesis.record.deaggregation.enable` : Set this value as true if you want to de-aggregate individual Kinesis Record (aggregated using KPL) into separate Source Record(s)
* Type: boolean * Default: false * Importance: low ### Connection details `kinesis.record.limit` : The number of records to read in each poll of the Kinesis shard.
* Type: int * Default: 500 * Valid Values: [1,…,10000] * Importance: low `kinesis.throughput.exceeded.backoff.ms` : The number of milliseconds to backoff when a throughput exceeded exception is thrown.
* Type: long * Default: 10000 (10 seconds) * Valid Values: [500,…] * Importance: low `kinesis.empty.records.backoff.ms` : The number of milliseconds to backoff when the stream is empty.
* Type: long * Default: 5000 (5 seconds) * Valid Values: [500,…] * Importance: low ### Number of tasks for this connector `tasks.max` : Maximum number of tasks for the connector.
* Type: int * Valid Values: [1,…] * Importance: high ### Auto-restart policy `auto.restart.on.user.error` : Enable connector to automatically restart on user-actionable errors.
* Type: boolean * Default: true * Importance: medium ### Additional Configs `header.converter` : The converter class for the headers. This is used to serialize and deserialize the headers of the messages.
* Type: string * Importance: low `producer.override.compression.type` : The compression type for all data generated by the producer. Valid values are none, gzip, snappy, lz4, and zstd.
* Type: string * Importance: low `value.converter.allow.optional.map.keys` : Allow optional string map key when converting from Connect Schema to Avro Schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.auto.register.schemas` : Specify if the Serializer should attempt to register the Schema.
* Type: boolean * Importance: low `value.converter.connect.meta.data` : Allow the Connect converter to add its metadata to the output schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.avro.schema.support` : Enable enhanced schema support to preserve package information and Enums. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.protobuf.schema.support` : Enable enhanced schema support to preserve package information. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.flatten.unions` : Whether to flatten unions (oneofs). Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.index.for.unions` : Whether to generate an index suffix for unions. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.struct.for.nulls` : Whether to generate a struct variable for null values. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.int.for.enums` : Whether to represent enums as integers. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.latest.compatibility.strict` : Verify latest subject version is backward compatible when use.latest.version is true.
* Type: boolean * Importance: low `value.converter.object.additional.properties` : Whether to allow additional properties for object schemas. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.optional.for.nullables` : Whether nullable fields should be specified with an optional label. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.optional.for.proto2` : Whether proto2 optionals are supported. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.use.latest.version` : Use latest version of schema in subject for serialization when auto.register.schemas is false.
* Type: boolean * Importance: low `value.converter.use.optional.for.nonrequired` : Whether to set non-required properties to be optional. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.wrapper.for.nullables` : Whether nullable fields should use primitive wrapper messages. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.wrapper.for.raw.primitives` : Whether a wrapper message should be interpreted as a raw primitive at root level. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `key.converter.key.schema.id.serializer` : The class name of the schema ID serializer for keys. This is used to serialize schema IDs in the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.PrefixSchemaIdSerializer * Importance: low `key.converter.key.subject.name.strategy` : How to construct the subject name for key schema registration.
* Type: string * Default: TopicNameStrategy * Importance: low `value.converter.decimal.format` : Specify the JSON/JSON_SR serialization format for Connect DECIMAL logical type values with two allowed literals:
BASE64 to serialize DECIMAL logical types as base64 encoded binary data and
NUMERIC to serialize Connect DECIMAL logical type values in JSON/JSON_SR as a number representing the decimal value.
* Type: string * Default: BASE64 * Importance: low `value.converter.flatten.singleton.unions` : Whether to flatten singleton unions. Applicable for Avro and JSON_SR Converters.
* Type: boolean * Default: false * Importance: low `value.converter.reference.subject.name.strategy` : Set the subject reference name strategy for value. Valid entries are DefaultReferenceSubjectNameStrategy or QualifiedReferenceSubjectNameStrategy. Note that the subject reference name strategy can be selected only for PROTOBUF format with the default strategy being DefaultReferenceSubjectNameStrategy.
* Type: string * Default: DefaultReferenceSubjectNameStrategy * Importance: low `value.converter.value.schema.id.serializer` : The class name of the schema ID serializer for values. This is used to serialize schema IDs in the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.PrefixSchemaIdSerializer * Importance: low `value.converter.value.subject.name.strategy` : Determines how to construct the subject name under which the value schema is registered with Schema Registry.
* Type: string * Default: TopicNameStrategy * Importance: low ## Suggested Reading The following blog post includes steps to set up an example pipeline to get a mock payments stream from Amazon Kinesis into Confluent Cloud using the Confluent Cloud Amazon Kinesis Source connector. Blog post: [How Merging Companies Will Give Rise to Unified Data Streams](https://www.confluent.io/blog/merging-data-streams/) ## Frequently asked questions Find answers to frequently asked questions about the Amazon Kinesis Source connector for Confluent Cloud. ### What are common configuration errors for this connector? Common configuration errors include: * **Incorrect provider integration**: Ensure you select a provider integration with sufficient permissions. * **Missing stream name**: The `kinesis.stream` property must specify the exact name of your Kinesis stream. * **Incorrect offset position**: If using `AT_TIMESTAMP`, you must also specify `kinesis.shard.timestamp` with a valid timestamp. * **Insufficient tasks**: For high-throughput streams, ensure `tasks.max` is set appropriately based on the number of shards. Setting `tasks.max` higher than the shard count has no effect. ### Why am I getting a `KMSAccessDeniedException` error? The `KMSAccessDeniedException` error occurs when the connector lacks permission to decrypt data in your Kinesis stream. To resolve this: 1. Verify that your AWS IAM role includes permissions for AWS KMS operations. 2. Add the following permissions to your IAM policy: ```json { "confluent_resource": "Kinesis Source Connector", "policy_document": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetRecords", "kinesis:GetShardIterator", "kms:Decrypt", "kms:DescribeKey" ], "Resource": [ "arn:aws:kinesis:*:*:stream/", ] } ] } } ``` 3. Ensure the KMS key policy allows the IAM role to use the key for decryption. 4. Update your provider integration with the corrected IAM role. ### Why can’t the connector find my Kinesis stream? If you receive an error indicating the connector cannot find your Kinesis stream, verify the following: * The Kinesis stream name is spelled correctly in the connector configuration. * Verify that `kinesis.region` is set to the region where your Kinesis stream is located. If not specified, it defaults to `us-east-1`. * The IAM role has `kinesis:DescribeStream` permission for the specified stream. Example IAM policy: ```json { "confluent_resource": "Kinesis Source Connector", "policy_document": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:GetRecords", "kinesis:GetShardIterator" ], "Resource": [ "arn:aws:kinesis:*:*:stream/" ] } ] } } ``` * Network connectivity is established between Confluent Cloud and your AWS account. * If using a private network, ensure VPC peering or PrivateLink is properly configured. ### Why is my `Kinesis Iterator Age Max` spiking? Spikes in `Kinesis Iterator Age Max` indicate that the connector is falling behind in consuming records from your Kinesis stream. Common causes and solutions include: * **Insufficient connector tasks**: Increase `tasks.max` to match the number of shards in your Kinesis stream as more tasks can improve parallelism. * **AWS throttling**: Check your Kinesis stream’s provisioned throughput. If you’re exceeding the read capacity, either increase the stream’s provisioning or reduce the read rate. * **Downstream Kafka topic latency**: Verify that your Kafka topic has sufficient partitions and that there are no issues writing to the topic. * **Network latency**: High latency between Confluent Cloud and AWS can slow down data transfer. Consider using a Confluent Cloud cluster in the same region as your Kinesis stream. ### How can I improve connector throughput? To optimize connector throughput: 1. **Increase tasks**: Set `tasks.max` based on your stream’s shard count. More tasks can improve parallelism and throughput. Setting `tasks.max` higher than the shard count has no effect. 2. **Adjust record limit**: Use `kinesis.record.limit` to control the maximum number of records fetched per `GetRecords` call. Higher values can improve throughput but may increase memory usage. 3. **Configure backoff**: Set `kinesis.throughput.exceeded.backoff.ms` to an appropriate value to handle throttling gracefully without overwhelming the stream. 4. **Monitor metrics**: Use Confluent Cloud metrics to track connector lag, throughput, and error rates. ### How does the connector handle base64-encoded data? The connector does not automatically decode base64-encoded data from Kinesis. Data is written to Kafka in the same format it appears in the Kinesis stream. To decode base64 data, use SMT or process the data downstream. ### How do I handle schema validation errors? Schema validation errors typically occur when the data format in your Kinesis stream doesn’t match the expected schema in Confluent Cloud Schema Registry. To troubleshoot: 1. Verify that the data in your Kinesis stream is properly formatted. 2. If using Confluent Cloud Schema Registry, ensure the schema is registered and the connector is configured to use it. 3. Check the connector logs for error messages. These logs typically indicate issues with missing or incompatible schemas, or malformed data that cannot be serialized in the configured `output.data.format`. 4. Consider using the `ByteArrayConverter` or `StringConverter` if you don’t need schema enforcement and want to ingest raw data. ### Can I specify custom offsets when creating the connector? Yes, the connector supports custom offset management. You can specify the starting position using the `kinesis.position` property when creating the connector. Available options are: * `TRIM_HORIZON` (default): Start with the oldest available record in the stream. * `LATEST`: Start with the most recent record. * `AT_TIMESTAMP`: Start at a specific point in time. Requires `kinesis.shard.timestamp` to be set. For more information about managing offsets after the connector is running, see [Manage custom offsets](#cc-kinesis-source-custom-offsets). ### Why is my connector stuck in a `Provisioning` state? If the connector remains in the `Provisioning` state for an extended period: 1. Check that your Confluent Cloud environment and Kafka cluster are healthy. 2. Verify that the Kinesis stream exists and is accessible from Confluent Cloud. 3. Review the connector configuration for errors, particularly authentication and network settings. 4. If the issue persists, contact [Confluent Support](https://support.confluent.io) with your connector ID and environment details. ## Next Steps For an example that shows fully managed Confluent Cloud connectors in action with Confluent Cloud for Apache Flink, see the [Cloud ETL Demo](/platform/current/tutorials/examples/cloud-etl/docs/index.html). This example also shows how to use Confluent CLI to manage your resources in Confluent Cloud. [![image](images/topology.png)](https://docs.confluent.io/platform/current/tutorials/examples/cloud-etl/docs/index.html)