.. _aws-cloudwatch-logs-source-connector: |kconnect-long| |aws| CloudWatch Logs Source Connector ====================================================== The |aws| CloudWatch Logs source connector is used to import data from |aws| CloudWatch Logs, and write them into a Kafka topic. Moreover, the connector sources from a single log group and writes to one topic per log stream. There is a topic format configuration available to customize the topic names of each log stream. If specific customizations for topics such as multiple log streams writing to the same topic are desired, SMTs can be used for such actions. This connector can start at one task supporting all importation of data and can scale up to one task per log stream which will raise performance to the highest that Amazon supports (100,000 logs per second or 10MB per second). Prerequisites ------------- The following are required to run the |kconnect-long| |aws| CloudWatch Logs Connector: * |ak| Broker: |cp| 3.3.0 or above * |kconnect|: |cp| 4.1.0 or above * Java 1.8 * |aws| account * At least one |aws| CloudWatch log group and log stream in |aws| CloudWatch Logs Features -------- The |aws| CloudWatch Logs connector offers a variety of features: * **At Least Once Delivery**: Records imported from |aws| CloudWatch Logs are delivered with at least once semantics. Duplicates will generally be limited, however, as there will only be repeats in the chance of unexpected termination of the connector. * **Topic Format Customizability**: Because this connector is designed to write to a topic per log stream, custom topic formats can be created or all records can be written to exactly one topic. * **Log Stream Selection**: The log streams from which logs are imported from can be specified, or as a default, all will be used. Install the |aws| CloudWatch Logs Connector ------------------------------------------- .. include:: ../includes/connector-install.rst .. include:: ../includes/connector-install-hub.rst .. codewithvars:: bash confluent-hub install confluentinc/kafka-connect-aws-cloudwatch:latest .. include:: ../includes/connector-install-version.rst .. codewithvars:: bash confluent-hub install confluentinc/kafka-connect-aws-cloudwatch:1.0.0-preview -------------------------- Install Connector Manually -------------------------- `Download and extract the ZIP file `__ for your connector and then follow the manual connector installation :ref:`instructions `. License ------- .. include:: ../includes/enterprise-license.rst See :ref:`aws-cloudwatch-logs-source-connector-license-config` for license properties and :ref:`aws-cloudwatch-logs-source-license-topic-configuration` for information about the license topic. .. _aws_cloudwatch_logs_quickstart: |aws| CloudWatch Logs Source Connector Quick Start -------------------------------------------------- ----------------- Preliminary Setup ----------------- To add a new connector plugin you must restart |kconnect|. Use the :ref:`Confluent CLI ` command to restart |kconnect|: .. codewithvars:: bash |confluent_stop| connect && |confluent_start| connect Your output should resemble: :: Using CONFLUENT_CURRENT: /Users/username/Sandbox/confluent-snapshots/var/confluent.NuZHxXfq Starting zookeeper zookeeper is [UP] Starting kafka kafka is [UP] Starting schema-registry schema-registry is [UP] Starting kafka-rest kafka-rest is [UP] Starting connect connect is [UP] Check if the |aws| CloudWatch Logs plugin has been installed correctly and picked up by the plugin loader: :: curl -sS localhost:8083/connector-plugins | jq '.[].class' | grep cloudwatch Your output should resemble: :: "io.confluent.connect.aws.cloudwatch.AwsCloudWatchSourceConnector" .. _cloudwatch-awscredentials: ----------------- |aws| Credentials ----------------- By default, the |aws| CloudWatch Logs connector looks for |aws| credentials in the following locations and in the following order: #. The ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables accessible to the Connect worker processes where the connector will be deployed. These variables are recognized by the |aws| CLI and all |aws| SDKs (except for the |aws| SDK for .NET). You use export to set these variables. .. sourcecode:: bash export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= The ``AWS_ACCESS_KEY`` and ``AWS_SECRET_KEY`` can be used instead, but are not recognized by the |aws| CLI. #. The ``aws.accessKeyId`` and ``aws.secretKey`` Java system properties on the Connect worker processes where the connector will be deployed. However, these variables are only recognized by the |aws| SDK for Java and are not recommended. #. The ``~/.aws/credentials`` file located in the home directory of the operating system user that runs the Connect worker processes. These credentials are recognized by most |aws| SDKs and the |aws| CLI. Use the following |aws| CLI command to create the credentials file: .. sourcecode:: bash aws configure You can also manually create the credentials file using a text editor. The file should contain lines in the following format: .. sourcecode:: bash [default] aws_access_key_id = aws_secret_access_key = .. note:: When creating the credentials file, make sure that the user creating the credentials file is the same user that runs the Connect worker processes and that the credentials file is in this user's home directory. Otherwise, the kinesis connector will not be able to find the credentials. See `AWS Credentials File Format `__ for additional details. Choose one of the above to define the |aws| credentials that the |aws| CloudWatch Logs connectors use, verify the credentials implementation is set correctly, and then restart all of the Connect worker processes. .. note:: Confluent recommends using either **Environment variables** or a **Credentials file** because these are the most straightforward, and they can be checked using the |aws| CLI tool before running the connector. Credentials Providers ^^^^^^^^^^^^^^^^^^^^^ A *credentials provider* is a Java class that implements the `com.amazon.auth .AWSCredentialsProvider `__ interface in the |aws| Java library and returns |aws| credentials from the environment. By default the |aws| CloudWatch Logs connector configuration property ``aws.credentials.provider.class`` uses the `com.amazon.auth.DefaultAWSCredentialsProviderChain `__ class. This class and interface implementation chains together five other credential provider classes. The `com.amazonaws.auth.DefaultAWSCredentialsProviderChain `__ implementation looks for credentials in the following order: #. **Environment variables** using the `com.amazonaws.auth.EnvironmentVariableCredentialsProvider `__ class implementation. This implementation uses environment variables ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY``. Environment variables ``AWS_ACCESS_KEY`` and ``AWS_SECRET_KEY`` are also supported by this implementation; however, these two variables are only recognized by the |aws| SDK for Java and are not recommended. #. **Java system properties** using the `com.amazonaws.auth.SystemPropertiesCredentialsProvider `__ class implementation. This implementation uses Java system properties ``aws.accessKeyId`` and ``aws.secretKey``. #. **Credentials file** using the `com.amazonaws.auth.profile.ProfileCredentialsProvider `__ class implementation. This implementation uses a credentials file located in the path ``~/.aws/credentials``. This credentials provider can be used by most |aws| SDKs and the |aws| CLI. Use the following |aws| CLI command to create the credentials file: .. sourcecode:: bash aws configure You can also manually create the credentials file using a text editor. The file should contain lines in the following format: .. sourcecode:: bash [default] aws_access_key_id = aws_secret_access_key = .. note:: When creating the credentials file, make sure that the user creating the credentials file is the same user that runs the Connect worker processes and that the credentials file is in this user's home directory. Otherwise, the kinesis connector will not be able to find the credentials. See `AWS Credentials File Format `__ for additional details. .. _aws-cloud-other-credentials-implementations: Using Other Implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can use a different credentials provider. To do this, set the ``aws.credentials.provider.class`` property to the name of any class that implements the `com.amazon.auth.AWSCredentialsProvider `__ interface. .. important:: If you are using a different credentials provider, do not include the ``aws.access.key.id`` and ``aws.secret.key.id`` in the connector configuration file. If these parameters are included, they will override the custom credentials provider class. Complete the following steps to use a different credentials provider: #. Find or create a Java credentials provider class that implements the `com.amazon.auth.AWSCredentialsProvider `__ interface. #. Put the class file in a JAR file. #. Place the JAR file in the ``share/java/kafka-connect-aws-cloudwatch`` directory on **all Connect workers**. #. Restart the Connect workers. #. Change the |aws| CloudWatch Logs connector property file to use your custom credentials. Add the provider class entry ``aws.credentials.provider.class=`` in the |aws| CloudWatch Logs connector properties file. .. important:: You must use the fully qualified class name in the ```` entry. --------------------------- |aws| CloudWatch Logs Setup --------------------------- You can use the |aws| Management Console to set up your |aws| CloudWatch log group and log stream as shown `here`_ or you can complete the following steps: #. Make sure you have an `AWS account`_. #. Set up :ref:`cloudwatch-awscredentials`. #. `Create a log group`_ in |aws| CloudWatch Logs. :: aws logs create-log-group --log-group my-log-group #. `Create a log stream`_ in |aws| CloudWatch Logs. :: aws logs create-log-stream --log-group my-log-group --log-stream my-log-stream #. `Insert Records`_ into your log stream. If this is the first time inserting logs into a new log stream, then no sequence token is needed. However, after the first put, a sequence token is returned. You will need this token as a parameter for the next put. :: aws logs put-log-events --log-group my-log-group --log-stream my-log-stream --log-events timestamp=