Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Splunk Sink Connector for Confluent Platform

The Kafka Connect Splunk Sink connector is used to move messages from Apache Kafka® to Splunk.

The connector has the following features:

  • Data ingestion from Kafka topics into Splunk via Splunk HTTP Event Collector (HEC)

    The Splunk HTTP Event Collector (HEC) receives data from Kafka topics via HTTP or HTTPS connection using an Event Collector token configured in Splunk.

  • In-flight data transformation and enrichment.

    This feature is used to enrich raw data with extra metadata fields. The configured enrichment metadata is indexed along with raw event data by the Splunk software. See Indexed Field Extractions for more information.

    Note

    Data enrichment for /event HEC endpoint is only available in Splunk Enterprise 6.5 and above.

  • Acknowledgement mode

    This feature implements guaranteed delivery by polling Splunk for acknowledgement before committing the Kafka offset.

Prerequisites

The following are required to run the Splunk Sink Connector:

  • Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
  • Connect: Confluent Platform 4.0 or above, or Kafka 1.0 or above
  • Java 1.8
  • Splunk 6.5 or above, configured with valid HTTP Event Collector (HEC) tokens
  • Splunk Indexers and Heavy Forwarders that send information to this connector should have the same HEC token settings as this connector.
  • Task configuration parameters vary depending on acknowledgement setting. See the Configuration Properties for details.

Note

HEC Acknowledgement prevents potential data loss but may slow down event ingestion.

Install the Splunk Sink Connector

You can install this connector by using the Confluent Hub client (recommended) or you can manually download the ZIP file.

Install the connector using Confluent Hub

Prerequisite
Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.

Navigate to your Confluent Platform installation directory and run the following command to install the latest (latest) connector version. The connector must be installed on every machine where Connect will run.

confluent-hub install splunk/kafka-connect-splunk:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install splunk/kafka-connect-splunk:1.1.1

Install Connector Manually

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License

The Splunk Sink connector is an open source connector and does not require a Confluent Enterprise License.

Quick Start

Important

The default port used by a Splunk HEC is 8088. However, the KSQL component of Confluent Platform also uses that port. For this quick start, since both Splunk and Confluent Platform will be running, we configure the HEC to use port 8889. If that port is in use by another process, change 8889 to a different, open port.

  1. Start a Splunk Enterprise instance by running the Splunk Docker container.

    docker run -d -p 8000:8000 -p 8889:8889 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=password" --name splunk splunk/splunk:7.3.0

  2. Open http://localhost:8000 to access Splunk Web. Login with username admin and password password.

  3. Configure a Splunk HEC using Splunk Web.

    • Click Settings > Data Inputs.
    • Click HTTP Event Collector.
    • Click Global Settings.
    • In the All Tokens toggle button, select Enabled.
    • Ensure SSL disabled is checked.
    • Change the HTTP Port Number to 8889.
    • Click Save.
    • Click New Token.
    • In the Name field, enter a name for the token: kafka
    • Click Next.
    • Click Review.
    • Click Submit.

    Important

    Note the token value on the “Token has been created successfully” page. This token value is needed for the connector configuration later.

  4. Install the connector through the Confluent Hub Client.

    # run from your Confluent Platform installation directory
confluent-hub install splunk/kafka-connect-splunk:latest

  5. Start Confluent Platform.

    Tip

    The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local start. For more information, see confluent local.

    confluent local start

  6. Produce test data to the splunk-qs topic in Kafka.

    echo event 1 | confluent local produce splunk-qs
echo event 2 | confluent local produce splunk-qs

  7. Create a splunk-sink.properties file with the properties below. Substitute <HEC_TOKEN> with the Splunk HEC token created earlier.

    name=SplunkSink
topics=splunk-qs
tasks.max=1
connector.class=com.splunk.kafka.connect.SplunkSinkConnector
splunk.indexes=main
splunk.hec.uri=http://localhost:8889
splunk.hec.token=<HEC_TOKEN>
splunk.sourcetypes=my_sourcetype
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
value.converter=org.apache.kafka.connect.storage.StringConverter

  8. Start the connector.

    Caution

    You must include a double dash (--) between the topic name and your flag. For more information, see this post.

    confluent local load splunk -- -d splunk-sink.properties

  9. In the Splunk UI, verify that data is flowing into your Splunk platform instance by searching using the search parameter source="http:kafka".

  10. Shut down Confluent Platform.

    confluent local destroy

  11. Shut down the Docker container.

    docker stop splunk
docker rm splunk

Additional Documentation