Splunk Sink Connector for Confluent Platform

The Splunk Sink connector is used to move messages from Apache Kafka® to Splunk.

Features

The Splunk Sink connector includes the following features:

At least once delivery

This connector guarantees that records are delivered at least once from the Kafka topic.

Dead Letter Queue

This connector supports the Dead Letter Queue (DLQ) functionality. For information about accessing and using the DLQ, see Confluent Platform Dead Letter Queue.

Multiple tasks

The Splunk Sink connector supports running one or more tasks. You can specify the number of tasks in the tasks.max configuration parameter. This can lead to performance gains when multiple files need to be parsed.

Data ingestion

The Splunk HTTP Event Collector (HEC) receives data from Kafka topics through HTTP or HTTPS connection using an Event Collector token configured in Splunk. When configuring the connector to consume records from multiple Kafka topics, the records will be routed to Splunk indexes in the order you define them in the splunk.indexes configuration property. For example,

{
  ...

  "topics":"topic1,topic2,topic3",
  "splunk.indexes": "index1,index2,index3",

  ...
}

will result in the following message routing:

topic1 > index1
topic2 > index2
topic3 > index3

Alternatively, if you define only one index, all records from multiple topics will be routed to the same index–anything else will trigger a configuration exception during initialization.

In-flight data transformation and enrichment

This feature is used to enrich raw data with extra metadata fields. The configured enrichment metadata is indexed along with raw event data by the Splunk software. See Indexed Field Extractions for more information.

Note

Data enrichment for /event HEC endpoint is only available in Splunk Enterprise 6.5 and above.

Acknowledgement mode

This feature implements guaranteed delivery by polling Splunk for acknowledgement before committing the Kafka offset.

Install the Splunk Sink Connector

You can install this connector by using the confluent connect plugin install command, or by manually downloading the ZIP file.

Prerequisites

Note

  • Kafka Broker: Confluent Platform 3.3.0 or later, or Kafka 0.11.0 or later.

  • Connect: Confluent Platform 4.0 or later, or Kafka 1.0 or later.

  • Java 1.8.

  • Splunk 6.5 or later, configured with valid HTTP Event Collector (HEC) tokens.

  • Splunk Indexers and Heavy Forwarders that send information to this connector should have the same HEC token settings as this connector.

  • An install of the Confluent Hub Client. This is installed by default with Confluent Enterprise.

  • An install of the latest (latest) connector version.

    To install the latest connector version, navigate to your Confluent Platform installation directory and run the following command:

    confluent connect plugin installsplunk/kafka-connect-splunk:latest
    

    You can install a specific version by replacing latest with a version number as shown in the following example:

    confluent connect plugin installsplunk/kafka-connect-splunk:2.2.0-SNAPSHOT
    

Install the connector manually

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License

The Splunk Sink connector is an open source connector and does not require a Confluent Enterprise License.

Configuration Properties

For a complete list of configuration properties for this connector, see Configuration Reference for Splunk Sink Connector for Confluent Platform.

For an example of how to get Kafka Connect connected to Confluent Cloud, see Connect Self-Managed Kafka Connect to Confluent Cloud.

Quick Start

Important

The default port used by a Splunk HEC is 8088. However, the ksqlDB component of Confluent Platform also uses that port. For this quick start, since both Splunk and Confluent Platform will be running, we configure the HEC to use port 8889. If that port is in use by another process, change 8889 to a different, open port.

  1. Start a Splunk Enterprise instance by running the Splunk Docker container.

    docker run -d -p 8000:8000 -p 8889:8889 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=password" --name splunk splunk/splunk:7.3.0
    
  2. Open http://localhost:8000 to access Splunk Web. Log in with username admin and password password.

  3. Configure a Splunk HEC using Splunk Web.

    • Click Settings > Data Inputs.
    • Click HTTP Event Collector.
    • Click Global Settings.
    • In the All Tokens toggle button, select Enabled.
    • Ensure SSL disabled is checked.
    • Change the HTTP Port Number to 8889.
    • Click Save.
    • Click New Token.
    • In the Name field, enter a name for the token: kafka
    • Click Next.
    • Click Review.
    • Click Submit.

    Important

    Note the token value on the Token has been created successfully page. This token value is needed for the connector configuration later.

  4. Install the connector through the Confluent Hub Client.

    # run from your Confluent Platform installation directory
    confluent connect plugin installsplunk/kafka-connect-splunk:latest
    
  5. Start Confluent Platform.

    Tip

    The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local services start. For more information, see confluent local.

    confluent local services start
    
  6. Produce test data to the splunk-qs topic in Kafka.

    echo event 1 | confluent local services kafka produce splunk-qs
    echo event 2 | confluent local services kafka produce splunk-qs
    
  7. Create a splunk-sink.properties file with the properties below. Substitute <HEC_TOKEN> with the Splunk HEC token created earlier.

    name=SplunkSink
    topics=splunk-qs
    tasks.max=1
    connector.class=com.splunk.kafka.connect.SplunkSinkConnector
    splunk.indexes=main
    splunk.hec.uri=http://localhost:8889
    splunk.hec.token=<HEC_TOKEN>
    splunk.sourcetypes=my_sourcetype
    confluent.topic.bootstrap.servers=localhost:9092
    confluent.topic.replication.factor=1
    value.converter=org.apache.kafka.connect.storage.StringConverter
    
  8. Start the connector.

    Caution

    You must include a double dash (--) between the topic name and your flag. For more information, see this post.

    confluent local services connect connector load splunk --config splunk-sink.properties
    
  9. In the Splunk user interface, verify that data is flowing into your Splunk platform instance by searching using the search parameter source="http:kafka".

  10. Shut down Confluent Platform.

    confluent local destroy
    
  11. Shut down the Docker container.

    docker stop splunk
    docker rm splunk