SNMP Trap Source Connector for Confluent Platform

The Kafka Connect SNMP Trap Source Connector is used to receive data (SNMP traps) from devices through SNMP and convert the trap messages into Kafka records.

Simple Network Management Protocol (SNMP) is an application-layer protocol used to manage and monitor network devices and their functions. SNMP provides a common language for network devices to relay management information within single and multi-vendor environments in a local area network (LAN) or wide area network (WAN). The most recent iteration of SNMP, version 3, includes security enhancements that authenticate and encrypt SNMP messages as well as protect packets during transit.

The connector should be configured to listen to PDUs (a SNMP protocol data unit) from an SNMP device.

The connector interprets only those PDUs that are of type trap.

This connector supports both SNMP V2 and V3 protocols and it can be configured using snmp.v3.enabled parameter. If the connector is configured to listen an SNMP V3, then the following SNMPv3 options need to be provided.

  • Authentication Protocol: Specifies the SNMPv3 authentication protocol or type and its value can be any of [md5, sha, hmac128sha224, hmac192sha256, hmac256sha384, hmac384sha512].
  • Authentication Password: Specifies the SNMPv3 authentication passphrase or password.
  • Privacy/Encryption Protocol: Specifies the SNMPv3 privacy/encryption protocol and its value can be any of [des, 3des, aes, aes128, aes192, aes256].
  • Privacy Password: Specifies the SNMPv3 encryption password.
  • Security Name: Specifies the SNMPv3 security name or user name.

Prerequisites

The following are required to run the Kafka Connect SNMP Trap Source Connector:

  • Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
  • Connect: Confluent Platform 4.0.0 or above, or Kafka 1.0.0 or above
  • Java: 1.8

Limitations

  • The connector supports only one task because the connector will open a listener port based on the configurations of snmp.listen.protocol, snmp.listen.address, and snmp.listen.port. For multiple tasks, multiple ports need to be opened. Currently this is not supported.
  • The authentication protocol with sha2 is not currently supported.

Install the SNMP Trap Source Connector

You can install this connector by using the Confluent Hub client (recommended) or you can manually download the ZIP file.

Install the connector using Confluent Hub

Prerequisite
Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.

Navigate to your Confluent Platform installation directory and run the following command to install the latest (latest) connector version. The connector must be installed on every machine where Connect will run.

confluent-hub install confluentinc/kafka-connect-snmp-trap:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install confluentinc/kafka-connect-snmp-trap:1.0.0-preview

Install the connector manually

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License

You can use this connector for a 30-day trial period without a license key.

After 30 days, this connector is available under a Confluent enterprise license. Confluent issues enterprise license keys to subscribers, along with providing enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

See Confluent Platform license for license properties and License topic configuration for information about the license topic.

Configuration Properties

For a complete list of configuration properties for this connector, see Solace Sink Connector Configuration Properties.

Note

For an example of how to get Kafka Connect connected to Confluent Cloud, see Distributed Cluster in Connect Kafka Connect to Confluent Cloud.

Quick Start

The following uses the SnmpTrapSourceConnector to receive data (SNMP traps) from devices through SNMP and send them to the Kafka topic.

  1. Install the connector through the Confluent Hub Client.

    # run from your Confluent Platform installation directory
    confluent-hub install confluentinc/kafka-connect-snmp-trap:latest
    

    Tip

    By default, the plugin is installed into share/confluent-hub-components and the directory is added to the plugin path. If this is the first connector you have installed, you may need to restart the Connect server for the plugin path change to take effect.

Property-based example

  1. Create a snmp-trap-source-quickstart.properties file with the following contents or use the snmp-trap-source-quickstart.properties. This configuration is used typically along with standalone workers.:

    name=SnmpTrapSourceConnector
    tasks.max=1
    connector.class=io.confluent.connect.snmp.SnmpTrapSourceConnector
    snmp.v3.enabled=true
    kafka.topic=snmp-kafka-topic
    snmp.batch.size=50
    snmp.listen.address=<ip-address to listen trap from>
    snmp.listen.port=<port to listen trap from>
    auth.password=<Auth-Password>
    privacy.password=<privacy-password>
    security.name=<security-name>
    confluent.topic.bootstrap.servers=localhost:9092
    confluent.topic.replication.factor=1
    confluent.license=
    

    Tip

    The following defines the Confluent license stored in Kafka, so we need the Kafka bootstrap addresses. The replication.factor may not be larger than the number of Kafka brokers in the destination cluster, so here we set this to ‘1’ for demonstration purposes. Always use at least ‘3’ in production configurations.

  2. Load the SNMP Trap Source Connector.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    Tip

    The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local services start. For more information, see confluent local.

    confluent local services connect connector load snmp-trap-source --config snmp-trap-source-quickstart.properties
    

    Important

    Don’t use the Confluent CLI in production environments.

  3. Confirm that the connector is in a RUNNING state.

    confluent local services connect connector status snmp-trap-source
    
  4. The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to a Kafka topic.

  5. Confirm that the messages are being sent to Kafka.

    kafka-avro-console-consumer --bootstrap-server localhost:9092 --property schema.registry.url=http://localhost:8081 --topic snmp-kafka-topic --from-beginning
    

A sample SNMP PDU of type trap might look like this for sysDescr Oid. Refer - https://www.alvestrand.no/objectid/1.3.6.1.2.1.1.1.html:

TRAP[
      {
        contextEngineID=80:00:00:59:03:78:d2:94:b8:9f:95,
        contextName=
      },
      requestID=2058388122,
      errorStatus=0,
      errorIndex=0,
      VBS[
           1.3.6.1.2.1.1.1.0 = 24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks
         ]
    ]

Data in Kafka topic:

{
  "peerAddress":"127.0.0.1/55159",
  "securityName":"admin",
  "variables":[
    {
      "oid":"1.3.6.1.2.1.1.1.0",
      "type":"octetString",
      "counter32":null,
      "counter64":null,
      "gauge32":null,
      "integer":null,
      "ipaddress":null,
      "null":null,
      "objectIdentifier":null,
      "octetString":null,
      "opaque":null,
      "timeticks":null,
      "metadata":{
        "string":"24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks"
      }
   }]
}

REST-based example

Use this setting with distributed workers. Write the following JSON to snmp-trap-source-config.json, configure all of the required values, and use the following command to post the configuration to one of the distributed connect workers. For more information, see the Kafka Connect REST API.

 {
 "name": "SnmpTrapSourceConnector",
 "config": {
     "name":"SnmpTrapSourceConnector",
     "connector.class":"io.confluent.connect.snmp.SnmpTrapSourceConnector",
     "tasks.max":"1",
     "kafka.topic":"snmp-kafka-topic",
     "snmp.v3.enabled":"true",
     "snmp.batch.size":"50",
     "snmp.listen.address":"<ip-address to listen trap from>",
     "snmp.listen.port":"<port to listen trap from>",
     "auth.password":"<Auth-Password>",
     "privacy.password":"<privacy-password>",
     "security.name":"<security-name>",
     "confluent.topic.bootstrap.servers":"localhost:9092",
     "confluent.topic.replication.factor":"1"
 }
}

Use curl to post the configuration to one of the Kafka Connect Workers. Change http://localhost:8083/ the endpoint of one of your Kafka Connect worker(s).

curl -sS -X POST -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors

Use the following command to update the configuration of existing connector.

curl -s -X PUT -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors/snmpTrapSourceConnector/config

Check that the connector started successfully. Review the Connect worker’s log by entering the following:

confluent local services connect log

The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to Kafka topic.

Record Schema

The source connector creates records in the following format:

Key Schema

The Key is a struct with the following fields:

Field Name Schema Type Optional? Description
peerAddress string mandatory Remote address of the host sending the trap.

Value Schema

The Value is a struct with the following fields:

Field Name Schema Type Optional? Description
peerAddress string mandatory Remote address of the host sending the trap.
securityName string mandatory Community name the event was sent to.
variables array of struct mandatory Variables for this trap.

Variable Schema

The Variable is a struct with the following fields:

Field Name Schema Type Optional? Description
oid string mandatory OID
type string mandatory Syntax type for variable binding.
counter32 int32 Optional Counter32 value. Ranges from 0 to 4294967295.
counter64 int64 Optional Counter64 value. Ranges from 0 to 18446744073709551615.
gauge32 int32 Optional Gauge32 value. Ranges from 0 to 4294967295.
integer int32 Optional Integer value.
ipaddress string Optional IpAddress value.
null string Optional null value.
objectIdentifier string Optional OID value.
octetString string Optional Octet string value.
opaque string Optional opaque value.
timeticks int32 Optional timeticks value. Ranges from 0 to 4294967295.
metadata string Optional metadata field that contains value object of variables.