SNMP Trap Source Connector for Confluent Platform

The SNMP Trap Source connector is used to receive data (SNMP traps) from devices through SNMP and convert the trap messages into Kafka records.

Simple Network Management Protocol (SNMP) is an application-layer protocol used to manage and monitor network devices and their functions. SNMP provides a common language for network devices to relay management information within single and multi-vendor environments in a local area network (LAN) or wide area network (WAN). The most recent iteration of SNMP, version 3, includes security enhancements that authenticate and encrypt SNMP messages as well as protect packets during transit.

The connector should be configured to listen to PDUs (an SNMP protocol data unit) from an SNMP device. The connector interprets only those PDUs that are of type trap.

Features

At least once delivery

This connector guarantees that records are delivered at least once to the Kafka topic. If the connector restarts, there may be some duplicate records in the Kafka topic.

Supports one task

The SNMP Trap Source connector supports running only one task.

SNMP protocols

The SNMP connector supports both SNMP V2 and V3 protocols. You can configure either of the protocols using the snmp.v3.enabled parameter. If you configure the connector to listen for SNMP V3, you must provide the following:

  • Security Name: Specifies the SNMPv3 security name(s) or user name(s).
  • Authentication Protocol: Specifies the SNMPv3 authentication protocol or type and its value. You can use any of the following algorithms:
    • MD5
    • SHA
    • HMAC128SHA224
    • HMAC192SHA256
    • HMAC256SHA384
    • HMAC384SHA512
  • Authentication Password: Specifies the SNMPv3 authentication passphrase or password.
  • Privacy/Encryption Protocol: Specifies the SNMPv3 privacy/encryption protocol and its value. You can use any of the following algorithms:
    • DES
    • 3DES
    • AES
    • AES128
    • AES192
    • AES256
  • Privacy Password: Specifies the SNMPv3 encryption password.

Limitations

  • The connector supports only one task because the connector will open a listener port based on the configurations of snmp.listen.protocol, snmp.listen.address, and snmp.listen.port. For multiple tasks, multiple ports need to be opened. Currently this is not supported.
  • The authentication protocol with sha2 is not currently supported.

Install the SNMP Trap Source connector

You can install this connector by using the Confluent Hub client installation instructions or by manually downloading the ZIP file.

Prerequisites

  • You must install the connector on every machine where Connect will run.
  • Kafka Broker: Confluent Platform 3.3.0 or later, or Kafka 0.11.0 or later.
  • Connect: Confluent Platform 4.0.0 or later, or Kafka 1.0.0 or later.
  • Java 1.8.
  • An installation of the Confluent Hub Client. This is installed by default with Confluent Enterprise.
  • An installation of the latest (latest) connector version.

Install the connector using the Confluent CLI

To install the latest connector version, navigate to your Confluent Platform installation directory and run the following command:

confluent connect plugin install confluentinc/kafka-connect-snmp-trap:latest

You can install a specific version by replacing latest with a version number as shown in the following example:

confluent connect plugin install confluentinc/kafka-connect-snmp-trap:1.3.0

Install the connector manually

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License

You can use this connector for a 30-day trial period without a license key.

After 30 days, you must purchase a connector subscription which includes Confluent enterprise license keys to subscribers, along with enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, you can contact Confluent Support at support@confluent.io for more information.

For license properties, see Confluent Platform license, and for information about the license topic, refer to License topic configuration.

Configuration properties

For a complete list of configuration properties for this connector, see SNMP Trap Source connector Configuration Properties.

For an example of how to get Kafka Connect connected to Confluent Cloud, see Connect Self-Managed Kafka Connect to Confluent Cloud.

Quick start

The following uses the SnmpTrapSourceConnector to receive data (SNMP traps) from devices through SNMP and send them to the Kafka topic.

  1. Install the connector through the Confluent Hub Client.

    # run from your Confluent Platform installation directory
confluent connect plugin install confluentinc/kafka-connect-snmp-trap:latest

    Note that by default, the connector will install the plugin into the share/confluent-hub-components directory and add the directory to the plugin path. For the plugin path change to take effect, you must restart the Connect worker.

Property-based example

  1. Create a snmp-trap-source-quickstart.properties file with the following contents or use the snmp-trap-source-quickstart.properties. This configuration is used typically along with standalone workers.:

    name=SnmpTrapSourceConnector
tasks.max=1
connector.class=io.confluent.connect.snmp.SnmpTrapSourceConnector
kafka.topic=snmp-kafka-topic
snmp.batch.size=50
snmp.listen.address=<ip-address to listen trap from>
snmp.listen.port=<port to listen trap from>
snmp.v3.enabled=true
v3.security.context.users=<list-of-usernames>
v3.$username.auth.password=<auth-password>
v3.$username.privacy.password=<privacy-password>
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
confluent.license=<your-confluent-license>

    The following defines the Confluent license stored in Kafka, so you need the Kafka bootstrap addresses. The replication.factor may not be larger than the number of Kafka brokers in the destination cluster, so here set this to a value of 1 for demonstration purposes. Always set this to a value of at least 3 in production configurations.

  2. Load the SNMP Trap Source connector.

    Caution

    You must include a double dash (--) between the connector name and your flag. For more information, see this post.

    confluent local services connect connector load snmp-trap-source --config snmp-trap-source-quickstart.properties

    It’s important that you don’t use the Confluent CLI in production environments.

  3. Confirm that the connector is in a RUNNING state.

    confluent local services connect connector status snmp-trap-source

  4. The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to a Kafka topic.

  5. Confirm that the messages are being sent to Kafka.

    kafka-avro-console-consumer --bootstrap-server localhost:9092 --property schema.registry.url=http://localhost:8081 --topic snmp-kafka-topic --from-beginning

A sample SNMP PDU of type trap might look like this for sysDescr Oid. Refer - https://www.alvestrand.no/objectid/1.3.6.1.2.1.1.1.html:

TRAP[
      {
        contextEngineID=80:00:00:59:03:78:d2:94:b8:9f:95,
        contextName=
      },
      requestID=2058388122,
      errorStatus=0,
      errorIndex=0,
      VBS[
           1.3.6.1.2.1.1.1.0 = 24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks
         ]
    ]

Data in Kafka topic:

{
  "peerAddress":"127.0.0.1/55159",
  "securityName":"admin",
  "variables":[
    {
      "oid":"1.3.6.1.2.1.1.1.0",
      "type":"octetString",
      "counter32":null,
      "counter64":null,
      "gauge32":null,
      "integer":null,
      "ipaddress":null,
      "null":null,
      "objectIdentifier":null,
      "octetString":null,
      "opaque":null,
      "timeticks":null,
      "metadata":{
        "string":"24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks"
      }
   }]
}

REST-based example

Use this setting with distributed workers. Write the following JSON to snmp-trap-source-config.json, configure all of the required values, and use the following command to post the configuration to one of the distributed connect workers. For more information, see the Kafka Connect Kafka Connect REST Interface.

 {
 "name": "SnmpTrapSourceConnector",
 "config": {
     "name":"SnmpTrapSourceConnector",
     "connector.class":"io.confluent.connect.snmp.SnmpTrapSourceConnector",
     "tasks.max":"1",
     "kafka.topic":"snmp-kafka-topic",
     "snmp.v3.enabled":"true",
     "snmp.batch.size":"50",
     "snmp.listen.address":"<ip-address to listen trap from>",
     "snmp.listen.port":"<port to listen trap from>",
     "auth.password":"<Auth-Password>",
     "privacy.password":"<privacy-password>",
     "security.name":"<security-name>",
     "confluent.topic.bootstrap.servers":"localhost:9092",
     "confluent.topic.replication.factor":"1"
 }
}

Use curl to post the configuration to one of the Kafka Connect Workers. Change http://localhost:8083/ the endpoint of one of your Kafka Connect worker(s).

curl -sS -X POST -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors

Use the following command to update the configuration of existing connector.

curl -s -X PUT -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors/snmpTrapSourceConnector/config

Check that the connector started successfully. Review the Connect worker’s log by entering the following:

confluent local services connect log

The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to Kafka topic.

Record Schema

The source connector creates records in the following format:

Key schema

The Key is a struct with the following fields:

Field Name Schema Type Optional? Description
peerAddress String Mandatory Remote address of the host sending the trap.

Value schema

The Value is a struct with the following fields:

Field Name Schema Type Optional? Description
peerAddress String Mandatory Remote address of the host sending the trap.
peerPort String Mandatory Remote port of the host sending the trap.
securityName String Mandatory Community name the event was sent to.
sysUpTime Int64 Optional The time (in hundredths of a second) since the network management portion of the system was last re-initialized.
enterprise String Optional The authoritative identification of the enterprise associated with the trap currently being sent.
genericTrap Int32 Optional Generic trap type of the PDU.
specificTrap Int32 Optional Specific trap type of the PDU.
pduBerEncoded Bytes Optional BER encoded trap PDU.
pduRawBytes Bytes Optional Trap PDU raw bytes.
variables Array of struct Mandatory Variables for this trap.

Variable schema

The Variable is a struct with the following fields:

Field Name Schema Type Optional? Description
oid String Mandatory OID
type String Mandatory Syntax type for variable binding.
counter32 Int32 Optional Counter32 value. Ranges from 0 to 4294967295.
counter64 Int64 Optional Counter64 value. Ranges from 0 to 18446744073709551615.
gauge32 Int32 Optional Gauge32 value. Ranges from 0 to 4294967295.
integer Int32 Optional Integer value.
ipaddress String Optional IpAddress value.
null String Optional Null value.
objectIdentifier String Optional OID value.
octetString String Optional Octet string value.
opaque String Optional Opaque value.
timeticks Int32 Optional Timeticks value. Ranges from 0 to 4294967295.
metadata String Optional Metadata field that contains value object of variables.