Kafka Connect Syslog Connector

You can use the syslog connector to consume data from network devices. Supported formats are rfc 3164, rfc 5424, and Common Event Format (CEF).

Important

This connector listens on a network port. Running more than one task or running in distributed mode can cause some undesired effects if another task already has the port open. It is recommended that you run this connector in Standalone Workers.

License

This connector is available under the Confluent Software Evaluation License. You can use this connector for a 30-day trial period. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

For more information, see syslog-source-connector-license-config.

Quick Start

  1. Install the connector:

    confluent-hub install confluentinc/kafka-connect-syslog:latest
    
  2. Start Confluent Platform:

    |confluent_start| connect
    
  3. Create a config file with the following contents:

    name=syslog-tcp
    tasks.max=1
    connector.class=io.confluent.connect.syslog.SyslogSourceConnector
    syslog.port=5454
    syslog.listener=TCP
    confluent.license=
    confluent.topic.bootstrap.servers=localhost:9092
    confluent.topic.replication.factor=1
    
  4. Load the Syslog Connector.

    confluent load syslog-tcp -d path/to/config.properties
    

    Important

    Don't use the Confluent CLI in production environments. Always run the Syslog connector in standalone mode, for example, with bin/connect-standalone.

  5. Test with sample syslog-formatted message sent via netcat:

    echo "<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - Your refrigerator is running" | nc -v -w 0 localhost 5454
    
  6. Confirm that the message is logged to Kafka:

    kafka-avro-console-consumer \
        --bootstrap-server localhost:9092 \
        --property schema.registry.url=http://localhost:8081 \
        --topic syslog --from-beginning | jq '.'
    

Output Schema

The Syslog Connector accepts Syslog messages as strings, and produces structured messages as output to the configured Kafka topic.

Record Key Schema

host

Host of the Syslog message, as specified in the message body. If no host is specified, the remote address of the message will be used.

  • Type: string

Record Value Schema

Some fields are relevant for only some message types, and will be null for others.

name

String representing a human-readable and understandable description of the event. The event name should not contain information that is specifically mentioned in other fields. Used in CEF.

  • Type: string
type

Type of message received (either "RFC5424", "RFC3164", "CEF", or "UNKNOWN").

  • Type: string
message

The freeform message extracted from full message. "MSG" field of the Syslog spec.

  • Type: string
host

Extracted host from the syslog message.

  • Type: string
version

"VERSION" field of the Syslog spec.

  • Type: int
level

Level as determined by "PRIORITY" field of Syslog spec.

  • Type: int
tag

"TAG" field of Syslog spec

  • Type: string
facility

"FACILITY" as determined by the "PRIORITY" of the Syslog spec.

  • Type: int
severity

Severity of CEF messages.

  • Type: string
appName

"APP-NAME" field of the Syslog spec.

  • Type: string
remoteAddress

Remote address of the request received by the connector.

  • Type: string
rawMessage

The full, unmodified, unparsed message as received by the connector.

  • Type: string
processId

"PROCID" field of the Syslog spec.

  • Type: string
messageId

"MSGID" field of the Syslog spec.

  • Type: string
deviceVendor

Vendor identifier that is used to group products.

  • Type: string
deviceProduct

Product identifier that is used for message logging.

  • Type: string
deviceVersion

The version of the logging device product.

  • Type: string
deviceEventClassId

The device event class ID. This is a unique per event-type identifier. The device event class ID identifies the type of event reported. In the intrusion detection system (IDS) world, each signature or rule that detects certain activity has a unique device event class ID assigned. This is a requirement for other types of devices as well, and helps correlation engines process the events. Also known as Signature ID.

  • Type: string
extension

"Extension" mapping for the CEF message format.

  • Type: map<string, string>
structuredData

"STRUCTURED-DATA" field of the Syslog spec.

  • Type: Array of StructuredDatum structs:

    id: String StructuredDataElements: map<string, string>