.. _syslog-source-connector:

Kafka Connect Syslog Source Connector
-------------------------------------

You can use the syslog connector to consume data from network devices. Supported formats are `rfc 3164 <https://tools.ietf.org/html/rfc3164>`__, `rfc 5424 <https://tools.ietf.org/html/rfc5424>`__, and Common Event Format (CEF).


.. important::
    This connector listens on a network port. Running more than one task or running in distributed mode can cause some undesired effects if another task already has the port open. It is recommended that you run this connector in :ref:`standalone-workers`.

Install Syslog Source Connector
-------------------------------

.. include:: ../includes/connector-install.rst

.. include:: ../includes/connector-install-hub.rst

.. codewithvars:: bash

   confluent-hub install confluentinc/kafka-connect-syslog:latest

.. include:: ../includes/connector-install-version.rst

.. codewithvars:: bash

   confluent-hub install confluentinc/kafka-connect-syslog:|release|

--------------------------
Install Connector Manually
--------------------------

`Download and extract the ZIP file <https://www.confluent.io/connector/kafka-connect-syslog/#download>`_ for your
connector and then follow the manual connector installation :ref:`instructions <connect_install_connectors>`.

-------
License
-------

.. include:: ../includes/enterprise-license.rst

See :ref:`syslog-source-connector-license-config` for license properties and :ref:`syslog-source-license-topic-configuration` for information about the license topic.

.. _syslog-source-connector-quickstart:

-----------
Quick Start
-----------

#.  Install the connector:

    ::

        confluent-hub install confluentinc/kafka-connect-syslog:latest

#.  Start Confluent Platform:

    ::

        |confluent_start| connect

#.  Create a config file with the following contents:

    ::

        name=syslog-tcp
        tasks.max=1
        connector.class=io.confluent.connect.syslog.SyslogSourceConnector
        syslog.port=5454
        syslog.listener=TCP
        confluent.license=
        confluent.topic.bootstrap.servers=localhost:9092
        confluent.topic.replication.factor=1


#.  Load the Syslog Connector.

    .. include:: ../../includes/confluent-local-consume-limit.rst

    .. codewithvars:: bash

       |confluent_load| syslog-tcp|dash| -d path/to/config.properties

    .. important:: Don't use the :ref:`cli` in production environments. Always run the Syslog connector in standalone mode, for example, with ``bin/connect-standalone``.

#.  Test with sample syslog-formatted message sent via ``netcat``:

    ::

        echo "<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - Your refrigerator is running" | nc -v -w 0 localhost 5454

#.  Confirm that the message is logged to |ak-tm|:

    ::

        kafka-avro-console-consumer \
            --bootstrap-server localhost:9092 \
            --property schema.registry.url=http://localhost:8081 \
            --topic syslog --from-beginning | jq '.'

-------------
Output Schema
-------------

The Syslog Connector accepts Syslog messages as strings, and produces structured messages as output to the configured |ak| topic.

Record Key Schema
^^^^^^^^^^^^^^^^^

``host``
  Host of the Syslog message, as specified in the message body. If no host is specified, the remote address of the message will be used.

  * Type: string

Record Value Schema
^^^^^^^^^^^^^^^^^^^

Some fields are relevant for only some message types, and will be null for others.

``name``
  String representing a human-readable and understandable description of the
  event. The event name should not contain information that is specifically
  mentioned in other fields. Used in CEF.

  * Type: string

``type``
  Type of message received (either "RFC5424", "RFC3164", "CEF", or "UNKNOWN").

  * Type: string

``message``
  The freeform message extracted from full message. "MSG" field of the Syslog spec.

  * Type: string

``host``
  Extracted host from the syslog message.

  * Type: string

``version``
  "VERSION" field of the Syslog spec.

  * Type: int

``level``
  Level as determined by "PRIORITY" field of Syslog spec.

  * Type: int

``tag``
  "TAG" field of Syslog spec

  * Type: string

``facility``
  "FACILITY" as determined by the "PRIORITY" of the Syslog spec.

  * Type: int

``severity``
  Severity of CEF messages.

  * Type: string

``appName``
  "APP-NAME" field of the Syslog spec.

  * Type: string

``remoteAddress``
  Remote address of the request received by the connector.

  * Type: string

``rawMessage``
  The full, unmodified, unparsed message as received by the connector.

  * Type: string

``processId``
  "PROCID" field of the Syslog spec.

  * Type: string

``messageId``
  "MSGID" field of the Syslog spec.

  * Type: string

``deviceVendor``
  Vendor identifier that is used to group products.

  * Type: string

``deviceProduct``
  Product identifier that is used for message logging.

  * Type: string

``deviceVersion``
  The version of the logging device product.

  * Type: string

``deviceEventClassId``
  The device event class ID. This is a unique per event-type identifier. The device event class ID identifies the type of event reported. In the intrusion detection system (IDS) world, each signature or rule that detects certain activity has a unique device event class ID assigned.
  This is a requirement for other types of devices as well, and helps
  correlation engines process the events. Also known as Signature ID.

  * Type: string

``extension``
  "Extension" mapping for the CEF message format.

  * Type: map<string, string>

``structuredData``
  "STRUCTURED-DATA" field of the Syslog spec.

  * Type: Array of StructuredDatum structs:

    ``id``: String
    ``StructuredDataElements``: map<string, string>


Additional Documentation
------------------------

.. toctree::
   :maxdepth: 1

   hostname_resolver_transform
   syslog_source_connector_config
   changelog