Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Kafka Connect ServiceNow Source Connector¶

The Kafka Connect ServiceNow Source Connector is used to move creations and updates in ServiceNow Table to Kafka in real-time. Data is consumed from the ServiceNow table to Kafka topics in a timely manner using range queries against the ServiceNow Table API.

Features¶

The ServiceNow Source Connector offers the following features:

Exactly-Once Semantics: The connector delivers messages only once under normal operation. The connector produces once for each real time updates from ServiceNow Table. However, duplicates are still possible to occur when failure, rescheduling or re-configuration happens.
Automatic Retries: The ServiceNow connector may experience network failures connecting to the ServiceNow endpoint. The connector will automatically retry to poll from the endpoint. The property retry.max.times controls how many times retries will be attempted. An exponential backoff will be added to each retry interval.
Supports HTTPS Proxy: The connector can connect to ServiceNow using an HTTPS proxy server.
Elasticity: The connector allows user to configure two parameters to enforce throughput limit: batch.max.rows and poll.interval.s. The connector uses 10000 records and 20s interval as a default. If a large number of updates occur within the given interval, the connector will paginate records according to configurable batch size. Note that since ServiceNow provides precision down to 1 second, the ServiceNow Connector provides lowest poll.interval.s to be the same.
Real-time and Historical Lookup: The connector supports a specific starting point in the history. It is capable of retrieving all historical records in one run and catching up with real-time records.

Prerequisites¶

The following are required to run the Kafka Connect ServiceNow Source Connector:

Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
Connect: Confluent Platform 4.1.0 or above, or Kafka 1.1.0 or above
ServiceNow API: New York, Madrid or London.

Install ServiceNow Source Connector¶

You can install this connector by using the Confluent Hub client (recommended) or you can manually download the ZIP file.

Install the connector using Confluent Hub¶

Prerequisite: Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.

Navigate to your Confluent Platform installation directory and run this command to install the latest (latest) connector version. The connector must be installed on every machine where Connect will be run.

confluent-hub install confluentinc/kafka-connect-servicenow-source:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install confluentinc/kafka-connect-servicenow-source:1.0.0-preview

Install Connector Manually¶

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License¶

You can use this connector for a 30-day trial period without a license key.

After 30 days, this connector is available under a Confluent enterprise license. Confluent issues enterprise license keys to subscribers, along with providing enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

See Confluent Platform license for license properties and License topic configuration for information about the license topic.

Quick Start¶

The quick start guide uses ServiceNow Connector to consume records from a ServiceNow Table and send them to Kafka. This guide assumes multi-tenant environment is used. For local testing, please refer to Running Connect in standalone mode.

Intsall the connector through the Confluent Hub Client.

# run from your confluent platform installation directory
confluent-hub install confluentinc/kafka-connect-servicenow-source:latest

Start the Confluent Platform.
```
confluent start
```
Check the status of all services.
```
confluent status
```

Create a servicenow.json file with the following contents:

// subsitute <> with your config
{
    "name": "ServiceNowConnector",
    "config": {
        "connector.class": "io.confluent.connect.servicenow.ServiceNowSourceConnector",
        "kafka.topic": "topic-servicenow",
        "servicenow.url": "https://<endpoint>.service-now.com/",
        "tasks.max": "1",
        "servicenow.table": "<table_name>",
        "servicenow.user": "<username>",
        "servicenow.password": "<password>",
        "servicenow.since": "2019-01-01",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "confluent.topic.bootstrap.servers": "<server hostname>:9092",
        "confluent.license": "<license>", // leave it empty for evaluation license
        "poll.interval.s": "10",
        "confluent.topic.replication.factor": "1"
    }
}

Load the ServiceNow Source Connector by posting configuration to Connect REST server.
```
confluent load servicenow -d servicenow.json
```
Confirm that the connector is in a RUNNING state.
```
confluent status ServiceNowConnector
```

Create one record to ServiceNow.

curl -X POST \
    https://<endpoint>.service-now.com/api/now/table/<table_name> \
    -H 'Accept: application/json' \
    -H 'Authorization: Basic <token>'
    -H 'Content-Type: application/json' \
    -H 'cache-control: no-cache' \
    -d '{"short_description": "This is test"}'

Confirm the messages were delivered to the topic-servicenow topic in Kafka.
```
confluent consume topic-servicenow --from-beginning
```