Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

ServiceNow Source Connector for Confluent Platform¶

The Kafka Connect ServiceNow Source Connector is used to move creations and updates in ServiceNow Table to Apache Kafka® in real-time. Data is consumed from the ServiceNow table to Kafka topics in a timely manner using range queries against the ServiceNow Table API.

Features¶

The ServiceNow Source Connector offers the following features:

Delivery Guarantee: Under normal operation, the connector delivers each ServiceNow update to Kafka only once. However, duplicates are still possible in several situations. These situations include: during temporary network degradation, after non-graceful shutdowns, or when connectors and tasks are rescheduled.
Automatic Retries: The connector may experience network failures connecting to the ServiceNow endpoint. The connector will automatically retry to poll from the endpoint. The property retry.max.times controls how many times retries are attempted. An exponential backoff is added to each retry interval.
Supports HTTPS Proxy: The connector can connect to ServiceNow using an HTTPS proxy server.
Elasticity: The connector allows you to configure two parameters that enforce the throughput limit: batch.max.rows and poll.interval.s. The connector uses 10000 records and 20s interval as a default. If a large number of updates occur within the given interval, the connector will paginate records according to configurable batch size. Note that since ServiceNow provides precision down to one second, the ServiceNow Connector provides one second as the lowest poll.interval.s configuration property setting.
Real-time and Historical Lookup: The connector supports a specific starting point in the history. It is capable of retrieving all historical records in one run and catching up with real-time records.

Configuration Properties¶

For a complete list of configuration properties for this connector, see ServiceNow Source Connector Configuration Properties.

Note

For an example of how to get Kafka Connect connected to Confluent Cloud, see Distributed Cluster in Connect Kafka Connect to Confluent Cloud.

Prerequisites¶

The following are required to run the Kafka Connect ServiceNow Source Connector:

Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
Connect: Confluent Platform 4.1.0 or above, or Kafka 1.1.0 or above
ServiceNow API: New York, Madrid or London.

Quick Start¶

The quick start guide uses ServiceNow source connector to consume records from a ServiceNow Table and send them to Kafka. This guide assumes multi-tenant environment is used. For local testing, please refer to Running Connect in standalone mode.

Install the connector through the Confluent Hub Client.

# run from your confluent platform installation directory
confluent-hub install confluentinc/kafka-connect-servicenow:latest

Start the Confluent Platform.

Tip

The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local start. For more information, see confluent local.
```
confluent local start
```
Check the status of all services.
```
confluent local status
```

Create a servicenow-source.json file with the following contents:

// subsitute <> with your config
{
    "name": "ServiceNowSourceConnector",
    "config": {
        "connector.class": "io.confluent.connect.servicenow.ServiceNowSourceConnector",
        "kafka.topic": "topic-servicenow",
        "servicenow.url": "https://<endpoint>.service-now.com/",
        "tasks.max": "1",
        "servicenow.table": "<table_name>",
        "servicenow.user": "<username>",
        "servicenow.password": "<password>",
        "servicenow.since": "2019-01-01",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "confluent.topic.bootstrap.servers": "<server hostname>:9092",
        "confluent.license": "<license>", // leave it empty for evaluation license
        "poll.interval.s": "10",
        "confluent.topic.replication.factor": "1"
    }
}

Load the ServiceNow Source Connector by posting configuration to Connect REST server.

Caution

You must include a double dash (--) between the topic name and your flag. For more information, see this post.
```
confluent local load servicenow -- -d servicenow-source.json
```

Confirm that the connector is in a RUNNING state.

confluent local status ServiceNowSourceConnector

Create one record to ServiceNow.

curl -X POST \
    https://<endpoint>.service-now.com/api/now/table/<table_name> \
    -H 'Accept: application/json' \
    -H 'Authorization: Basic <token>'
    -H 'Content-Type: application/json' \
    -H 'cache-control: no-cache' \
    -d '{"short_description": "This is test"}'

Confirm the messages were delivered to the topic-servicenow topic in Kafka.
```
confluent local consume topic-servicenow --from-beginning
```

Additional Documentation¶

ServiceNow Source Connector Configuration Properties