HTTP Source Connector for Confluent Platform¶

The Kafka Connect HTTP Source connector makes HTTP requests to an API, processes JSON responses, and produce those responses to Kafka. You can use it to periodically poll a JSON-based HTTP API and produce the resulting data to Kafka.

Features¶

The HTTP Source connector for Confluent Platform includes the following features:

At least once delivery: The connector guarantees that records are delivered at least once to the Kafka topic.

Offset modes: The connector supports the following modes:

SIMPLE_INCREMENTING: For the first request, the connector computes the offset based on the initial offset set in the http.initial.offset property. For subsequent requests, the connector increments the offset by the number of records in the previous response. For instance, the connector increments the offset by one for an object type response that indicates a single record. For an array type response, the connector increments the offset by the length of the array.
CHAINING: For the first request, the connector computes the offset based on the initial offset set in the http.initial.offset property. Upon receiving a successful response to the first request, the connector parses the response and generates one or more records (depending on whether the JSON response is an object or an array) with the offset value set to the value from the response at the path configured in the http.offset.json.pointer configuration property.
CURSOR_PAGINATION: In this mode, the property http.next.page.json.pointer is used to configure the offset value. The offset for the last record in each page is set to the next page value. For additional information, see Offset Mode Use Case Examples.

Multiple tasks: The connector supports running one or more tasks. More tasks may improve performance when multiple entities are configured.

Output data formats: The connector supports Avro, JSON Schema, Protobuf, and JSON (schemaless) output record value formats. Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON Schema, or Protobuf).

Limitations¶

The HTTP Source connector for Confluent Platform includes the following limitations:

The connector does not support APIs that rely on timestamp range-based queries.
The connector cannot parse responses in any format other than JSON.

Install the HTTP Source connector¶

You can install this connector by using the confluent connect plugin install command, or by manually downloading the ZIP file.

Prerequisites¶

You must install the connector on every machine where Connect will run.
An installation of the Confluent CLI.
An installation of the latest (latest) connector version.

Install the connector using the Confluent CLI¶

To install the latest connector version, navigate to your Confluent Platform installation directory and run the following command:

confluent connect plugin install confluentinc/kafka-connect-http-source:latest

You can install a specific version by replacing latest with a version number as shown in the following example:

confluent connect plugin install confluentinc/kafka-connect-http-source:0.2.0

Install the connector manually¶

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License¶

You can use this connector for a 30-day trial period without a license key.

After 30 days, you must purchase a connector subscription which includes Confluent enterprise license keys to subscribers, along with enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, you can contact Confluent Support at support@confluent.io for more information.

For license properties, see Configuration Reference for HTTP Source Connector for Confluent Platform.

Configuration properties¶

For a complete list of configuration properties for this connector, see Configuration Reference for HTTP Source Connector for Confluent Platform.

Quick start¶

This quick start guide will help you set up the Confluent Platform HTTP Source Connector to consume records from an HTTP endpoint and send them to Kafka. This guide assumes you are using a multi-tenant environment. For local testing, refer to Running Connect in standalone mode.

Install the connector using the following Confluent CLI command. Note that you must run this command from your Confluent Platform installation directory:
```
confluent-hub install confluentinc/kafka-connect-http-source:latest
```
Start Confluent Platform.
```
confluent local start
```
Check the status of all services.
```
confluent local services status
```

Create a http-source.json file with the following contents:

  {
    "name": "HttpSourceConnector",
    "config": {
      "connector.class": "io.confluent.connect.http.HttpSourceConnector",
      "topic.name.pattern": "topic-http",
      "url": "https://<endpoint>.com/api/data",
      "tasks.max": "1",
      "http.offset.mode": "SIMPLE_INCREMENTING",
      "key.converter": "org.apache.kafka.connect.json.JsonConverter",
      "value.converter": "org.apache.kafka.connect.json.JsonConverter",
      "confluent.topic.bootstrap.servers": "<server hostname>:9092",
      "confluent.license": "", // leave it empty for evaluation license
      "confluent.topic.replication.factor": "1"
  }
}

Load the Http Source connector by posting configuration to Connect REST server. Note that you must include a double dash (--) between the topic name and your flag. For more information, see What does “–” (double dash / double hyphen) mean?.

Confirm the connector is in RUNNING state by entering the following command:

confluent local services connect connector status HttpSourceConnector

Verify the connector is working by simulating sending data to the HTTP endpoint. For example, use curl to POST data.

curl -X POST \
  https://<endpoint>.com/api/data \
  -H 'Accept: application/json' \
  -H 'Authorization: Basic <token>' \
  -H 'Content-Type: application/json' \
  -H 'cache-control: no-cache' \
  -d '{"data": "This is a test"}'

Confirm the messages were delivered to the topic-http topic in Kafka.

confluent local services kafka consume topic-http --from-beginning