HTTP Source Connector for Confluent Platform¶
The Kafka Connect HTTP Source connector makes HTTP requests to an API, processes JSON responses, and produce those responses to Kafka. You can use it to periodically poll a JSON-based HTTP API and produce the resulting data to Kafka.
Features¶
The HTTP Source connector for Confluent Platform includes the following features:
At least once delivery: The connector guarantees that records are delivered at least once to the Kafka topic.
Offset modes: The connector supports the following modes:
SIMPLE_INCREMENTING
: For the first request, the connector computes the offset based on the initial offset set in thehttp.initial.offset
property. For subsequent requests, the connector increments the offset by the number of records in the previous response. For instance, the connector increments the offset by one for an object type response that indicates a single record. For an array type response, the connector increments the offset by the length of the array.CHAINING
: For the first request, the connector computes the offset based on the initial offset set in thehttp.initial.offset
property. Upon receiving a successful response to the first request, the connector parses the response and generates one or more records (depending on whether the JSON response is an object or an array) with the offset value set to the value from the response at the path configured in thehttp.offset.json.pointer
configuration property.CURSOR_PAGINATION
: In this mode, the propertyhttp.next.page.json.pointer
is used to configure the offset value. The offset for the last record in each page is set to the next page value. For additional information, see Offset Mode Use Case Examples.
Multiple tasks: The connector supports running one or more tasks. More tasks may improve performance when multiple entities are configured.
Output data formats: The connector supports Avro, JSON Schema, Protobuf, and JSON (schemaless) output record value formats. Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON Schema, or Protobuf).
Limitations¶
The HTTP Source connector for Confluent Platform includes the following limitations:
- The connector does not support APIs that rely on timestamp range-based queries.
- The connector cannot parse responses in any format other than JSON.
Install the HTTP Source connector¶
You can install this connector by using the confluent connect plugin install command, or by manually downloading the ZIP file.
Prerequisites¶
- You must install the connector on every machine where Connect will run.
- An installation of the Confluent CLI.
- An installation of the latest (
latest
) connector version.
Install the connector using the Confluent CLI¶
To install the latest
connector version, navigate to your Confluent Platform installation
directory and run the following command:
confluent connect plugin install confluentinc/kafka-connect-http-source:latest
You can install a specific version by replacing latest
with a version number
as shown in the following example:
confluent connect plugin install confluentinc/kafka-connect-http-source:0.2.0
Install the connector manually¶
Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.
License¶
You can use this connector for a 30-day trial period without a license key.
After 30 days, you must purchase a connector subscription which includes Confluent enterprise license keys to subscribers, along with enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, you can contact Confluent Support at support@confluent.io for more information.
For license properties, see Configuration Reference for HTTP Source Connector for Confluent Platform.
Configuration properties¶
For a complete list of configuration properties for this connector, see Configuration Reference for HTTP Source Connector for Confluent Platform.
Quick start¶
This quick start guide will help you set up the Confluent Platform HTTP Source Connector to consume records from an HTTP endpoint and send them to Kafka. This guide assumes you are using a multi-tenant environment. For local testing, refer to Running Connect in standalone mode.
Install the connector using the following Confluent CLI command. Note that you must run this command from your Confluent Platform installation directory:
confluent-hub install confluentinc/kafka-connect-http-source:latest
Start Confluent Platform.
confluent local start
Check the status of all services.
confluent local services status
Create a
http-source.json
file with the following contents:{ "name": "HttpSourceConnector", "config": { "connector.class": "io.confluent.connect.http.HttpSourceConnector", "topic.name.pattern": "topic-http", "url": "https://<endpoint>.com/api/data", "tasks.max": "1", "http.offset.mode": "SIMPLE_INCREMENTING", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "confluent.topic.bootstrap.servers": "<server hostname>:9092", "confluent.license": "", // leave it empty for evaluation license "confluent.topic.replication.factor": "1" } }
Load the Http Source connector by posting configuration to Connect REST server. Note that you must include a double dash (
--
) between the topic name and your flag. For more information, see What does “–” (double dash / double hyphen) mean?.Confirm the connector is in
RUNNING
state by entering the following command:confluent local services connect connector status HttpSourceConnector
Verify the connector is working by simulating sending data to the HTTP endpoint. For example, use
curl
to POST data.curl -X POST \ https://<endpoint>.com/api/data \ -H 'Accept: application/json' \ -H 'Authorization: Basic <token>' \ -H 'Content-Type: application/json' \ -H 'cache-control: no-cache' \ -d '{"data": "This is a test"}'
Confirm the messages were delivered to the
topic-http
topic in Kafka.confluent local services kafka consume topic-http --from-beginning