1. Home
  2. Connect External Systems to Confluent Cloud

GitHub Source Connector for Confluent Cloud

The Kafka Connect GitHub Source connector for Confluent Cloud is used to write metadata from GitHub to Apache Kafka®. This includes consuming real-time changes or historical data and writing these to a Kafka topic. The connector polls data from GitHub through GitHub APIs, converts data into Kafka records, and then pushes the records into a Kafka topic. Each record from GitHub is converted into one Kafka record.

Important

If you are still on Confluent Cloud Enterprise, please contact your Confluent Account Executive for more information about using this connector.

Features

The GitHub Source connector provides the following features:

  • At least once delivery: The connector guarantees that records are delivered at least once to the Kafka topic.
  • API rate limit awareness: The connector stops fetching records from GitHub when the API rate limit is exceeded. Once the API rate limit resets, the connector will resume fetching records.
  • Supported data formats: The connector supports Avro, JSON Schema (JSON-SR), Protobuf, and JSON (schemaless) output formats. Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON Schema, or Protobuf).

Note

Because of a GitHub API limitation, only one task per connector is supported.

See Configuration Properties for configuration property descriptions. See Cloud connector limitations for additional information.

GitHub Resources

The GitHub connector supports fetching records from the following resources:

  • assignees: Available assignees for the specified repositories. For more information, see the Assignees API doc.
  • collaborators: Collaborators for the specified repositories. For more information, see the Collaborators API doc.
  • issues: Issues in all GitHub states. For more information, see the Issues API doc.
  • comments: Issue comments. For more information, see the Comments API doc.
  • commits: Master branch commits (only). For more information, see the Commits API doc.
  • pull_requests: Pull Requests in all GitHub states. For more information, see the Pulls API doc.
  • releases: Release for the specified repositories. For more information, see the Releases API doc.
  • reviews: Reviews on pull requests. Reviews can only be fetched with Pull Requests. For more information, see the Pulls API doc.
  • review_comments: Review comments on pull requests. For more information, see the Pulls API doc.
  • stargazers: Stargazers for the specified repositories. For more information, see the Starring API doc.

Quick Start

Use this quick start to get up and running with the Confluent Cloud GitHub Source connector. The quick start provides the basics of selecting the connector and configuring it to stream events.

Prerequisites
  • Authorized access to a Confluent Cloud cluster on Amazon Web Services (AWS), Microsoft Azure (Azure), or Google Cloud Platform (GCP).
  • The Confluent Cloud CLI installed and configured for the cluster. See Install and Configure the Confluent Cloud CLI.
  • Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf).
  • Authorization and credentials to access the GitHub endpoint.
  • At least one Kafka topic must exist in your Confluent Cloud cluster before creating the source connector.

Using the Confluent Cloud Console

Step 1: Launch your Confluent Cloud cluster.

See the Quick Start for Apache Kafka using Confluent Cloud for installation instructions.

Step 2: Add a connector.

In the left navigation menu, click Data integration, and then click Connectors. If you already have connectors in your cluster, click + Add connector.

Step 3: Select your connector.

Click the GitHub Source connector icon.

GitHub Source Connector Icon

Step 4: Set up the connection.

Note

  • Make sure you have all your prerequisites completed.
  • An asterisk ( * ) designates a required entry.
  1. Enter a connector Name.
  2. Enter your Kafka Cluster credentials. The credentials are either the cluster API key and secret or the service account API key and secret.
  3. Enter the Topic Name Pattern. The pattern to use for the topic name. The pattern to use for the topic name, where ${resourceName} is replaced with a resource name. The default value is ${resourceName}.
  4. Enter the GitHub connection details:
    • GitHub Endpoint: The GitHub API root endpoint. The default used is https://api.github.com.
    • GitHub Repositories: GitHub repository or comma-separated list of repositories in the form owner/repo-name. For example, "apache/kafka, confluentinc/ksql".
    • GitHub Resources: One or more resources that the connector extracts and writes to Kafka. See GitHub Resources for details.
    • Since: Records created or updated after this time will be processed by the connector. If left blank, the default time is set to the time the connector is launched.
  5. Enter the connector details:
    • Maximum Batch Size: Defaults to 100 records.
    • Maximum In Flight requests: The maximum number of requests that may be in flight at one time. Defaults to 10 requests.
    • Maximum Poll interval (ms): The time in milliseconds (ms) between requests to fetch changed or updated resources. Defaults to 3000 ms (3 seconds).
    • Request Interval (ms): The time in milliseconds to wait before checking for updated records. Defaults to 15000 ms (15 seconds).
    • Maximum Retries: The maximum number of times the connector retries a task before the task fails. Defaults to 10 retries.
    • Retry Backoff (ms): The time in milliseconds to wait following an error before a retry attempt is made. Defaults to 3000 ms (3 seconds).
  6. Select an Output message format (data going to the Kafka topic): AVRO, JSON_SR (JSON Schema), PROTOBUF, or JSON (schemaless). Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf).
  7. Enter the number of tasks to use with the connector. Because of a GitHub API limitation, only one task per connector is supported.
  8. Transforms and Predicates: See the Single Message Transforms (SMT) documentation for details.

See Configuration Properties for configuration property values and descriptions.

Step 5: Launch the connector.

Verify the connection details and click Launch.

Launch the connector

Step 6: Check the connector status.

The status for the connector should go from Provisioning to Running.

Connector status

Step 7: Check for records.

Verify that records are being produced at the Kafka topic.

For more information and examples to use with the Confluent Cloud API for Connect, see the Confluent Cloud API for Connect section.

Using the Confluent Cloud CLI

Complete the following steps to set up and run the connector using the Confluent Cloud CLI.

Note

Make sure you have all your prerequisites completed.

Step 1: List the available connectors.

Enter the following command to list available connectors:

ccloud connector-catalog list

Step 2: Show the required connector configuration properties.

Enter the following command to show the required connector properties:

ccloud connector-catalog describe <connector-catalog-name>

For example:

ccloud connector-catalog describe GithubSource

Example output:

Following are the required configs:
connector.class: GithubSource
name
kafka.api.key
kafka.api.secret
github.service.url
github.access.token
github.repositories
github.resources
output.data.format
tasks.max

Step 3: Create the connector configuration file.

Create a JSON file that contains the connector configuration properties. The following example shows the required connector properties. See Configuration Properties for additional configuration property values and descriptions.

{
  "connector.class": "GithubSource",
  "name": "GithubSource_0",
  "kafka.api.key": "****************",
  "kafka.api.secret": "************************************************",
  "github.service.url": "https://api.github.com`",
  "github.access.token": "*********************************",
  "github.repositories": "<owner/repo-name>",
  "github.resources": "pull_requests, reviews, review_comments",
  "output.data.format": "AVRO",
  "tasks.max": "1",
}

Note the following property definitions:

  • "connector.class": Identifies the connector plugin name.
  • "name": Sets a name for your new connector.
  • "kafka.api.key" and ""kafka.api.secret": These credentials are either the cluster API key and secret or the service account API key and secret.
  • Enter the GitHub connection details.
    • "github.service.url": The GitHub API root endpoint. The default used is https://api.github.com.
    • "github.repositories": GitHub repository or comma-separated list of repositories in the form owner/repo-name. For example, "apache/kafka, confluentinc/ksql".
    • "github.resources": One or more resources that the connector extracts and writes to Kafka. See GitHub Resources for details.
  • “output.data.format": Enter an output data format (data going to the Kafka topic): AVRO, JSON_SR (JSON Schema), PROTOBUF, or JSON (schemaless). Schema Registry must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf).
  • "tasks.max": Enter the number of tasks to use with the connector. Because of a GitHub API limitation, only one task per connector is supported.
  1. Transforms and Predicates: See the Single Message Transforms (SMT) documentation for details.

See Configuration Properties for configuration property descriptions.

Step 4: Load the properties file and create the connector.

Enter the following command to load the configuration and start the connector:

ccloud connector create --config <file-name>.json

For example:

ccloud connector create --config github-source-config.json

Example output:

Created connector GithubSource_0 lcc-do6vzd

Step 5: Check the connector status.

Enter the following command to check the connector status:

ccloud connector list

Example output:

ID           |             Name         | Status  | Type  | Trace
+------------+--------------------------+---------+--------+-------+
lcc-do6vzd   | GithubSource_0           | RUNNING | source |       |

Step 6: Check for records.

Verify that records are being produced at the Kafka topic.

For more information and examples to use with the Confluent Cloud API for Connect, see the Confluent Cloud API for Connect section.

Configuration Properties

The following connector configuration properties are used with the GitHub Source connector for Confluent Cloud.

topic.name.pattern

The pattern to use for the topic name, where ${resourceName} is replaced with a resource name.

  • Type: string
  • Default: ${resourceName}
  • Importance: high
github.service.url

GitHub Service URL. For example, “https://api.github.com”.

  • Type: string
  • Valid Values: URL with one of these schemes: http, https
  • Importance: high
github.access.token

The supplied token is used as the value of Authorization header in HTTP requests, To generate an access token visit Github Documentation.

  • Type: string
  • Valid Values: Non-empty string
  • Importance: high
github.repositories

The Github repositories to read from, in the form of owner/repo-name. For example, "confluentinc/ksql, apache/kafka".

  • Type: list (List of repositories, separated by commas)
  • Valid Values: Non-empty list
  • Importance: high
github.resources

One or more resources that the connector extracts and writes to Kafka. See GitHub Resources for details.

  • Type: list (List of resources, separated by commas)
  • Valid Values: [ assignees, collaborators, comments, commits, issues, pull_requests, releases, review_comments, reviews, stargazers]
  • Importance: high
github.since

Records created or updated after this time will be processed by the connector. If left blank, the default time is set to the time the connector is launched.

  • Type: string
  • Default: “<Current DateTime>”
  • Valid Values: Either an empty string or date in one of the following formats: YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ.
  • Importance: medium
max.batch.size

The maximum number of batched records that can be returned and written to Kafka at one time.

  • Type: int
  • Default: 100
  • Valid Values: [1,…,2147483647]
  • Importance: high
max.in.flight.requests

The maximum number of requests that may be in-flight at once.

  • Type: int
  • Default: 10
  • Valid Values: [1,…,200]
  • Importance: high
max.poll.interval.ms

The time in milliseconds to wait while polling for a full batch of records.

  • Type: long
  • Default: 3000
  • Valid Values: [1,…,300000]
  • Importance: medium
request.interval.ms

The time in milliseconds to wait before checking for updated records.

  • Type: long
  • Default: 15000
  • Valid Values: [1,…,86400000]
  • Importance: medium
max.retries

The maximum number of times to retry on errors before failing the task.

  • Type: int
  • Default: 10
  • Valid Values: [0,…,2147483647]
  • Importance: medium
retry.backoff.ms

The time in milliseconds to wait following an error before a retry attempt is made.

  • Type: int
  • Default: 3000
  • Valid Values: [0,…,2147483647]
  • Importance: medium

Next Steps

See also

For an example that shows fully-managed Confluent Cloud connectors in action with Confluent Cloud ksqlDB, see the Cloud ETL Demo. This example also shows how to use Confluent Cloud CLI to manage your resources in Confluent Cloud.

../_images/topology.png