<a id="github-source-connector"></a>

# GitHub Source Connector for Confluent Platform

The Kafka Connect GitHub Source connector is used to write meta data (detect
changes in real time or consume the history) from GitHub to Apache Kafka® topics. This
connector polls data from GitHub through [GitHub APIs](https://developer.github.com/v3/), converts data into Kafka records, and then
pushes the records into a Kafka topic. Each record from GitHub is converted into
exactly one Kafka record.

#### IMPORTANT
<!-- WARNING: THIS IS A SHARED FILE AND THE SOURCE IS LOCATED IN DOCS-COMMON. DO NOT ADD TO ANY OTHER REPO. -->

Effective July 6, 2025, only self-managed connector versions that meet or exceed the minimum version listed on the
[Supported Connector Versions](https://docs.confluent.io/platform/7.8/connect/supported-connector-version.html#supported-connector-versions-till-cp-7-8)
page receive support from Confluent. Older, unsupported connector versions have been removed from Confluent Marketplace and
are no longer available for download.

## Features

The GitHub Source connector offers the following features:

- [At least once delivery](#github-source-at-least-once-delivery)
- [Multiple tasks](#github-source-supports-one-task)
- [API rate limit awareness](#github-source-api-rate-limit)
- [Supports HTTPS proxy](#github-source-supports-https-proxy)
- [Client-side encryption](#github-source-sink-csfle-sm)

<a id="github-source-at-least-once-delivery"></a>

### At least once delivery

This connector guarantees that records are delivered at least once to the Kafka
topic. If the connector restarts, there may be some duplicate records in the
Kafka topic.

<a id="github-source-supports-one-task"></a>

### Multiple tasks

The GitHub Source connector supports running only one task–one repository is
covered by one task.

<a id="github-source-api-rate-limit"></a>

### API rate limit awareness

The connector stops fetching records from GitHub when the API rate limit is
exceeded. Once the API rate limit resets, the connector will resume fetching
records.

<a id="github-source-supports-https-proxy"></a>

### Supports HTTPS proxy

The connector can connect to GitHub using an HTTPS proxy server. To configure
the proxy, you can set `http.proxy.host`, `http.proxy.port`,
`http.proxy.user` and `http.proxy.password` in the configuration file. The
connector has been tested with HTTPS proxy with basic authentication.

<a id="github-source-sink-csfle-sm"></a>

### Client-side encryption

This connector supports Client-Side Field Level Encryption (CSFLE) and Client-Side Payload Encryption (CSPE). For more information, see [Manage Client-Side Encryption](https://docs.confluent.io/platform/current/connect/manage-csfle.html).

## Limitations

* For resources that do not support fetching records by datetime, new records are fetched at an interval specified by the `request.interval.ms` configuration. Records for these resources might get duplicated every time connector restarts.
* The connector is not be able to detect the deletion of data on GitHub.
* In the case of connector restarts, the Kafka topic might end up having records that are out of order.
* GitHub has a defined API request limit. This limit is [5,000 requests](https://developer.github.com/apps/building-github-apps/understanding-rate-limits-for-github-apps/) per hour. Once this rate limit is exceeded, the connector waits until the API request limit resets.

## GitHub Resources

The GitHub connector supports fetching records from the following resources:

* **assignees:** Available assignees for the specified repositories, refer the following [schema](https://developer.github.com/v3/issues/assignees/).
* **collaborators:** Collaborators for the specified repositories, refer the following [schema](https://developer.github.com/v3/repos/collaborators/).
* **issues:** Issues in all GitHub states, refer the following [schema](https://developer.github.com/v3/issues/).
* **comments:** Issue comments, refer the following [schema](https://docs.github.com/en/rest/reference/collaborators).
* **commits:** Base branch commits only, refer the following [schema](https://docs.github.com/en/rest/reference/commits).
* **pull_requests:** Pull Requests in all GitHub states, refer the following [schema](https://developer.github.com/v3/pulls/).
* **releases:** Release for the specified repositories, refer the following [schema](https://docs.github.com/en/rest/reference/releases).
* **reviews:** Reviews on pull requests. Reviews can only be fetched with Pull Requests, refer the following [schema](https://developer.github.com/v3/pulls/reviews/).
* **review_comments:** Review comments on pull requests, refer the following [schema](https://docs.github.com/en/rest/reference/pulls#review-comments).
* **stargazers:** Stargazers for the specified repositories, refer the following [schema](https://developer.github.com/v3/activity/starring/).

## License

You can use this connector for a 30-day trial period without a license key.

After 30 days, you must purchase a connector subscription which includes [Confluent enterprise license](/platform/current/installation/license.html#enterprise-subscription-license) keys to subscribers, along with [enterprise-level support](https://www.confluent.io/subscription/) for Confluent Platform and your connectors. If you are a subscriber, you can contact [Confluent Support](https://support.confluent.io/) for more information.

See [Confluent Platform license](configuration_options.md#github-source-connector-license-config) for license properties and [Confluent License Properties](configuration_options.md#github-source-license-topic-configuration) for information about the license topic

## Configuration Properties

For a complete list of configuration properties for this connector, see [Configuration Reference for GitHub Source Connector for Confluent Platform](configuration_options.md#configuration-options).

For an example of how to get Kafka Connect connected to [Confluent Cloud](/cloud/current/index.html), see
[Connect Self-Managed Kafka Connect to Confluent Cloud](/cloud/current/cp-component/connect-cloud-config.html#distributed-cluster).

## Install the GitHub Source Connector

You can install this connector by using the [confluent connect
plugin
install](https://docs.confluent.io/confluent-cli/current/command-reference/connect/plugin/confluent_connect_plugin_install.html)
command, or by manually downloading the ZIP file.

### Prerequisites

- You must install the connector on every machine where Connect will run.
- Kafka Broker: Confluent Platform 3.3.0 or later.
- Connect: Confluent Platform 4.1.0 or later.
- Java 1.8.
- No additional setup is required on GitHub account for this connector to work,
  other than access token with repository and user privileges. For more
  details, [Creating a personal access token for the command line](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line).
- An installation of the latest (`latest`) connector version.

  To install the `latest` connector version, navigate to your Confluent Platform
  installation directory and run the following command:
  ```bash
  confluent connect plugin install confluentinc/kafka-connect-github:latest
  ```

  You can install a specific version by replacing `latest` with a version
  number as shown in the following example:
  ```bash
  confluent connect plugin install confluentinc/kafka-connect-github:2.1.1
  ```

### Install the connector manually

[Download and extract the ZIP file](https://www.confluent.io/hub/confluentinc/kafka-connect-github) for your connector and then follow the manual connector installation [instructions](/kafka-connectors/self-managed/userguide.html#connect-installing-plugins).

## Quick Start

In this quick start, you configure the GitHub Source connector to fetch GitHub users who have stared [Apache Kafka repository](https://github.com/apache/kafka) since 2019-01-01 to a Kafka topic called github-stargazers.

### Start Confluent

Start the Confluent services using the following [Confluent CLI](https://docs.confluent.io/confluent-cli/current/index.html) command:

```bash
confluent local services start
```

#### IMPORTANT
Do not use the [Confluent CLI](https://docs.confluent.io/confluent-cli/current/index.html) in production environments.

### Properties-based example

Create a file called github-source-quickstart.properties file with following properties:

```bash
name=MyGithubConnector
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
tasks.max=1
connector.class=io.confluent.connect.github.GithubSourceConnector
github.service.url=https://api.github.com
github.access.token=<ACCESS-TOKEN>
github.repositories=apache/kafka
github.resources=stargazers
github.since=2019-01-01
topic.name.pattern=github-${resourceName}
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
```

Next, load the Source connector.

```bash
.confluent local load MyGithubConnector --config github-source-quickstart.properties
```

Your output should resemble the following:

```bash
{
    "name": "MyGithubConnector",
    "config": {
        "connector.class": "io.confluent.connect.github.GithubSourceConnector",
        "tasks.max": "1",
        "confluent.topic.bootstrap.servers":"localhost:9092",
        "confluent.topic.replication.factor":"1",
        "github.service.url":"https://api.github.com",
        "github.repositories":"apache/kafka",
        "github.resources":"stargazers",
        "github.since":"2019-01-01",
        "github.access.token":"<Your-Github-Access-Token>",
        "topic.name.pattern":"github-${resourceName}",
        "key.converter":"io.confluent.connect.avro.AvroConverter",
        "key.converter.schema.registry.url":"http://localhost:8081",
        "value.converter":"io.confluent.connect.avro.AvroConverter",
        "value.converter.schema.registry.url":"http://localhost:8081"
    },
    "tasks": [],
    "type": null
}
```

Enter the following command to confirm that the connector is in a `RUNNING` state:

```bash
confluent local status MyGithubConnector
```

The output should resemble:

```bash
{
   "name":"MyGithubConnector",
   "connector":
   {
      "state":"RUNNING",
      "worker_id":"127.0.1.1:8083"
   },
   "tasks":
   [
      {
         "id":0,
         "state":"RUNNING",
         "worker_id":"127.0.1.1:8083"
      }
   ],
   "type":"source"
}
```

### REST-based example

Use this setting with [distributed workers](/platform/current/connect/concepts.html#distributed-workers). Write the following JSON to `config.json`, configure all of the required values, and use the following command to post the configuration to one of the distributed Connect workers. Check here for more information about the Kafka Connect [REST API](/platform/current/connect/references/restapi.html).

```json
{
   "name" : "MyGithubConnector",
   "config" :
   {
      "connector.class" : "io.confluent.connect.github.GithubSourceConnector",
      "confluent.topic.bootstrap.servers": "localhost:9092",
      "confluent.topic.replication.factor": "1",
      "tasks.max" : "1",
      "github.service.url":"https://api.github.com",
      "github.access.token":"< Github-Access-Token >",
      "github.repositories":"apache/kafka",
      "github.resources":"stargazers",
      "github.since":"2019-01-01",
      "topic.name.pattern":"github-${resourceName}",
      "key.converter":"io.confluent.connect.avro.AvroConverter",
      "key.converter.schema.registry.url":"http://localhost:8081",
      "value.converter":"io.confluent.connect.avro.AvroConverter",
      "value.converter.schema.registry.url":"http://localhost:8081"
   }
}
```

#### NOTE
For staging or production use:

- Change the `confluent.topic.bootstrap.servers` property to include your
  broker address(es).
- Change the `confluent.topic.replication.factor` to `3` for staging or
  production use.
- Change `http://localhost:8083/` to the endpoint of one of your Connect
  worker(s).

Use curl to post a configuration to one of the Connect workers.

```bash
curl -sS -X POST -H 'Content-Type: application/json' --data @config.json http://localhost:8083/connectors
```

Confirm that the connector is in a `RUNNING` state by running the following command:

```bash
curl http://localhost:8083/connectors/MyGithubConnector/status
```

The output should resemble the example below:

```bash
{
   "name":"MyGithubConnector",
   "connector":{
      "state":"RUNNING",
      "worker_id":"127.0.1.1:8083"
   },
   "tasks":[
      {
         "id":0,
         "state":"RUNNING",
         "worker_id":"127.0.1.1:8083"
      }
   ],
   "type":"source"
}
```

Enter the following command to consume records written by the connector to the Kafka topic:

```bash
./kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic github-stargazers --from-beginning
```

The output should resemble the example below:

```bash
{
    "type": {
      "string": "STARGAZERS"
    },
    "createdAt": null,
    "data": {
      "data": {
        "login": {
          "string": "User.Name"
        },
        "id": {
          "int": 1234
        },
        "node_id": {
          "string": "MDQ6VXNlcjM0OTE3MTE="
        },
        "avatar_url": {
          "string": "https://avatars2.githubusercontent.com/u/1234?v=4"
        },
        "gravatar_id": {
          "string": ""
        },
        "url": {
          "string": "https://api.github.com/users/User.Name"
        },
        "html_url": {
          "string": "https://github.com/User.Name"
        },
        "followers_url": {
          "string": "https://api.github.com/users/User.Name/followers"
        },
        "following_url": {
          "string": "https://api.github.com/users/User.Name/following{/other_user}"
        },
        "gists_url": {
          "string": "https://api.github.com/users/User.Name/gists{/gist_id}"
        },
        "starred_url": {
          "string": "https://api.github.com/users/User.Name/starred{/owner}{/repo}"
        },
        "subscriptions_url": {
          "string": "https://api.github.com/users/User.Name/subscriptions"
        },
        "organizations_url": {
          "string": "https://api.github.com/users/User.Name/orgs"
        },
        "repos_url": {
          "string": "https://api.github.com/users/User.Name/repos"
        },
        "events_url": {
          "string": "https://api.github.com/users/User.Name/events{/privacy}"
        },
        "received_events_url": {
          "string": "https://api.github.com/users/User.Name/received_events"
        },
        "type": {
          "string": "User"
        },
        "site_admin": {
          "boolean": false
        }
      }
    },
    "id": {
      "string": "1234"
    }
  }
```

### Clean up resources

To clean up the resources, complete the following steps:

1. Delete the connector:
   ```bash
   confluent local unload MyGithubConnector
   ```
2. Stop Confluent Platform:
   ```bash
   confluent local stop
   ```