# GitHub Source Connector for Confluent Cloud The fully managed GitHub Source connector for Confluent Cloud writes metadata from GitHub to Apache Kafka® topics. This includes consuming real-time changes or historical data and writing these to a Kafka topic. The connector polls data from GitHub through GitHub APIs, converts data into Kafka records, and then pushes the records into a Kafka topic. Each record from GitHub is converted into one Kafka record. #### NOTE * This Quick Start is for the fully managed Confluent Cloud connector. If you are installing the connector locally for Confluent Platform, see [GitHub Source Connector for Confluent Platform](https://docs.confluent.io/kafka-connectors/github/current/). * If you require private networking for fully managed connectors, make sure to set up the proper networking beforehand. For more information, see [Manage Networking for Confluent Cloud Connectors](networking/internet-resource.md#clusters-connect-cloud). ## Features The GitHub Source connector provides the following features: * **At least once delivery**: The connector guarantees that records are delivered at least once to the Kafka topic. * **API rate limit awareness**: The connector stops fetching records from GitHub when the API rate limit is exceeded. Once the API rate limit resets, the connector will resume fetching records. * **Supported data formats**: The connector supports Avro, JSON Schema (JSON-SR), Protobuf, and JSON (schemaless) output formats. [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) must be enabled to use a Schema Registry-based format (for example, Avro, JSON Schema, or Protobuf). * **Offset management capabilities**: Supports offset management. For more information, see [Manage custom offsets](#cc-github-source-custom-offsets). For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## Limitations Be sure to review the following information. * For connector limitations, see [GitHub Source Connector](limits.md#cc-github-source-limits) limitations. * If you plan to use one or more Single Message Transformations (SMTs), see [SMT Limitations](single-message-transforms.md#cc-single-message-transforms-limitations). #### NOTE Because of a GitHub API limitation, only one task per connector is supported. ## GitHub Resources The GitHub connector supports fetching records from the following resources: * **assignees:** Available assignees for the specified repositories. For more information, see the [Assignees API doc](https://docs.github.com/en/rest/reference/issues#assignees). * **collaborators:** Collaborators for the specified repositories. For more information, see the [Collaborators API doc](https://docs.github.com/en/rest/reference/repos#collaborators). * **issues:** Issues in all GitHub states. For more information, see the [Issues API doc](https://docs.github.com/en/rest/reference/issues). * **comments:** Issue comments. For more information, see the [Comments API doc](https://docs.github.com/en/rest/reference/repos#comments). * **commits:** Master branch commits (only). For more information, see the [Commits API doc](https://docs.github.com/en/rest/reference/repos#commits). * **pull_requests:** Pull Requests in all GitHub states. For more information, see the [Pulls API doc](https://docs.github.com/en/rest/reference/pulls). * **releases:** Release for the specified repositories. For more information, see the [Releases API doc](https://docs.github.com/en/rest/reference/repos#releases). * **reviews:** Reviews on pull requests. Reviews can only be fetched with Pull Requests. For more information, see the [Pulls API doc](https://docs.github.com/en/rest/reference/pulls#reviews). * **review_comments:** Review comments on pull requests. For more information, see the [Pulls API doc](https://docs.github.com/en/rest/reference/pulls#review-comments). * **stargazers:** Stargazers for the specified repositories. For more information, see the [Starring API doc](https://docs.github.com/en/rest/reference/activity#starring). ## Manage custom offsets You can manage the offsets for this connector. Offsets provide information on the point in the system from which the connector is accessing data. For more information, see [Manage Offsets for Fully Managed Connectors in Confluent Cloud](offsets.md#connect-custom-offsets). **To manage offsets**: - Manage offsets using Confluent Cloud APIs. For more information, see [Connect offsets API reference](https://docs.confluent.io/cloud/current/ccloud/offsets-connect-v-1/). ### Get the current offset To get the current offset, make a `GET` request that specifies the environment, Kafka cluster, and connector name. ```bash GET /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets Host: https://api.confluent.cloud ``` **Response:** Successful calls return HTTP `200` with a JSON payload that describes the offset. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [ { "partition": { "name": "_" }, "offset": { "etag": "\"736d269d94ebc0f1c3e4ceceb2fbbd28f73e80bcc0d1bd6b202596b349afadc5\"", "id": "2230425971", "since": "2024-04-08T07:05:26Z" } }, { "partition": { "name": "_" }, "offset": { "etag": "\"20e5d952ac21f31ac368394ed0b509a7d00d5727a4fea8f6eab0b949997029db\"", "id": "2217061690", "since": "2024-03-31T16:04:29Z" } }, { "partition": { "name": "_" }, "offset": { "etag": "\"727dc3d687ad3f6a0122aad0b1fbc24b003189df1729bc9180f5895345ec93b7\"", "id": "2234852695", "since": "2024-04-10T06:29:14Z" } } ], "metadata": { "observed_at": "2024-03-28T17:57:48.139635200Z" } } ``` Responses include the following information: - The position of latest offset. - The observed time of the offset in the metadata portion of the payload. The `observed_at` time indicates a snapshot in time for when the API retrieved the offset. A running connector is always updating its offsets. Use `observed_at` to get a sense for the gap between real time and the time at which the request was made. By default, offsets are observed every minute. Calling `GET` repeatedly will fetch more recently observed offsets. - Information about the connector. - In these examples, the curly braces around “{connector_name}” indicate a replaceable value. ### Update the offset To update the offset, make a `POST` request that specifies the environment, Kafka cluster, and connector name. Include a JSON payload that specifies new offset and a patch type. ```bash POST /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request Host: https://api.confluent.cloud { "type": "PATCH", "offsets": [ { "partition": { "name": "_" }, "offset": { "since": "2023-04-10T05:41:15Z" } } ] } ``` **Considerations:** - You can only make one offset change at a time for a given connector. - This is an asynchronous request. To check the status of this request, you must use the check offset status API. For more information, see **Get the status of an offset request**. - For source connectors, the connector attempts to read from the position defined by the requested offsets. **Response:** Successful calls return HTTP `202 Accepted` with a JSON payload that describes the offset. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [ { "partition": { "name": "_" }, "offset": { "since": "2023-04-10T05:41:15Z" } } ], "requested_at": "2024-03-28T17:58:45.606796307Z", "type": "PATCH" } ``` Responses include the following information: - The requested position of the offsets in the source. - The time of the request to update the offset. - Information about the connector. ### Delete the offset To delete the offset, make a `POST` request that specifies the environment, Kafka cluster, and connector name. Include a JSON payload that specifies the delete type. ```bash POST /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request Host: https://api.confluent.cloud { "type": "DELETE" } ``` **Considerations:** - Delete requests delete the offset for the provided partition and reset to the base state. A delete request is as if you created a fresh new connector. - This is an asynchronous request. To check the status of this request, you must use the check offset status API. For more information, see **Get the status of an offset request**. - Do not issue delete and patch requests at the same time. - For source connectors, the connector attempts to read from the position defined in the base state. **Response**: Successful calls return HTTP `202 Accepted` with a JSON payload that describes the result. ```bash { "id": "lcc-example123", "name": "{connector_name}", "offsets": [], "requested_at": "2024-03-28T17:59:45.606796307Z", "type": "DELETE" } ``` Responses include the following information: - Empty offsets. - The time of the request to delete the offset. - Information about Kafka cluster and connector. - The type of request. ### Get the status of an offset request To get the status of a previous offset request, make a `GET` request that specifies the environment, Kafka cluster, and connector name. ```bash GET /connect/v1/environments/{environment_id}/clusters/{kafka_cluster_id}/connectors/{connector_name}/offsets/request/status Host: https://api.confluent.cloud ``` **Considerations:** - The status endpoint always shows the status of the most recent PATCH/DELETE operation. **Response**: Successful calls return HTTP `200` with a JSON payload that describes the result. The following is an example of an applied patch. ```bash { "request": { "id": "lcc-example123", "name": "{connector_name}", "offsets": [ { "partition": { "name": "_" }, "offset": { "since": "2023-04-10T05:41:15Z" } } ], "requested_at": "2024-03-28T17:58:45.606796307Z", "type": "PATCH" }, "status": { "phase": "APPLIED", "message": "The Connect framework-managed offsets for this connector have been altered successfully. However, if this connector manages offsets externally, they will need to be manually altered in the system that the connector uses." }, "previous_offsets": [ { "partition": { "name": "_" }, "offset": { "etag": "\"736d269d94ebc0f1c3e4ceceb2fbbd28f73e80bcc0d1bd6b202596b349afadc5\"", "id": "2230425971", "since": "2024-04-08T07:05:26Z" } }, { "partition": { "name": "_" }, "offset": { "etag": "\"20e5d952ac21f31ac368394ed0b509a7d00d5727a4fea8f6eab0b949997029db\"", "id": "2217061690", "since": "2024-03-31T16:04:29Z" } }, { "partition": { "name": "_" }, "offset": { "etag": "\"727dc3d687ad3f6a0122aad0b1fbc24b003189df1729bc9180f5895345ec93b7\"", "id": "2234852695", "since": "2024-04-10T06:29:14Z" } } ], "applied_at": "2024-03-28T17:58:48.079141883Z" } ``` Responses include the following information: - The original request, including the time it was made. - The status of the request: applied, pending, or failed. - The time you issued the status request. - The previous offsets. These are the offsets that the connector last updated prior to updating the offsets. Use these to try to restore the state of your connector if a patch update causes your connector to fail or to return a connector to its previous state after rolling back. ### Entities Static entities The complete list of static entities are loaded each time. For modifications to static entities, you should delete offsets instead of updating. - [ASSIGNEES](https://docs.github.com/en/rest/issues/assignees) - [COLLABORATORS](https://docs.github.com/en/rest/collaborators/collaborators) - [RELEASES](https://docs.github.com/en/rest/releases/releases) - [STARGAZERS](https://docs.github.com/en/rest/activity/starring) Dynamic entities For dynamic entities, use `since` to retrieve a list of results. With the exception of REVIEWS, `ETag` is not useful for offset updates. - [COMMITS](https://docs.github.com/en/rest/commits/commits) - [ISSUES](https://docs.github.com/en/rest/issues/issues) - [COMMENTS](https://docs.github.com/en/rest/issues/comments) - [PULL_REQUESTS](https://docs.github.com/en/rest/pulls/pulls) and [REVIEWS](https://docs.github.com/en/rest/pulls/reviews) - [REVIEW_COMMENTS](https://docs.github.com/en/rest/pulls/comments) ### JSON payload The table below offers a description of the unique fields in the JSON payload for managing offsets of the GitHub Source connector. | Field | Definition | Required/Optional | |---------|------------------------------------------------------------------------------------------------------------------------------------------|---------------------| | `name` | The connector uses the following format for the partition: {`repository`}_{`entitiy`} | Required | | `id` | The value of the ID field for the last entity. | Optional | | `since` | Records created or updated after this time will be processed by the connector. Expected format is yyyy- MM-dd’T’HH:mm:ssX or yyyy-MM-dd. | Required | | `etag` | ETag of the previous request, used for normal processing. | Optional | ## Quick Start Use this quick start to get up and running with the Confluent Cloud GitHub Source connector. The quick start provides the basics of selecting the connector and configuring it to stream events. Prerequisites : - Authorized access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster on Amazon Web Services (AWS), Microsoft Azure (Azure), or Google Cloud. - The Confluent CLI installed and configured for the cluster. See [Install the Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html). - [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf). - Authorization and credentials to access the GitHub endpoint. - At least one Kafka topic must exist in your Confluent Cloud cluster before creating the source connector. ### Using the Confluent Cloud Console #### Step 1: Launch your Confluent Cloud cluster To create and launch a Kafka cluster in Confluent Cloud, see [Create a kafka cluster in Confluent Cloud](../get-started/index.md#cloud-create-kafka-cluster). #### Step 2: Add a connector In the left navigation menu, click **Connectors**. If you already have connectors in your cluster, click **+ Add connector**. #### Step 3: Select your connector Click the **GitHub Source** connector card. ![GitHub Source Connector Card](images/ccloud-github-source-icon.png) #### Step 4: Enter the connector details #### NOTE * Make sure you have all your [prerequisites](#cc-github-source-prereqs) completed. * An asterisk ( \* ) designates a required entry. At the **Add GitHub Source Connector** screen, complete the following: ### Kafka access 1. Select the way you want to provide **Kafka Cluster credentials**. You can choose one of the following options: - **My account**: This setting allows your connector to globally access everything that you have access to. With a user account, the connector uses an API key and secret to access the Kafka cluster. This option is not recommended for production. - **Service account**: This setting limits the access for your connector by using a [service account](service-account.md#s3-cloud-service-account). This option is recommended for production. - **Use an existing API key**: This setting allows you to specify an API key and a secret pair. You can use an existing pair or create a new one. This method is not recommended for production environments. #### NOTE Freight clusters support only service accounts for Kafka authentication. 2. Click **Continue**. ### Authentication 1. Configure the authentication properties: - **GitHub Endpoint**: The [GitHub API root endpoint](https://docs.github.com/en/rest/overview/resources-in-the-rest-api). For example, `https://api.github.com`. - **GitHub Access Token**: Used as the value of `Authorization` in HTTP requests. For information about how to set these up, see [Creating a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token). 2. Click **Continue**. ### Configuration - **Topic Name Pattern**: The pattern to use for the topic name, where the `${resourceName}` literal is replaced with each resource name. - **GitHub Repositories**: GitHub repository or comma-separated list of repositories in the form `owner/repo-name`. For example, `"apache/kafka, confluentinc/ksql"`. - **GitHub Resources**: One or more resources that the connector extracts and writes to Kafka. For details, see [GitHub Resources](#cc-github-source-resources). - **Since**: Records created or updated after this time will be processed by the connector. If left blank, the default time will be set to the time this connector is launched. Expected format is `yyyy- MM-dd'T'HH:mm:ssX` or `yyyy-MM-dd`. **Output messages** - **Select output record value format**: Select the output record value format (data going to the Kafka topic). Valid entires are AVRO, JSON, JSON_SR (JSON Schema), or PROTOBUF. [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) must be enabled to use a Schema Registry-based format (for example, Avro, JSON Schema, or Protobuf). ### **Show advanced configurations** - **Schema context**: Select a schema context to use for this connector, if using a schema-based data format. This property defaults to the **Default** context, which configures the connector to use the default schema set up for Schema Registry in your Confluent Cloud environment. A schema context allows you to use separate schemas (like schema sub-registries) tied to topics in different Kafka clusters that share the same Schema Registry environment. For example, if you select a non-default context, a **Source** connector uses only that schema context to register a schema and a **Sink** connector uses only that schema context to read from. For more information about setting up a schema context, see [What are schema contexts and when should you use them?](../sr/faqs-cc.md#faq-schema-contexts). - **Maximum Batch Size**: The maximum number of records that should be returned and written to Kafka at one time. - **Maximum In Flight Requests**: The maximum number of requests that may be in-flight at once. - **Maximum Poll Interval (ms)**: The time in milliseconds between requests to fetch changed or updated entities. - **Request Interval (ms)**: The time in milliseconds to wait before checking for updated records. - **Maximum Retries**: The maximum number of times to retry on errors before failing the task. - **Retry Backoff (ms)**: The time in milliseconds to wait following an error before a retry attempt is made. **Auto-restart policy** - **Enable Connector Auto-restart**: Enables the auto-restart behavior of the connector and its task in the event of user-actionable errors. Defaults to `true`, enabling the connector to automatically restart in case of user-actionable errors. Set this property to `false` to disable auto-restart for failed connectors. If disabled, you must manually restart the connector. **Additional Configs** - **Value Converter Decimal Format**: Specifies the `JSON` or `JSON_SR` serialization format for Connect `DECIMAL` logical type values with two allowed literals: `BASE64` to serialize `DECIMAL` logical types as base64 encoded binary data, and `NUMERIC` to serialize `DECIMAL` logical type values in `JSON` or `JSON_SR` as a number representing the decimal value. - **Key Converter Schema ID Serializer**: The class name of the schema ID serializer for keys. This is used to serialize schema IDs in the message headers. - **Value Converter Reference Subject Name Strategy**: Sets the subject reference name strategy for values. Valid entries are `DefaultReferenceSubjectNameStrategy` or `QualifiedReferenceSubjectNameStrategy`. You can use this strategy only with `PROTOBUF` format; the default strategy is `DefaultReferenceSubjectNameStrategy`. - **Value Converter Connect Meta Data**: Enables the Connect converter to add its metadata to the output schema. Applies to Avro converters. - **Value Converter Value Subject Name Strategy**: Determines how to construct the subject name under which the value schema is registered with Schema Registry. - **Key Converter Key Subject Name Strategy**: Determines how to construct the subject name for key schema registration. - **Value Converter Schema ID Serializer**: The class name of the schema ID serializer for values. This is used to serialize schema IDs in the message headers. **Transforms** - **Single Message Transformations**: To add a new SMT, see [Add transforms](single-message-transforms.md#cc-single-message-transforms-ui). For more information about unsupported SMTs, see [Unsupported transformations](single-message-transforms.md#cc-single-message-transforms-unsupported-transforms). **Processing position** - **Set offsets**: Click **Set offsets** to define a specific offset for this connector to begin procession data from. For more information on managing offsets, see [Manage offsets](offsets.md#connect-custom-offsets). See [Configuration Properties](#cc-github-source-config-properties) for all property values and definitions. - Click **Continue**. ### Sizing Based on the number of topic partitions you select, you will be provided with a recommended number of tasks. 1. To change the number of tasks, use the Range Slider to select the desired number of tasks. 2. Click **Continue**. ### Review and Launch 1. Verify the connection details by previewing the running configuration. ![Launch the connector](images/ccloud-github-source-launch-connector.png) 2. After you’ve validated that the properties are configured to your satisfaction, click **Launch**. The status for the connector should go from **Provisioning** to **Running**. ![Connector status](images/ccloud-github-source-status.png) #### Step 5: Check for records Verify that records are being produced at the Kafka topic. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ### Using the Confluent CLI Complete the following steps to set up and run the connector using the Confluent CLI. #### NOTE Make sure you have all your [prerequisites](#cc-github-source-prereqs) completed. #### Step 1: List the available connectors Enter the following command to list available connectors: ```none confluent connect plugin list ``` #### Step 2: List the connector configuration properties Enter the following command to show the connector configuration properties: ```none confluent connect plugin describe ``` The command output shows the required and optional configuration properties. #### Step 3: Create the connector configuration file Create a JSON file that contains the connector configuration properties. The following example shows the required connector properties. See [Configuration Properties](#cc-github-source-config-properties) for additional configuration property values and descriptions. ```json { "connector.class": "GithubSource", "name": "GithubSource_0", "kafka.auth.mode": "KAFKA_API_KEY", "kafka.api.key": "", "kafka.api.secret": "", "github.service.url": "https://api.github.com", "github.access.token": "*********************************", "github.repositories": "", "github.resources": "pull_requests, reviews, review_comments", "output.data.format": "AVRO", "tasks.max": "1" } ``` Note the following property definitions: * `"connector.class"`: Identifies the connector plugin name. * `"name"`: Sets a name for your new connector. * `"kafka.auth.mode"`: Identifies the connector authentication mode you want to use. There are two options: `SERVICE_ACCOUNT` or `KAFKA_API_KEY` (the default). To use an API key and secret, specify the configuration properties `kafka.api.key` and `kafka.api.secret`, as shown in the example configuration (above). To use a [service account](service-account.md#s3-cloud-service-account), specify the **Resource ID** in the property `kafka.service.account.id=`. To list the available service account resource IDs, use the following command: ```bash confluent iam service-account list ``` For example: ```bash confluent iam service-account list Id | Resource ID | Name | Description +---------+-------------+-------------------+------------------- 123456 | sa-l1r23m | sa-1 | Service account 1 789101 | sa-l4d56p | sa-2 | Service account 2 ``` * Enter the GitHub connection details. - `"github.service.url"`: The [GitHub API root endpoint](https://docs.github.com/en/rest/overview/resources-in-the-rest-api). The default used is `https://api.github.com`. - `"github.repositories"`: GitHub repository or comma-separated list of repositories in the form `owner/repo-name`. For example, `"apache/kafka, confluentinc/ksql"`. - `"github.resources"`: One or more resources that the connector extracts and writes to Kafka. See [GitHub Resources](#cc-github-source-resources) for details. * “`output.data.format"`: Enter an output data format (data going to the Kafka topic): AVRO, JSON_SR (JSON Schema), PROTOBUF, or JSON (schemaless). [Schema Registry](../get-started/schema-registry.md#cloud-sr-config) must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf). * `"tasks.max"`: Enter the number of [tasks](/platform/current/connect/concepts.html#tasks) to use with the connector. With a single repository, the connector always uses exactly one task regardless of this setting. With multiple repositories, tasks are distributed across the configured repositories. 1. **Transforms and Predicates**: See the [Single Message Transformations](single-message-transforms.md#cc-single-message-transforms) documentation for details. See [Configuration Properties](#cc-github-source-config-properties) for all property values and descriptions. #### Step 4: Load the properties file and create the connector Enter the following command to load the configuration and start the connector: ```none confluent connect cluster create --config-file .json ``` For example: ```none confluent connect cluster create --config-file github-source-config.json ``` Example output: ```none Created connector GithubSource_0 lcc-do6vzd ``` #### Step 5: Check the connector status Enter the following command to check the connector status: ```none confluent connect cluster list ``` Example output: ```none ID | Name | Status | Type | Trace +------------+--------------------------+---------+--------+-------+ lcc-do6vzd | GithubSource_0 | RUNNING | source | | ``` #### Step 6: Check for records. Verify that records are being produced at the Kafka topic. For more information and examples to use with the Confluent Cloud API for Connect, see the [Confluent Cloud API for Connect Usage Examples](connect-api-section.md#ccloud-connect-api) section. ## Configuration Properties Use the following configuration properties with the fully managed connector. For self-managed connector property definitions and other details, see the connector docs in [Self-managed connectors for Confluent Platform](/platform/current/connect/kafka_connectors.html). ### How should we connect to your data? `name` : Sets a name for your connector.
* Type: string * Valid Values: A string at most 64 characters long * Importance: high ### Kafka Cluster credentials `kafka.auth.mode` : Kafka Authentication mode. It can be one of KAFKA_API_KEY or SERVICE_ACCOUNT. It defaults to KAFKA_API_KEY mode, whenever possible.
* Type: string * Valid Values: SERVICE_ACCOUNT, KAFKA_API_KEY * Importance: high `kafka.api.key` : Kafka API Key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high `kafka.service.account.id` : The Service Account that will be used to generate the API keys to communicate with Kafka Cluster.
* Type: string * Importance: high `kafka.api.secret` : Secret associated with Kafka API key. Required when kafka.auth.mode==KAFKA_API_KEY.
* Type: password * Importance: high ### Which topic name pattern do you want to send data to? `topic.name.pattern` : The pattern to use for the topic name, where the `${resourceName}` literal will be replaced with each resource name.
* Type: string * Default: ${resourceName} * Importance: high ### Schema Config `schema.context.name` : Add a schema context name. A schema context represents an independent scope in Schema Registry. It is a separate sub-schema tied to topics in different Kafka clusters that share the same Schema Registry instance. If not used, the connector uses the default schema configured for Schema Registry in your Confluent Cloud environment.
* Type: string * Default: default * Importance: medium ### How should we connect to GitHub? `github.service.url` : GitHub API Root Endpoint. Ex: [https://api.github.com](https://api.github.com)
* Type: string * Importance: medium `github.access.token` : The supplied token will be used as the value of ‘Authorization’ header in HTTP requests.
* Type: password * Importance: high `github.repositories` : The GitHub repositories to read from in the form of owner/repo-name. Ex: apache/kafka, apache/superset
* Type: list * Importance: high `github.resources` : The resources that are to be extracted and written to Kafka.
* Type: list * Importance: high `github.since` : Records created or updated after this time will be processed by the connector. If left blank, the default time will be set to the time this connector is launched. Expected format is yyyy-MM-dd’T’HH:mm:ssX or yyyy-MM-dd
* Type: string * Importance: high ### Connection details `max.batch.size` : The maximum number of records that should be returned and written to Kafka at one time.
* Type: int * Default: 100 * Importance: low `max.in.flight.requests` : The maximum number of requests that may be in-flight at once.
* Type: int * Default: 10 * Importance: low `max.poll.interval.ms` : The time in milliseconds between requests to fetch changed or updated entities.
* Type: long * Default: 3000 (3 seconds) * Importance: low `request.interval.ms` : The time in milliseconds to wait before checking for updated records.
* Type: long * Default: 15000 (15 seconds) * Importance: low `max.retries` : The maximum number of times to retry on errors before failing the task.
* Type: int * Default: 10 * Importance: low `retry.backoff.ms` : The time in milliseconds to wait following an error before a retry attempt is made.
* Type: long * Default: 3000 (3 seconds) * Importance: low ### Output messages `output.data.format` : Sets the output Kafka record value format. Valid entries are AVRO, JSON_SR, PROTOBUF, or JSON. Note that you need to have Confluent Cloud Schema Registry configured if using a schema-based message format like AVRO, JSON_SR, and PROTOBUF
* Type: string * Importance: high ### Number of tasks for this connector `tasks.max` : Maximum number of tasks for the connector.
* Type: int * Valid Values: [1,…] * Importance: high ### Auto-restart policy `auto.restart.on.user.error` : Enable connector to automatically restart on user-actionable errors.
* Type: boolean * Default: true * Importance: medium ### Additional Configs `header.converter` : The converter class for the headers. This is used to serialize and deserialize the headers of the messages.
* Type: string * Importance: low `producer.override.compression.type` : The compression type for all data generated by the producer. Valid values are none, gzip, snappy, lz4, and zstd.
* Type: string * Importance: low `value.converter.allow.optional.map.keys` : Allow optional string map key when converting from Connect Schema to Avro Schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.auto.register.schemas` : Specify if the Serializer should attempt to register the Schema.
* Type: boolean * Importance: low `value.converter.connect.meta.data` : Allow the Connect converter to add its metadata to the output schema. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.avro.schema.support` : Enable enhanced schema support to preserve package information and Enums. Applicable for Avro Converters.
* Type: boolean * Importance: low `value.converter.enhanced.protobuf.schema.support` : Enable enhanced schema support to preserve package information. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.flatten.unions` : Whether to flatten unions (oneofs). Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.index.for.unions` : Whether to generate an index suffix for unions. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.generate.struct.for.nulls` : Whether to generate a struct variable for null values. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.int.for.enums` : Whether to represent enums as integers. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.latest.compatibility.strict` : Verify latest subject version is backward compatible when use.latest.version is true.
* Type: boolean * Importance: low `value.converter.object.additional.properties` : Whether to allow additional properties for object schemas. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.optional.for.nullables` : Whether nullable fields should be specified with an optional label. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.optional.for.proto2` : Whether proto2 optionals are supported. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.use.latest.version` : Use latest version of schema in subject for serialization when auto.register.schemas is false.
* Type: boolean * Importance: low `value.converter.use.optional.for.nonrequired` : Whether to set non-required properties to be optional. Applicable for JSON_SR Converters.
* Type: boolean * Importance: low `value.converter.wrapper.for.nullables` : Whether nullable fields should use primitive wrapper messages. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `value.converter.wrapper.for.raw.primitives` : Whether a wrapper message should be interpreted as a raw primitive at root level. Applicable for Protobuf Converters.
* Type: boolean * Importance: low `key.converter.key.schema.id.serializer` : The class name of the schema ID serializer for keys. This is used to serialize schema IDs in the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.PrefixSchemaIdSerializer * Importance: low `key.converter.key.subject.name.strategy` : How to construct the subject name for key schema registration.
* Type: string * Default: TopicNameStrategy * Importance: low `value.converter.decimal.format` : Specify the JSON/JSON_SR serialization format for Connect DECIMAL logical type values with two allowed literals:
BASE64 to serialize DECIMAL logical types as base64 encoded binary data and
NUMERIC to serialize Connect DECIMAL logical type values in JSON/JSON_SR as a number representing the decimal value.
* Type: string * Default: BASE64 * Importance: low `value.converter.flatten.singleton.unions` : Whether to flatten singleton unions. Applicable for Avro and JSON_SR Converters.
* Type: boolean * Default: false * Importance: low `value.converter.reference.subject.name.strategy` : Set the subject reference name strategy for value. Valid entries are DefaultReferenceSubjectNameStrategy or QualifiedReferenceSubjectNameStrategy. Note that the subject reference name strategy can be selected only for PROTOBUF format with the default strategy being DefaultReferenceSubjectNameStrategy.
* Type: string * Default: DefaultReferenceSubjectNameStrategy * Importance: low `value.converter.value.schema.id.serializer` : The class name of the schema ID serializer for values. This is used to serialize schema IDs in the message headers.
* Type: string * Default: io.confluent.kafka.serializers.schema.id.PrefixSchemaIdSerializer * Importance: low `value.converter.value.subject.name.strategy` : Determines how to construct the subject name under which the value schema is registered with Schema Registry.
* Type: string * Default: TopicNameStrategy * Importance: low ## Frequently asked questions Find answers to frequently asked questions about the GitHub Source connector for Confluent Cloud. ### Availability and deployment model #### Can I run GitHub Source as a self-managed connector on my own Connect cluster? Yes. The GitHub Source connector is available as both a fully managed connector on Confluent Cloud and as a self-managed connector for Confluent Platform. For self-managed installation, see the [GitHub Source Connector for Confluent Platform](https://docs.confluent.io/kafka-connectors/github/current/overview.html). ### Authentication and access #### What type of GitHub access token should I use with this connector? You should use a GitHub personal access token (PAT) or a GitHub App token with appropriate permissions: * **Personal access token (PAT):** Generate a classic PAT or fine-grained PAT from your GitHub settings. For fine-grained tokens, grant `Read-only` access to the following repository permissions: `Contents`, `Issues`, `Metadata`, and `Pull requests`. * **GitHub App token:** If using a GitHub App, ensure the app has the necessary read permissions for the repositories you want to monitor. Store the token securely and rotate it periodically following your organization’s security policies. #### Why am I getting authentication errors when connecting to GitHub? Authentication errors typically occur due to one of the following reasons: * **Expired or invalid token:** Verify that your GitHub access token is still valid and has not expired. * **Insufficient permissions:** Ensure your token has the required permissions to access the repositories and resources specified in `github.repositories` and `github.resources`. * **Rate limit exceeded:** GitHub can return authentication errors when rate limits are exceeded. Check your rate limit status using the GitHub API. **Resolution:** 1. Verify token validity in your GitHub account settings. 2. Regenerate the token if necessary and update the connector configuration. 3. Confirm the token has access to the specified repositories. ### API rate limiting #### How does the connector handle GitHub API rate limits? The GitHub Source connector is designed to be rate-limit aware. When the GitHub API rate limit is exceeded: * The connector automatically stops fetching records from GitHub. * The connector waits until the rate limit resets based on the `X-RateLimit-Reset` header from GitHub. * After the rate limit resets, the connector resumes fetching records automatically. GitHub enforces the following rate limits: * **Authenticated requests:** 5,000 requests per hour per token. * **Unauthenticated requests:** 60 requests per hour per IP address (not recommended for production). To minimize rate limit issues, ensure you are using an authenticated token and consider the number of resources and repositories you are monitoring. #### Why does my connector keep pausing due to rate limit errors? If your connector frequently pauses due to rate limit errors, you might be polling too many repositories or resources, or polling too frequently for your GitHub API quota. Consider the following: * **Reduce polling frequency:** Adjust the connector’s scheduling or offset lookback configuration to reduce how often the connector polls for updates. * **Limit resources:** Reduce the number of resources in `github.resources` to only those you need. * **Limit repositories:** Reduce the number of repositories in `github.repositories`. * **Use multiple connectors:** If you need to monitor many repositories, consider using multiple connectors with separate GitHub tokens to distribute the API load. Monitor your GitHub API rate limit status using the GitHub API: `https://api.github.com/rate_limit`. ### Offset management and data collection #### How do I reset offsets for the GitHub Source connector? The GitHub Source connector supports the Offset Management API. To reset offsets: 1. Use the `DELETE` operation to remove all offsets, which resets the connector to its initial state. 2. Use the `PATCH` operation to update specific partition offsets using the `since` field. For detailed information and examples, see [Manage custom offsets](#cc-github-source-custom-offsets). **Important:** Deleting offsets causes the connector to re-fetch data from the beginning, which can result in duplicate records in your Kafka topic. #### Why is my connector re-fetching old data or missing new data? This issue typically occurs due to offset misalignment: * **Re-fetching old data:** The offset might have been reset or corrupted. Verify the current offset using the `GET /offsets` API and confirm the `since` timestamp is correct. * **Missing new data:** The offset `since` timestamp might be set too far in the future, causing the connector to skip recent updates. **Resolution:** 1. Use the Offset Management API to check the current offset. 2. Update the offset using the `PATCH` operation if the `since` timestamp is incorrect. 3. For dynamic entities like commits, issues, and pull requests, ensure the `since` field is properly formatted as ISO 8601 (`yyyy-MM-dd'T'HH:mm:ssX`). ### Resource configuration #### How does `tasks.max` affect the GitHub Source connector? The behavior of `tasks.max` depends on how many repositories you configure: * **Single repository**: The connector always uses exactly one task regardless of the `tasks.max` setting. All resources for a given repository run in the same task, sharing the same HTTP client and rate limit state. * **Multiple repositories**: The connector distributes the work across tasks up to the `tasks.max` limit, which can improve throughput. If you need to scale ingestion across many repositories, increase `tasks.max` and configure multiple repositories in `github.repositories`. #### Which GitHub resources should I monitor for real-time repository activity? For real-time repository activity monitoring, configure the following resources: * **commits:** Monitor all commits to the default branch. * **pull_requests:** Track pull request creation, updates, and state changes. * **issues:** Monitor issue creation and updates. * **comments:** Capture issue and pull request comments. * **reviews and review_comments:** Monitor pull request reviews and review comments. You can specify multiple resources in the `github.resources` configuration as a comma-separated list, for example: `pull_requests, issues, comments, commits`. See [GitHub Resources](#cc-github-source-resources) for the complete list of supported resources. ### Connector errors and troubleshooting #### Why does my connector fail with `Could not find repository` or `404 Not Found`? This error indicates the connector cannot access the specified repository. Common causes include: * **Incorrect repository format:** Verify the repository is specified correctly in `owner/repo-name` format (for example, `confluentinc/kafka-connect-github`). * **Insufficient token permissions:** Ensure your GitHub token has access to the repository. For private repositories, the token must have the `repo` scope. * **Repository does not exist:** Confirm the repository name is spelled correctly and exists. **Resolution:** 1. Verify the repository format in `github.repositories`. 2. Test access to the repository using the GitHub API with your token. 3. For private repositories, ensure the token has `repo` scope. #### Why is my connector failing with “Connection refused” or timeout errors? Connection errors typically occur due to network issues: * **Firewall or network restrictions:** If your Confluent Cloud cluster uses PrivateLink or VPC peering, ensure outbound connectivity to `api.github.com` is allowed. * **Incorrect GitHub URL:** Verify `github.service.url` is set to `https://api.github.com` (or your GitHub Enterprise URL). * **DNS resolution issues:** Ensure the Confluent Cloud environment can resolve `api.github.com`. **Resolution:** 1. Check network connectivity and firewall rules. 2. Verify the GitHub API URL configuration. 3. For GitHub Enterprise, ensure the URL is correct and accessible from Confluent Cloud. ## Next Steps For an example that shows fully managed Confluent Cloud connectors in action with Confluent Cloud for Apache Flink, see the [Cloud ETL Demo](/platform/current/tutorials/examples/cloud-etl/docs/index.html). This example also shows how to use Confluent CLI to manage your resources in Confluent Cloud. [![image](images/topology.png)](https://docs.confluent.io/platform/current/tutorials/examples/cloud-etl/docs/index.html)