.. _confluent-hub-component-archive: |c-hub| Component Archive Specification ======================================== Components published on |c-hub| can only be downloaded if an *archive file* is provided by the owner. That archive file must have specific content and organization, including a *manifest file* with information about the component. Confluent has provided a :ref:`Maven plugin ` that you can use in the Maven builds of your component's source code project to build the manifest and archive file. When you release your software, verify your component satisfies the :ref:`requirements ` and :ref:`submit your component `. The rest of this page describes the :ref:`archive file `, the :ref:`manifest file `, and the :ref:`Maven plugin ` in more detail. .. _terminology: ----------- Terminology ----------- Component A packaged implementation of one or more |ak-tm| or |cp| APIs. Many components include a set of libraries, configuration files, documentation, and other resources that you install into your local system and then configure and use. Some components, however, cannot be downloaded and must be accessed from the component's owner as detailed on |c-hub|. Component archive A ZIP file that contains the packaged component for easy download and installation. Not all components have an archive. Component owner The individual or organization providing a component. Owners develop and test a component, and optionally package it into a component archive for publishing on |c-hub|. Owner names namespace all components from that owner into a unique identifier. Owner names are used as part of the directory name when the component is installed. The name of a component owner must contain two to 255 characters, and must contain only lowercase letters, numbers, and underscore (``_``) characters. An example is ``confluentinc``. Component name Each component has a name that, combined with the owner name, provides a unique identifier for that component relative to all other components in |c-hub|. Names should be meaningful to end users, and will be used as part of the directory name when the component is installed. The name of a component must contain only lowercase letters, numbers, minus sign (``-``), and underscore (``_``) characters. An example is ``kafka-connect-jdbc``. Component version Components have versions that can be used to identify a specific release of that component, and should correspond to the same versioning scheme used in version control and releases. They may contain letters, numbers, minus sign (``-``), period (``.``), and underscore (``_``) characters. An example is ``5.0.0``. .. _confluent-hub-component-archive-spec: ------------------------ Component Archive Format ------------------------ A component archive is a ZIP file that contains the files that are required by the component, organized into a specific structure. This structure makes it easy to extract the contents of the archive into particular locations within your local installation. .. _confluent-hub-component-archive-spec-name: Component Archive Name `````````````````````` The component archive name must follow this convention: .. codewithvars:: bash ${componentOwner}-${componentName}-${componentVersion}.zip where: - ``${componentOwner}`` is the name of the owner or developer of the component that acts as a namespace and is unique across |c-hub|. - ``${componentName}`` is the name of the component that is unique within the scope of the owner name. - ``${componentVersion}`` is the version of the component as chosen by the owner or developer. For example, in the file named ``confluentinc-kafka-connect-elasticsearch-3.3.2.zip``, the owner is ``confluentinc``, the name is ``kafka-connect-elasticsearch``, and the version is ``3.3.2``. These are somewhat analogous to the Maven ``groupId``, ``artifactId``, and ``version``. .. _confluent-hub-component-archive-spec-structure: Component Archive Structure ``````````````````````````` Here are the contents of the ZIP file: .. codewithvars:: bash confluentinc-kafka-connect-elasticsearch-3.3.2/ manifest.json doc/ LICENSE licenses notices README.md licenses.html version.txt etc/ quickstart-elasticsearch.properties lib/ commons-codec-1.9.jar commons-lang3-3.4.jar commons-logging-1.2.jar gson-2.4.jar guava-18.0.jar httpasyncclient-4.1.1.jar httpclient-4.5.1.jar httpcore-4.4.4.jar httpcore-nio-4.4.4.jar jest-2.0.0.jar jest-common-2.0.0.jar kafka-connect-elasticsearch-3.3.2.jar slf4j-api-1.7.25.jar assets/ elastic_logo.png confluent_logo.jpg apache_logo.gif The ``doc/`` directory contains any sort of human-readable documents concerning the component, such as READMEs and licenses. The ``etc/`` directory contains any sample or default properties files that may be helpful to include with the component. The ``lib/`` directory contains all of the files necessary for actually running the component. For example, the Elasticsearch connector JAR (``kafka-connect-elasticsearch-3.3.2.jar``) and all of its runtime dependencies, excluding those already provided by the |kconnect-long| framework. Notice that there is no JAR for the |kconnect-long| API included (e.g., ``connect-api-1.0.1.jar``); it’s assumed that this will already be provided by the |kconnect-long| framework when the connector is run. The ``assets/`` directory includes any asset files that should be hosted by |c-hub|. This is currently limited to logo files. Subdirectories are not included. The ``manifest.json`` file contains all of the metadata to be associated with the component and made available on |c-hub|. .. _confluent-hub-component-manifest: ------------------- Component Manifests ------------------- A sample manifest ````````````````` Here are the contents of the ``manifest.json`` file: .. sourcecode:: json { "component_types": [ "sink" ], "description": "This is a connector for getting data out of Apache Kafka into Elasticsearch.\nIt is built off of the Kafka Connect framework, and therefore automatically supports pluggable encoding converters, single message transforms, and other useful features.\nElasticsearch 5.x is required (6.x is not supported due to a known issue).", "docker_image": { "tag": "3.3.2", "name": "kafka-connect-elasticsearch", "namespace": "confluentinc", "registry": "hub.docker.io" }, "documentation_url": "https://docs.confluent.io/3.3.1/connect/connect-elasticsearch/docs/index.html", "features": { "confluent_control_center_integration": true, "delivery_guarantee": ["at_least_once"], "kafka_connect_api": true, "single_message_transforms": true, "supported_encodings": ["any"] }, "license": [ { "name": "Apache License, Version 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0", "logo": "assets/apache_logo.gif" } ], "logo": "assets/elastic_logo.png", "name": "kafka-connect-elasticsearch", "owner": { "logo": "assets/confluent_logo.jpg", "name": "Confluent, Inc.", "type": "organization", "url": "https://confluent.io/", "username": "confluentinc" }, "requirements": [ "Elasticsearch 5.x" ], "support": { "logo": "assets/confluent_logo.jpg", "provider_name": "Confluent, Inc.", "summary": "We provide full support for this connector, alongside community members who contribute to it as an open-source project.", "url": "https://confluent.io/" }, "tags": [ "analytics", "elasticsearch", "elastic" ], "title": "Kafka Connect Elasticsearch", "version": "3.3.2" } .. _confluent-hub-component-manifest-fields: Manifest Field Descriptions ``````````````````````````` Here’s what each of the fields in the provided ``manifest.json`` file should contain: ``component_types`` *Required.* |c-hub| currently contains four different categories of component: sink connector, source connector, transformer, and converter. This field is a list of one or more of the following values: ``“sink”``, ``“source”``, ``“transform”``, and ``“converter”``. A component can qualify for several different types. For example, one might contain a sink connector, a source connector, and multiple transformations. ``description`` *Required.* The description is one of the most essential elements of the manifest, and this is among the most prominent content shown on |c-hub|. It provides a summary of the component’s functionality and features, as well as any information that can’t be specified in any other part of the manifest. We recommend the first short paragraph provide a good overview, and that subsequent paragraphs list features, capabilities, limitations, and external requirements in detailed, human-readable sentences. This is where you convince users to use your component rather than other components that serve the same or a similar purpose. ``docker_image`` Information on a Docker image available for the component. ``docker_image.namespace`` The namespace that the Docker image can be found under. ``docker_image.name`` The name of the Docker image. ``docker_image.tag`` The tag of the Docker image that corresponds to this version of the component. ``docker_image.registry`` The registries that contain the Docker image. ``documentation_url`` A link to documentation for the component. Can be anything from a custom web page to a GitHub Wiki. ``features`` These are specific features that are tracked by Confluent and used to help compare components. Any other features that make a component noteworthy should be detailed in the description field. ``features.supported_encodings`` The serialized encodings the component supports. If it is a |kconnect-long| connector, the availability of pluggable converters means that basically any encoding is supported. If, however, the component is not a |kconnect-long| connector and is limited in which encodings it supports, they should be specified here. ``features.delivery_guarantee`` What delivery guarantees the component supports. Can be empty if no guarantees are made, or a list containing one or both of “at_least_once” and “exactly_once”. ``features.kafka_connect_api`` Whether the component is built on top of the |kconnect-long| API. ``features.confluent_control_center_integration`` Whether the component supports integration with |c3|. ``features.single_message_transforms`` Whether the component supports single message transforms. If it is a |kconnect-long| connector, this is automatically true. ``logo`` A logo to display on |c-hub| for the component. If the component is a |kconnect-long| connector, this should probably be a logo for the source/sink of the connector, since the relation to |ak-tm| will be implied already. ``license`` A list of licenses associated with the component. The example only contains one, but several can be specified if need be. It is strongly recommended that at least one license be provided. ``license.name`` The human-readable name of the license. ``license.url`` A link to the actual text of the license. ``license.logo`` A logo for the license. ``name`` *Required.* This is the ``${componentName}`` portion of the component used in its archive filename. It is also the name used to refer to the component via the back-end REST API; because of this, restrictions are placed on what characters it can consist of, and the result is often slightly less than what may be desired for human readability. ``owner`` Information regarding the owner of the component. ``owner.username`` *Required.* This is the ``${componentOwner}`` portion of the component’s archive filename. Similar to the name field, it is restricted to work as part of a resource for a REST API. ``owner.type`` *Required.* The type of owner for the component; either ``"organization"`` or ``"user"``. ``owner.name`` *Required.* As the ``title`` field relates to the ``name`` field, the ``owner.name`` field relates to the ``owner.username`` field. This is a friendlier, human-readable way to present who owns the component, and will be used on |c-hub|. If left unspecified, will default to the ``owner.username`` field. ``owner.url`` A website or web page to associate with the owner. Useful for attracting traffic from people who want to know more about who provided the component, possibly to a company’s website (as in this example) or a public GitHub profile (for an individual user who would like recognition for their work). ``owner.logo`` An image to associate with the component owner. ``requirements`` A list of requirements given in a concise, bullet-listable format. ``support`` Information on who, if anyone, provides support for the component, and what kind of support is provided. ``support.provider_name`` A friendly, human-readable name for who supports the component. Can be ``Community`` if the component is fully open-sourced and maintained by voluntary contributors. ``support.summary`` What exactly users should expect in terms of support for this component. Should be as detailed as possible with regards to how to file bug reports, ask questions, etc. ``support.url`` Where the users should go for support. If a public Jira is used for filing bug reports, for example, this should link there. Another option would be a link to a GitHub issues page. ``support.logo`` An image to associate with the support provider. ``tags`` A list of relevant search tags for the component. ``title`` *Required.* The human-readable name for the component, which will be displayed for it on Confluent Hub. This can essentially be thought of as a less restrictive version of the name field and, if not specified, will default to the name field. ``version`` *Required.* Pretty self-explanatory--the version of the component. This is used as the ``${componentVersion}`` portion of the component’s archive filename. Version is an important attribute since multiple versions of the same component can be uploaded, and |c-hub| displays the most recent version of each component by default. .. _confluent-hub-component-manifest-assets: Logo Files `````````` Logos are specified as references to files contained in the `assets/` directory of the component archive (e.g., ``logo``, ``support.logo``, ``owner.logo``, and ``license.logo``). |c-hub| hosts these images and displays them on its site on the relevant pages for the component. At this time, all logo files must be provided in the ``assets/`` directory and not one of its subdirectories. External URLs or files placed in other locations will not be recognized. Any valid image format is allowed for a logo file, as long as the corresponding extension is contained in its filename. Logos must be at least 400 pixels wide and 200 pixels tall, and can be no more than 45 MB in size. .. _confluent-hub-component-manifest-verification-level: Verification Level `````````````````` The ``confluent_verified`` nested document is added by |c-hub| based upon the owner’s participation in Confluent’s Partner Program, in which Confluent verifies the functionality of the components. Although it will appear in manifests hosted by |c-hub|, you should not specify a value for it in a manifest you submit; Confluent will add it automatically. .. _confluent-hub-maven-plugin: ---------------------- Maven Packaging Plugin ---------------------- Confluent provides a Maven plugin for automatically packaging your component in the archive format, including asset files, an auto-generated manifest, config files, READMEs, licenses, and JAR files. If you use Maven to build your project and would like to integrate our packaging plugin into your build process, get the Maven plugin from `Maven Central `_, and see its `documentation <./kafka-connect-maven-plugin/kafka-connect-mojo.html>`_. Check out `Confluent's Elasticsearch connector `_ for an `example of how it's used `_.