>
Confluent
  • Product
    • Confluent Platform
    • KSQL
    • Confluent Hub
    • Subscription
    • Professional Services
    • Training
    • Customers
  • Cloud
    • Confluent Cloud
    • Support
    • Sign Up
    • Log In
    • Cloud FAQ
  • Developers
    • What is Kafka?
    • Resources
    • Events
    • Online Talks
    • Meetups
    • Kafka Summit
    • Kafka Tutorials
    • Confluent Developer
  • About Us
    • Company
    • Partners
    • News
    • Events
    • Careers
    • Contact
  • Blog
  • Docs
  • Download
Get Confluent | Sign up for Confluent Cloud or download Confluent Platform
  • What is the Confluent Platform?
    • What is included in the Confluent Platform?
      • Overview
      • Apache Kafka
      • Focus: Kafka Streams
      • Focus: Kafka Connect
      • Confluent Control Center
      • Kafka Connectors
        • Kafka Connect JDBC Connector
        • Kafka Connect HDFS Connector
      • Kafka Clients
        • Java client library
        • C/C++ client library
        • Python client library
      • Confluent Schema Registry
      • Confluent Kafka REST Proxy
      • Confluent Proactive Support
    • Migrating an existing Kafka deployment
      • Migrating developer tools
      • Migrating to Control Center
  • Confluent Platform 3.0.0 Release Notes
    • Highlights
      • Kafka Streams
      • Confluent Control Center
      • Apache Kafka 0.10.0.0
      • Deprecating Camus
      • Other Notable Changes
    • Previous CP releases
      • What’s New in CP 2.0.1?
        • New Java consumer
        • Compatibility
        • Security
        • Performance/memory usage
        • Topic deletion
      • What’s New in CP 2.0.0?
        • Security
        • Kafka Connect
        • User Defined Quotas
        • New Consumer
        • New Client Library - librdkafka
        • Compatibility Notes
    • How to Download
    • Questions?
  • Data Serialization and Evolution
    • Avro
      • Backward Compatibility
      • Forward Compatibility
      • Full Compatibility
  • Installation
    • Requirements
    • ZIP and TAR archives
    • DEB packages via apt
    • RPM packages via yum
    • DEB/RPM packages with installer script
    • Available Packages
    • Maven repository for jars
  • Upgrade
    • Preparation
    • Step-by-step Guide
      • Upgrade All Kafka Brokers
      • Upgrade Client Applications
        • Kafka Connect Distributed Mode
  • Confluent Platform Quickstart
  • Application Development
    • Native Clients with Serializers
      • Java
      • C/C++
    • REST Proxy
  • Operations
  • Confluent Control Center
    • Introduction to Confluent Control Center
    • Quickstart
    • Concept Guide
      • Motivation
      • Time windows and metrics
      • Latency
    • Requirements for Control Center
    • Installing Confluent Control Center Server
      • Installation on RedHat, Centos, or Fedora
      • Installation on Debian Linux
      • Other systems
        • Installing Kafka Connect
        • Installing Confluent Metrics Clients with Kafka Connect
        • Starting Control Center
      • Removing Control Center
        • TAR and ZIP Archives
        • Debian packages
        • Redhat packages
        • License Keys
    • Client Guide
      • Installing the Confluent Monitoring Interceptor
      • Configuring Applicaitons to use the Confluent Monitoring Interceptor
        • Adding the interceptor to your Kafka Producer
        • Adding the interceptor to your Kafka Consumer
      • Confluent Monitoring Interceptor Configuration
        • Logging
        • Interceptor configuration options
    • Configuring Control Center
      • Required Settings
      • Optional Settings
      • SSL Settings
      • Rarely Used Settings
    • Using Control Center
      • Overview of the app
      • Data Stream Monitoring
        • Understanding the line charts
        • Changing the time range or time period
        • Getting details about consumer groups
        • Missing Metrics Data
      • Kafka Connect Configuration
        • Creating new Connectors
        • Editing Sources and Sinks
    • Security Overview
  • Kafka Security
    • Encryption and Authentication using SSL
      • Overview
      • Generate SSL key and certificate for each Kafka broker
      • Creating your own CA
      • Signing the certificate
      • Configuring Kafka Brokers
      • Configuring Kafka Clients
      • Enabling SSL Logging
    • Authentication using SASL
      • SASL configuration for Kafka brokers
      • SASL configuration for Kafka Clients
      • Authentication using SASL/Kerberos
        • Prerequisites
          • Kerberos
          • Kerberos Principals
          • All hosts must be reachable using hostnames
        • Configuring Kafka Brokers
        • Configuring Kafka Clients
      • Authentication using SASL/PLAIN
        • Configuring Kafka Brokers
        • Configuring Kafka Clients
        • Use of SASL/PLAIN in production
      • Enabling multiple SASL mechanisms in a broker
      • Modifying SASL mechanisms in a Running Cluster
      • Enabling Logging for SASL
    • Authorization and ACLs
      • Overview
        • Common cases:
        • Further configuration:
      • Command Line Interface
        • Adding ACLs
        • Removing ACLs
        • List ACLs
        • Adding or Removing a Principal as Producer or Consumer
        • Enabling Authorizer Logging for Debugging
    • Adding Security to a Running Cluster
    • ZooKeeper Authentication
      • New clusters
      • Migrating clusters
      • Migrating the ZooKeeper ensemble
    • Kafka Security & the Confluent Platform
      • Other Observations
  • Kafka Streams
    • Introduction
      • Kafka Streams – The Power without the Weight
      • A closer look
    • Requirements
    • Quickstart
      • Goal of this quickstart
      • What we want to do
      • Start the Kafka cluster
      • Prepare the input data
      • Process the input data with Kafka Streams
      • Inspect the output data
      • Stop the Kafka cluster
      • Where to go from here
    • Concepts
      • Kafka 101
      • Stream
      • Stream Processing Application
      • Processor Topology
      • Stream Processor
      • Time
      • Stateful Stream Processing
      • Duality of Streams and Tables
      • KStream (record stream)
      • KTable (changelog stream)
      • Windowing
      • Joins
      • Aggregations
    • Architecture
      • Processor Topology
      • Parallelism Model
        • Stream Partitions and Tasks
        • Threading Model
        • Example
      • State
      • Fault Tolerance
      • Processing Guarantees
      • Flow Control with Timestamps
      • Backpressure
    • Developer Guide
      • Code examples
        • Streams application examples in Apache Kafka
        • Streams application examples provided by Confluent
          • Simple examples
          • End-to-end demo applications
      • Configuring a Kafka Streams application
        • Overview
        • Required configuration parameters
        • Optional configuration parameters
        • Non-Streams configuration parameters
      • Writing a Kafka Streams application
        • Overview
        • Libraries and maven artifacts
        • Processor API
          • Overview
          • Defining a Stream Processor
          • Defining a State Store
          • Connecting Processors and Stores
        • Kafka Streams DSL
          • Overview
          • Creating source streams from Kafka
          • Transform a stream
            • Stateless transformations
            • Stateful transformations
            • Windowing a Stream
            • Joining Streams
            • Applying a custom processor
          • Writing streams back to Kafka
      • Running a Kafka Streams application
      • Data types and serialization
        • Overview
        • Configuring default serializers/deserializers (serdes)
        • Overriding default serializers/deserializers (serdes)
        • Available serializers/deserializers (serdes)
        • Implementing custom serializers/deserializers (serdes)
      • Migration guide (Confluent Kafka Streams Tech Preview to 0.10.0.0-cp1)
        • Project Setup
        • Application configuration
        • API changes
          • Implication of configuration changes
          • Renamed methods
          • Table grouping and aggregation
          • Windowing
    • FAQ
      • General
        • Is Kafka Streams a project separate from Apache Kafka?
        • Is Kafka Streams a proprietary library of Confluent?
        • What are the system dependencies of Kafka Streams?
        • How do I migrate my Kafka Streams Tech Preview applications to CP 3.0.0 / Kafka 0.10.0.0?
        • Can I use Kafka Streams with Kafka clusters running 0.9, 0.8, or 0.7?
      • Troubleshooting and debugging
        • Get more meaningful Java stacktraces?
    • Javadocs
  • Kafka Connect
    • Kafka Connect
      • Quickstart
      • Requirements
    • System Design
      • Motivation
      • Architecture
    • User Guide
      • Getting Started
      • Connectors, Tasks, and Workers
        • Offsets
        • Workers
          • Standalone Mode
          • Distributed Mode
      • Configuring Connectors
      • Configuring Workers
        • Common Worker Configs
        • Standalone Worker Configuration
        • Distributed Worker Configuration
        • Overriding Producer & Consumer Settings
      • Connect Administration
      • REST Interface
        • Content Types
        • Statuses & Errors
        • Connectors
        • Connector Plugins
      • Installing Connector Plugins
    • Connectors
      • Confluent JDBC Connector
        • JDBC Connector
          • Quickstart
          • Features
            • Incremental Query Modes
          • Configuration
            • JDBC Drivers
            • Examples
          • Schema Evolution
        • Configuration Options
        • Changelog
          • Version 3.0.0
            • JDBC Connector
      • Confluent HDFS Connector
        • HDFS Connector
          • Quickstart
          • Features
          • Configuration
            • Example
            • Format and Partitioner
            • Hive Integration
            • Secure HDFS and Hive Metastore
          • Schema Evolution
        • Configuration Options
        • Changelog
          • Version 3.0.0
            • HDFS Connector
    • Developer Guide
      • Core Concepts and APIs
        • Connectors and Tasks
        • Partitions and Records
        • Dynamic Connectors
      • Developing a Simple Connector
        • Connector Example
        • Task Example - Source Task
        • Sink Tasks
        • Resuming from Previous Offsets
      • Dynamic Input/Output Partitions
      • Configuration Validation
      • Working with Schemas
      • Schema Evolution
      • Testing
    • Security
    • Javadocs
  • Kafka Clients
    • Kafka Consumers
      • Concepts
      • Configuration
      • Initialization
      • Basic Usage
        • Java Client
        • C/C++ Client (librdkafka)
        • Python Client
      • Detailed Examples
        • Basic Poll Loop
        • Shutdown with Wakeup
        • Synchronous Commits
        • Asynchronous Commits
      • Administration
        • List Groups
        • Describe Group
    • Kafka Producers
      • Concepts
      • Configuration
      • Examples
        • Initial Setup
        • Asynchronous Writes
        • Synchronous Writes
    • API Docs
  • Schema Registry
    • Schema Registry
      • Quickstart
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.0.0
        • Schema Registry
        • Serializers, Formatters, and Converters
      • Version 2.0.1
        • Schema Registry
      • Version 2.0.0
        • Schema Registry
        • Serializers, Formatters, and Converters
    • API Reference
      • Overview
        • Compatibility
        • Content Types
        • Errors
      • Schemas
      • Subjects
      • Compatibility
      • Config
    • Configuration Options
    • Design Overview
      • Batch ID Allocation
      • Kafka Backend
      • Single Master Architecture
    • Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Important Configuration Options
        • Don’t Touch These Settings!
          • Storage settings
        • Kafka & ZooKeeper
        • Multi-DC Setup
          • Overview
          • Recommended Deployment
          • Important Settings
          • Setup
          • Run Book
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Security Overview
      • Kafka Store
    • Serializer and Formatter
      • Serializer
      • Formatter
  • Kafka REST Proxy
    • Kafka REST Proxy
      • Quickstart
        • Produce and Consume Avro Messages
        • Produce and Consume JSON Messages
        • Produce and Consume Binary Messages
        • Inspect Topic Metadata
      • Features
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.0.0
      • Version 2.0.1
      • Version 2.0.0
    • API Reference
      • Overview
        • Content Types
        • Errors
      • Topics
      • Partitions
      • Consumers
      • Brokers
    • Configuration Options
    • Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Deployment
        • Important Configuration Options
        • Don’t Touch These Settings!
        • Post Deployment
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Security Overview
  • Camus
    • Camus
      • Key Features
      • Quickstart
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.0.0
      • Version 2.0.1
      • Version 2.0.0
    • Configuration Options
      • Schema Registry Configuration
      • Camus Job Configuration
      • Kafka Configuration
      • Example Configuration
    • Design
    • Operations
      • Important Configuration Options
  • Previous Versions
  • Docs
  • Kafka Connect
  • View page source

Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Kafka Connect¶

Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

This section describes Kafka Connect, a component of open source Apache Kafka. Kafka Connect is a framework for scalably and reliably connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Contents:

  • Kafka Connect
    • Quickstart
    • Requirements
  • System Design
    • Motivation
    • Architecture
  • User Guide
    • Getting Started
    • Connectors, Tasks, and Workers
      • Offsets
      • Workers
    • Configuring Connectors
    • Configuring Workers
      • Common Worker Configs
      • Standalone Worker Configuration
      • Distributed Worker Configuration
      • Overriding Producer & Consumer Settings
    • Connect Administration
    • REST Interface
      • Content Types
      • Statuses & Errors
      • Connectors
      • Connector Plugins
    • Installing Connector Plugins
  • Connectors
    • Confluent JDBC Connector
      • JDBC Connector
      • Configuration Options
      • Changelog
    • Confluent HDFS Connector
      • HDFS Connector
      • Configuration Options
      • Changelog
  • Developer Guide
    • Core Concepts and APIs
      • Connectors and Tasks
      • Partitions and Records
      • Dynamic Connectors
    • Developing a Simple Connector
      • Connector Example
      • Task Example - Source Task
      • Sink Tasks
      • Resuming from Previous Offsets
    • Dynamic Input/Output Partitions
    • Configuration Validation
    • Working with Schemas
    • Schema Evolution
    • Testing
  • Security
  • Javadocs

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Please report any inaccuracies on this page or suggest an edit.

Next Previous

Last updated on Feb 25, 2021.

Built with Sphinx using a theme provided by Read the Docs.
    Expand Content v.--