>
Confluent
  • Product
    • Confluent Platform
    • KSQL
    • Confluent Hub
    • Subscription
    • Professional Services
    • Training
    • Customers
  • Cloud
    • Confluent Cloud
    • Support
    • Sign Up
    • Log In
    • Cloud FAQ
  • Developers
    • What is Kafka?
    • Resources
    • Events
    • Online Talks
    • Meetups
    • Kafka Summit
    • Kafka Tutorials
    • Confluent Developer
  • About Us
    • Company
    • Partners
    • News
    • Events
    • Careers
    • Contact
  • Blog
  • Docs
  • Download
Get Confluent | Sign up for Confluent Cloud or download Confluent Platform
  • What is the Confluent Platform?
    • What is included in the Confluent Platform?
      • Overview
      • Confluent Open Source
        • Confluent Connectors
          • Kafka Connect JDBC Connector
          • Kafka Connect HDFS Connector
          • Kafka Connect Elasticsearch Connector
          • Kafka Connect S3 Connector
        • Confluent Clients
          • C/C++ Client Library
          • Python Client Library
          • Go Client Library
          • .Net Client Library
        • Confluent Schema Registry
        • Confluent REST Proxy
      • Confluent Enterprise
        • Automatic Data Balancing
        • Multi-Datacenter Replication
        • Confluent Control Center
        • JMS Client
      • Confluent Proactive Support
    • Migrating an existing Kafka deployment
      • Migrating developer tools
      • Migrating to Control Center
  • Confluent Platform 3.2.0 Release Notes
    • Highlights
      • Enterprise Features
        • JMS Client
        • Confluent Control Center
        • Automatic Data Balancing
      • Open Source Features
        • Kafka Streams
          • Backwards Compatible to Older Kafka Clusters
          • Session Windows
          • Global KTables
          • ZooKeeper Dependency Removed
        • Kafka Connect
          • Single Message Transform
          • S3 Connector
        • .Net Client
        • REST Proxy
        • Apache Kafka 0.10.2
    • Previous releases
      • Confluent Platform 3.1.2 Release Notes
        • Highlights
          • Apache Kafka 0.10.1.1
            • Fixes
      • Confluent Platform 3.1.1 Release Notes
        • Highlights
          • Enterprise Features
            • Automatic Data Balancing
            • Multi-Datacenter Replication
            • Confluent Control Center
          • Open Source Features
            • Kafka Streams: Interactive Queries
            • Kafka Streams: Application Reset Tool
            • Kafka Streams: Improved memory management
            • Kafka Connect
            • Go Client
          • Apache Kafka 0.10.1
            • New Feature
            • Improvement
            • Bug
            • Task
            • Wish
            • Test
            • Sub-task
      • Confluent Platform 3.0.1 Release Notes
        • Highlights
          • Confluent Control Center
          • Kafka Streams
            • New Feature: Application Reset Tool
          • Apache Kafka 0.10.0.1
            • New Feature
            • Improvement
            • Bug
            • Test
            • Sub-task
      • Confluent Platform 3.0.0 Release Notes
        • Highlights
          • Kafka Streams
          • Confluent Control Center
          • Apache Kafka 0.10.0.0
          • Deprecating Camus
          • Other Notable Changes
      • Confluent Platform 2.0.1 Release Notes
        • New Java consumer
        • Compatibility
        • Security
        • Performance/memory usage
        • Topic deletion
      • Confluent Platform 2.0.0 Release Notes
        • Security
        • Kafka Connect
        • User Defined Quotas
        • New Consumer
        • New Client Library - librdkafka
        • Proactive Support
        • Compatibility Notes
    • How to Download
    • Questions?
  • Installation
    • Confluent Platform Installation
      • Requirements
      • ZIP and TAR archives
      • DEB packages via apt
      • RPM packages via yum
      • Available Packages
      • Migrating from Confluent Open Source to Confluent Enterprise
    • Clients
      • Maven repository for jars
      • C/C++
      • Python
      • Go
      • .NET
  • Upgrade
    • Preparation
    • Step-by-step Guide
      • Upgrade All Kafka Brokers
      • Upgrade Schema Registry
      • Upgrade Kafka REST Proxy
      • Upgrade Kafka Streams applications
      • Upgrade Kafka Connect
        • Upgrade Kafka Connect Standalone Mode
        • Upgrade Kafka Connect Distributed Mode
      • Upgrade Camus
      • Upgrade Other Client Applications
      • Confluent Control Center
  • Confluent Platform Quickstart
  • Application Development
    • Native Clients with Serializers
      • Java
      • C/C++
    • REST Proxy
  • Operations
    • Kafka Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
          • Filesystem
          • General considerations
        • JVM
        • Important Configuration Options
          • Replication configs
        • File descriptors and mmap
        • Zookeeper
          • Stable version
          • Operationalizing ZooKeeper
      • Post Deployment
        • Tweaking configs dynamically
        • Logging
          • Controller
          • State Change Log
          • Request logging
        • Admin operations
          • Adding topics
          • Modifying topics
          • Deleting topics
          • Graceful shutdown
          • Scaling the Cluster
          • Increasing replication factor
          • Limiting Bandwidth Usage during Data Migration
          • Balancing Replicas Across Racks
          • Enforcing Client Quotas
        • Performance Tips
          • Picking the number of partitions for a topic
          • Lagging replicas
          • Increasing consumer throughput
        • Software Updates
        • Backup and Restoration
      • Auto Data Balancing
        • Requirements
        • Quickstart
          • Start a Kafka cluster
          • Create topics and produce data
          • Execute the rebalancer
          • Check status and finish
          • Leader balance
          • Decommissioning brokers
          • Limiting Bandwidth Usage during Data Migration
        • Licensing
        • Configuration Options
          • Configuration Options for the rebalancer tool
      • Monitoring Kafka
        • Server Metrics
          • Broker Metrics
          • ZooKeeper Metrics
        • Producer Metrics
          • Global Request Metrics
          • Global Connection Metrics
          • Per-Broker Metrics
          • Per-Topic Metrics
        • New Consumer Metrics
          • Fetch Metrics
          • Topic-level Fetch Metrics
          • Consumer Group Metrics
          • Global Request Metrics
          • Global Connection Metrics
          • Per-Broker Metrics
        • Old Consumer Metrics
      • Confluent Metrics Reporter
        • Installation
        • Configuration Options
    • Schema Registry Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Important Configuration Options
        • Don’t Touch These Settings!
          • Storage settings
        • Kafka & ZooKeeper
        • Multi-DC Setup
          • Overview
          • Recommended Deployment
          • Important Settings
          • Setup
          • Run Book
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Kafka REST Proxy Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Deployment
        • Important Configuration Options
        • Don’t Touch These Settings!
        • Post Deployment
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Camus Operations
      • Important Configuration Options
  • Docker
    • Introduction
      • Choosing the Right Images
      • Requirements
      • Important Notes/Caveats
      • License
    • Quickstart
      • Installing & Running Docker
      • Running Confluent Platform on Docker
        • Zookeeper
        • Kafka
        • Schema Registry
        • REST Proxy
        • Confluent Control Center
          • Stream Monitoring
          • Alerts
        • Kafka Connect
          • Getting Started
          • Monitoring in Control Center
        • Cleanup
      • Getting Started with Docker Compose
    • Security Overview
      • Docker Security
    • Configuration
      • ZooKeeper
        • Required Settings
      • Confluent Kafka (cp-kafka)
        • Required Settings
      • Confluent Enterprise Kafka (cp-enterprise-kafka)
        • Required Settings
      • Schema Registry
        • Required Settings
      • Kafka REST Proxy
        • Required Settings
      • Kafka Connect
        • Required Settings
        • Optional Settings
      • Confluent Control Center
        • Docker Options
        • Required Settings
        • Optional Settings
      • Confluent Enterprise Replicator
        • Required Settings
        • Optional Settings
    • Operations
      • Monitoring
        • Using JMX
          • Security on JMX
          • Kafka & Zookeeper
            • Settings
            • Launching Kafka and Zookeeper with JMX Enabled
      • Configuring logging
        • log4j Log Levels
        • Component Names
      • Mounting External Volumes
        • Data Volumes for Kafka & Zookeeper
        • Security: Data Volumes for Configuring Secrets
        • Configuring Connect with External jars
    • More Tutorials
      • Clustered Deployment
        • Installing & Running Docker
        • Docker Client: Setting Up a Three Node Kafka Cluster
        • Docker Compose: Setting Up a Three Node Kafka Cluster
      • Clustered Deployment Using SSL
        • Installing & Running Docker
        • Docker Client: Setting Up a Three Node Kafka Cluster
        • Docker Compose: Setting Up a Three Node CP Cluster with SSL
      • Clustered Deployment Using SASL and SSL
        • Installing & Running Docker
        • Docker Client: Setting Up a Three Node Kafka Cluster
        • Docker Compose: Setting Up a Three Node CP Cluster with SASL
      • Kafka Connect Tutorial
        • Installing & Running Docker
        • Starting Up Confluent Platform & Kafka Connect
      • Automatic Data Balancing
        • Installing & Running Docker
      • Replicator Tutorial
        • Installing & Running Docker
    • Developer Guide
      • Image Design Overview
        • The Bootup Process
        • Configuration
        • Preflight Checks
        • Launching the Process
        • Development Guidelines
      • Setup
        • Building the Images
        • Running Tests
        • Make Targets
      • Extending the Docker Images
        • Prerequisites
        • Adding Connectors to the Kafka Connect Image
        • Examples
      • Utility Scripts
        • Docker Utility Belt (dub)
        • Confluent Platform Utility Belt (cub)
        • Client Properties
      • References
    • How to Contribute
      • General Guidelines
      • GitHub Workflow
  • Confluent Control Center
    • Introduction
    • Quickstart
      • Base Installation
      • Zookeeper Configuration
      • Broker Configuration
      • Connect Configuration
      • Control Center Configuration
      • Stream Monitoring Setup
    • Concept Guide
      • Motivation
      • Time windows and metrics
      • Latency
    • Requirements
    • Installation
      • Installation on RedHat, Centos, or Fedora
      • Installation on Debian Linux
      • Other systems
        • Installing Confluent Metrics Reporter with Kafka
        • Installing Kafka Connect
        • Installing Confluent Metrics Clients with Kafka Connect
        • Starting Control Center
      • License Keys
      • Removing Control Center
        • TAR and ZIP Archives
        • Debian packages
        • Redhat packages
    • Kafka Client Guide
      • Interceptor Installation
      • Application Configuration
        • Adding the interceptor to your Kafka Producer
        • Adding the interceptor to your Kafka Consumer
        • Adding interceptors to your Kafka Streams application
      • Interceptor Configuration
        • Configuration options
        • Security
        • Example Configuration
        • Logging
      • Multi Cluster Configuration
        • Dedicated Metric Data Cluster
    • Configuration
      • Required Settings
      • Logging
      • Optional Settings
      • UI Authentication Settings
      • Email Settings
      • Kafka Encryption, Authentication, Authorization Settings
      • HTTPS Settings
      • Internal Kafka Streams Settings
      • Internal Command Settings
    • Web User Interface
      • System Health
        • Navigation
        • UI Commonalities
        • Produced and Fetched Charts
        • Broker Aggregate Metrics
        • Broker Metrics Table
        • Topic Aggregate Metrics
        • Topic Metrics Table
      • Stream Monitoring
        • Chart Types
        • Page Layout
        • Navigation
        • Time Range and Cluster Selection
        • Summary statistics
        • Missing Metrics Data
        • Example Scenarios
          • Adding a new Consumer Group
            • Latency
            • Expected consumption
      • Kafka Connect
        • Creating new Connectors
        • Editing Sources and Sinks
      • Alerts
        • Concepts
        • User Interface
        • Trigger Management
          • New/Edit Trigger Form
          • Topic
          • Consumer Groups
          • Brokers
        • Actions Management
          • New/Edit Action Form
        • Alert History
      • Clusters
        • Setting a Cluster Name
      • Topic Management
        • Detail view for a topic
    • Security
      • Configuring SSL
        • Kafka Brokers
        • Control Center
        • Connect
      • Configuring SASL
        • ZooKeeper
        • Kafka Broker
        • Control Center Configuration
        • Connect Configuration
      • UI Authentication
      • UI HTTPS
      • Authorization with Kafka ACLS
    • Operations
      • Hardware
        • Memory
        • CPUs
        • Disks
        • Network
      • OS
      • JVM
      • Logging
      • User/Cluster Metadata
      • Kafka
    • Upgrading
      • Upgrading from version 3.1.x
      • Upgrading from version 3.0.x
    • Changelog
      • Version 3.2.0
        • Confluent Control Center
      • Version 3.1.0
        • Confluent Control Center
      • Version 3.0.1
        • Confluent Control Center
  • Kafka Security
    • Encryption and Authentication using SSL
      • Overview
      • Generate SSL key and certificate for each Kafka broker
      • Creating your own CA
      • Signing the certificate
      • Configuring Kafka Brokers
      • Configuring Kafka Clients
      • Enabling SSL Logging
    • Authentication using SASL
      • SASL configuration for Kafka brokers
      • SASL configuration for Kafka Clients
      • Authentication using SASL/Kerberos
        • Prerequisites
          • Kerberos
          • Kerberos Principals
          • All hosts must be reachable using hostnames
        • Configuring Kafka Brokers
        • Configuring Kafka Clients
      • Authentication using SASL/PLAIN
        • Configuring Kafka Brokers
        • Configuring Kafka Clients
        • Use of SASL/PLAIN in production
      • Enabling multiple SASL mechanisms in a broker
      • Modifying SASL mechanisms in a Running Cluster
      • Enabling Logging for SASL
    • Authorization and ACLs
      • Overview
        • Common cases:
        • Further configuration:
      • Command Line Interface
        • Adding ACLs
        • Removing ACLs
        • List ACLs
        • Adding or Removing a Principal as Producer or Consumer
        • Enabling Authorizer Logging for Debugging
    • Adding Security to a Running Cluster
    • ZooKeeper Authentication
      • New clusters
      • Migrating clusters
      • Migrating the ZooKeeper ensemble
    • Kafka Security & the Confluent Platform
      • Other Observations
  • Kafka Streams
    • Introduction
      • The Kafka Streams API in a Nutshell
      • Use Case Examples
      • A Closer Look
    • Requirements
      • Apache Kafka
      • Confluent
    • Quickstart
      • Goal of this quickstart
      • What we want to do
      • Start the Kafka cluster
      • Prepare the topics and the input data
      • Process the input data with Kafka Streams
      • Inspect the output data
      • Stop the Kafka cluster
      • Where to go from here
    • Concepts
      • Kafka 101
      • Stream
      • Stream Processing Application
      • Processor Topology
      • Stream Processor
      • Stateful Stream Processing
      • Duality of Streams and Tables
      • KStream
      • KTable
      • GlobalKTable
      • Time
      • Aggregations
      • Joins
      • Windowing
    • Architecture
      • Processor Topology
      • Parallelism Model
        • Stream Partitions and Tasks
        • Threading Model
        • Example
      • State
      • Memory management
        • Record caches
      • Fault Tolerance
      • Processing Guarantees
      • Flow Control with Timestamps
      • Backpressure
    • Developer Guide
      • Code examples
        • Application examples for Kafka Streams in Apache Kafka
        • Application examples for Kafka Streams provided by Confluent
          • Simple examples
          • Security examples
          • Interactive Queries examples
          • End-to-end demo applications
      • Configuring a Kafka Streams application
        • Overview
        • Required configuration parameters
        • Optional configuration parameters
        • Non-Streams configuration parameters
      • Writing a Kafka Streams application
        • Overview
        • Libraries and maven artifacts
        • Using Kafka Streams within your application code
        • Kafka Streams DSL
          • Overview
          • Creating source streams from Kafka
          • Transform a stream
            • Stateless transformations
            • Stateful transformations
              • Overview
              • Aggregating
              • Joining
                • Overview
                • Joins require co-partitioning of the input data
                • KStream-KStream Join
                • KTable-KTable Join
                • KStream-KTable Join
                • KStream-GlobalKTable Join
              • Windowing
                • Overview
                • Tumbling time windows
                • Hopping time windows
                • Sliding time windows
                • Session Windows
            • Applying processors and transformers (Processor API integration)
          • Writing streams back to Kafka
        • Processor API
          • Overview
          • Defining a Stream Processor
          • State Stores
            • Overview
            • Defining and creating a State Store
            • Fault-tolerant State Stores
            • Enable / Disable Fault Tolerance of State Stores (Store Changelogs)
            • Implementing custom State Stores
          • Connecting Processors and State Stores
      • Interactive Queries
        • Overview
        • Demo applications
        • Your application and interactive queries
        • Querying local state stores (for an application instance)
          • Querying local key-value stores
          • Querying local window stores
          • Querying local custom state stores
        • Querying remote state stores (for the entire application)
          • Adding an RPC layer to your application
          • Exposing the RPC endpoints of your application
          • Discovering and accessing application instances and their respective local state stores
      • Memory management
        • Record caches in the DSL
        • State store caches in the Processor API
        • Other memory usage
      • Running a Kafka Streams application
        • Starting a Kafka Streams application
        • Elastic scaling of your application
          • Adding capacity to your application (“expand”)
          • Removing capacity from your application (“shrink”)
          • State restoration during workload rebalance
          • How many application instances to run?
        • Application Reset Tool
          • Overview
          • Scope
          • Usage
            • Step 1: Run the application reset tool
            • Step 2: Reset the local environments of your application instances
          • Example
      • Managing topics of a Kafka Streams application
        • Overview
        • User topics
        • Internal topics
      • Data types and serialization
        • Overview
        • Configuring default serializers/deserializers (serdes)
        • Overriding default serializers/deserializers (serdes)
        • Available serializers/deserializers (serdes)
        • Implementing custom serializers/deserializers (serdes)
      • Security
        • Overview
        • Notable ACL settings
        • Security example
    • Operations
      • Capacity planning and sizing
        • Background and context
        • Stateless vs. stateful
          • Stateless applications
          • Stateful applications
        • Examples
        • Troubleshooting
      • Monitoring your application
        • Metrics
          • Accessing Metrics
            • Accessing Metrics via JMX and Reporters
            • Accessing Metrics Programmatically
          • Configuring Metrics Granularity
          • Built-in Metrics
            • Thread Metrics
            • Task Metrics
            • Processor Node Metrics
            • State Store Metrics
            • Record Cache Metrics
          • Adding Your Own Metrics
        • Run-time Status Information
          • Status of KafkaStreams instances
        • Integration with Confluent Control Center
    • Upgrade Guide
      • Upgrading from CP 3.1.x (Kafka 0.10.1.x-cp2) to CP 3.2.0 (Kafka 0.10.2.0-cp1)
        • Compatibility
        • Upgrading your Kafka Streams applications to CP 3.2.0
        • API changes
          • Handling Negative Timestamps and Timestamp Extractor Interface
          • Metrics
          • Scala
        • Full upgrade workflow
    • FAQ
      • General
        • Is Kafka Streams a project separate from Apache Kafka?
        • Is Kafka Streams a proprietary library of Confluent?
        • Do Kafka Streams applications run inside the Kafka brokers?
        • What are the system dependencies of Kafka Streams?
        • How do I migrate my Kafka Streams applications from CP 3.0.x and CP 3.1.x applications to CP 3.2.0
        • Which versions of Kafka clusters are supported by Kafka Streams?
        • What programming languages are supported?
      • Scalability
        • Maximum parallelism of my application? Maximum number of app instances I can run?
      • Processing
        • Accessing record metadata such as topic, partition, and offset information?
        • How can I replace RocksDB with a different store?
      • Failure and exception handling
        • Handling corrupted records and deserialization errors (“poison pill messages”)?
          • Option 1: Skip corrupted records with flatMap
          • Option 2: Quarantine corrupted records (dead letter queue) with branch
          • Option 3: Skip corrupted records with a custom serde
        • Sending corrupt records to a quarantine topic or dead letter queue?
      • Interactive Queries
        • Handling InvalidStateStoreException: “the state store may have migrated to another instance”?
      • Security
        • Application fails when running against a secured Kafka cluster?
      • Troubleshooting and debugging
        • Easier to interpret Java stacktraces?
        • Visualizing topologies?
        • Inspecting streams and tables?
        • Invalid Timestamp Exception
        • Why do I get an IllegalStateException when accessing record metadata?
        • Why is punctuate() not called?
        • Scala: compile error “no type parameter”, “Java-defined trait is invariant in type T”
        • How can I convert a KStream to a KTable without an aggregation step?
          • Option 1: Write KStream to Kafka, read back as KTable
          • Option 2: Perform a dummy aggregation
    • Javadocs
  • Kafka Connect
    • Introduction
    • Requirements
    • Quickstart
      • Goal of this quickstart
      • What we will do
      • Start ZooKeeper, Kafka, Schema Registry
      • Read File Data with Connect
      • Write File Data with Connect
    • Concepts
      • Connectors
      • Tasks
        • Task Rebalancing
      • Workers
        • Standalone Workers
        • Distributed Workers
      • Converters
    • Installing and Configuring Kafka Connect
      • Getting Started
      • Planning for Installation
        • Prerequisites
        • Standalone vs. Distributed
        • Deployment Considerations
      • Installing Connector Plugins
      • Running Workers
        • Standalone Mode
        • Distributed Mode
      • Configuring Workers
        • Common Worker Configs
        • Standalone Worker Configuration
        • Distributed Worker Configuration
        • Configuring Converters
        • Overriding Producer & Consumer Settings
      • Upgrading Kafka Connect Workers
    • Managing Connectors
      • Using Bundled Connectors
      • Configuring Connectors
        • Standalone Example
        • Distributed Example
      • Managing Running Connectors
        • Using the REST Interface
        • Connector and Task Status
        • Common REST Examples
      • Using Community Connectors
      • Upgrading a Connector Plugin
    • Bundled Connectors
      • Confluent JDBC Connector
        • JDBC Source Connector
          • Quickstart
          • Features
            • Incremental Query Modes
          • Configuration
            • JDBC Drivers
            • Examples
          • Schema Evolution
        • JDBC Source Configuration Options
          • Database
          • Connector
          • Mode
        • JDBC Sink Connector
          • Quickstart
          • Features
            • Data mapping
            • Key handling
            • Idempotent writes
            • Auto-creation and Auto-evoluton
        • JDBC Sink Configuration Options
          • Connection
          • Writes
          • Data Mapping
          • DDL Support
          • Retries
        • Changelog
          • Version 3.2.0
            • JDBC Source Connector
            • JDBC Sink Connector
          • Version 3.1.1
          • Version 3.1.0
            • JDBC Source Connector
            • JDBC Sink Connector
          • Version 3.0.1
            • JDBC Source Connector
          • Version 3.0.0
            • JDBC Source Connector
      • Confluent HDFS Connector
        • HDFS Connector
          • Quickstart
          • Features
          • Configuration
            • Example
            • Format and Partitioner
            • Hive Integration
            • Secure HDFS and Hive Metastore
          • Schema Evolution
        • Configuration Options
          • HDFS
          • Hive
          • Security
          • Schema
          • Connector
          • Internal
        • Changelog
          • Version 3.2.0
          • Version 3.1.1
          • Version 3.1.0
          • Version 3.0.1
            • HDFS Connector
          • Version 3.0.0
            • HDFS Connector
      • Confluent S3 Connector
        • S3 Connector
          • Features
          • Exactly-once delivery on top of eventual consistency
          • Quickstart
          • Configuration
            • Example
        • Configuration Options
          • Connector
          • S3
          • Storage
          • Partitioner
        • Changelog
          • Version 3.2.0
            • S3 Connector
      • Confluent Elasticsearch Connector
        • Elasticsearch Connector
          • Quickstart
          • Features
          • Delivery Semantics
          • Mapping Management
          • Schema Evolution
          • Reindexing
          • Security
        • Configuration Options
          • Connector
          • Data Conversion
        • Changelog
          • Version 3.2.0
      • Confluent Replicator
        • Confluent Replicator
          • Features
          • Requirements
          • Quickstart
            • Start the target cluster
            • Start the source cluster
            • Configure and run Replicator
          • Topic Renaming
          • Periodic Metadata Updates
          • Security
        • Changelog
          • Version 3.2.0
        • Configuration Options
          • Confluent Platform
          • Source Topics
          • Source Data Conversion
          • Source Zookeeper
          • Source Kafka
          • Source Kafka: Security
          • Source Kafka: Consumer
          • Destination Topics
          • Destination Zookeeper
          • Destination Data Conversion
      • Kafka FileStream Connectors
        • Quickstart
        • FileSource Connector
        • FileSink Connector
    • Security
      • Configuring Workers with Security
      • Configuring Connectors with Security
      • ACL Considerations
        • Worker ACL Requirements
        • Connector ACL Requirements
    • Architecture & Internals
      • Motivation
      • Architecture
        • Internal Connect Offsets
    • Connector Developer Guide
      • Core Concepts and APIs
        • Connectors and Tasks
        • Partitions and Records
        • Dynamic Connectors
      • Developing a Simple Connector
        • Connector Example
        • Task Example - Source Task
        • Sink Tasks
        • Resuming from Previous Offsets
      • Dynamic Input/Output Partitions
      • Configuration Validation
      • Working with Schemas
      • Schema Evolution
      • Testing
    • FAQ
      • How do I change the output data format of a SinkConnector?
      • Why does a connector configuration update trigger a task rebalance?
      • Why should I use distributed mode instead of standalone?
      • Do I need to write custom code to use Kafka Connect?
      • Is the Schema Registry a required service to run Kafka Connect?
      • How can I use plain JSON data with Connect?
      • Does source connector X support output format Y?
      • Why is CPU usage high for my Connect worker when no connectors have been deployed?
      • Can connect sink connectors read data written by other clients, e.g. a custom client?
      • After testing a connector in standalone mode, restarting it doesn’t write the data again?
      • Can I use a newer version of Connect with older brokers?
    • Reference
      • Javadocs
      • REST Interface
        • Content Types
        • Statuses & Errors
        • Connectors
        • Tasks
        • Connector Plugins
      • All Worker Configs
        • Common Worker Configs
        • Standalone Worker Configuration
        • Distributed Worker Configuration
  • Kafka Clients
    • Kafka Consumers
      • Concepts
      • Configuration
      • Initialization
      • Basic Usage
        • Java Client
        • C/C++ Client (librdkafka)
        • Python, Go and .NET Clients
      • Detailed Examples
        • Basic Poll Loop
        • Shutdown with Wakeup
        • Synchronous Commits
        • Asynchronous Commits
      • Administration
        • List Groups
        • Describe Group
    • Kafka Producers
      • Concepts
      • Configuration
      • Examples
        • Initial Setup
        • Asynchronous Writes
        • Synchronous Writes
    • Confluent’s JMS Client for Apache Kafka
      • Overview
      • Examples
        • Producing messages
        • Consuming messages
      • Requirements
      • Installation
      • JMS 1.1 Compatibility
      • Features
      • Unsupported Features
        • Session
        • ConnectionFactory
        • Connection
      • Developing JMS Clients
        • Dependancies
        • Configure and Create a ConnectionFactory
        • Setting Required Properties
        • Enabling Confluent Specific Features (Optional)
        • Enabling TLS Encryption (Optional)
        • Create a ConnectionFactory
        • Creating a Destination
        • Creating a Connection
        • Creating Sessions
        • Producing / Consuming Messages
      • Threading
    • API Docs
  • Schema Registry
    • Schema Registry
      • Quickstart
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.2.0
      • Version 3.1.1
      • Version 3.1.0
        • Schema Registry
        • Serializers, Formatters, and Converters
      • Version 3.0.1
        • Schema Registry
        • Serializers, Formatters, and Converters
      • Version 3.0.0
        • Schema Registry
        • Serializers, Formatters, and Converters
      • Version 2.0.1
        • Schema Registry
      • Version 2.0.0
        • Schema Registry
        • Serializers, Formatters, and Converters
    • API Reference
      • Overview
        • Compatibility
        • Content Types
        • Errors
      • Schemas
      • Subjects
      • Compatibility
      • Config
    • Configuration Options
    • Design Overview
      • Batch ID Allocation
      • Kafka Backend
      • Single Master Architecture
    • Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Important Configuration Options
        • Don’t Touch These Settings!
          • Storage settings
        • Kafka & ZooKeeper
        • Multi-DC Setup
          • Overview
          • Recommended Deployment
          • Important Settings
          • Setup
          • Run Book
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Security Overview
      • Kafka Store
      • ZooKeeper
    • Serializer and Formatter
      • Serializer
      • Formatter
      • Wire Format
        • Compatibility Guarantees
    • Maven Plugin
      • schema-registry:download
      • schema-registry:test-compatibility
      • schema-registry:register
  • Kafka REST Proxy
    • Kafka REST Proxy
      • Quickstart
        • Produce and Consume JSON Messages
        • Produce and Consume Avro Messages
        • Produce and Consume Binary Messages
        • Inspect Topic Metadata
      • Features
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.2.0
      • Version 3.1.2
      • Version 3.1.1
      • Version 3.1.0
      • Version 3.0.0
      • Version 2.0.1
      • Version 2.0.0
    • API Reference
      • Overview
        • Content Types
        • Errors
      • API v2
        • Topics
        • Partitions
        • Consumers
        • Brokers
      • API v1
        • Topics
        • Partitions
        • Consumers
        • Brokers
    • Configuration Options
      • Security Configuration Options
        • Configuration Options for HTTPS
        • Configuration Options for SSL Encryption between REST Proxy and Apache Kafka Brokers
        • Configuration Options for SASL Authentication between REST Proxy and Apache Kafka Brokers
    • Operations
      • Production Deployment
        • Hardware
          • Memory
          • CPUs
          • Disks
          • Network
        • JVM
        • Deployment
        • Important Configuration Options
        • Don’t Touch These Settings!
        • Post Deployment
      • Monitoring
        • Global Metrics
        • Per-Endpoint Metrics
        • Endpoints
    • Security Overview
  • Multi Data-Center Deployment
    • Replicator Quick Start Guide
      • Start the destination cluster
      • Start the origin cluster
      • Create a topic
      • Configure and run Replicator
    • Installing and Configuring Replicator
      • Install and Configure Kafka Connect Cluster for Replicator
        • Configuring origin and destination brokers
        • Where to Install Connect Workers
        • Running Replicator on Existing Connect Cluster
        • Configuring Logging for Connect Cluster
        • License Key
      • Configure and run a Confluent Replicator on the Connect Cluster
    • Tuning and Monitoring Replicator
      • Sizing Replicator Cluster
      • Getting More Throughput From Replicator Tasks
        • Improving CPU Utilization of a Connect Task
        • Improving Network Utilization of a Connect Task
      • Monitoring Replicator
        • Monitoring Replicator Lag
        • Monitoring Producer and Consumer Metrics
          • Important Producer Metrics
          • Important Consumer Metrics
    • Configuration Options
      • Confluent Platform
      • Source Topics
      • Source Data Conversion
      • Source Zookeeper
      • Source Kafka
      • Source Kafka: Security
      • Source Kafka: Consumer
      • Destination Topics
      • Destination Zookeeper
      • Destination Data Conversion
    • Apache Kafka’s Mirror Maker
      • Comparing Mirror Maker to Confluent Replicator
        • Greater control for reliable replication
        • Deploying, managing, and monitoring
        • Replication Patterns
        • License
      • Example of Mirror Maker Use
  • Data Serialization and Evolution
    • Introduction
    • Avro
      • Defining an Avro Schema
      • Schema Evolution
      • Backward Compatibility
      • Forward Compatibility
      • Full Compatibility
      • Confluent Schema Registry
  • Camus
    • Camus
      • Key Features
      • Quickstart
      • Installation
      • Deployment
      • Development
      • Requirements
      • Contribute
      • License
    • Changelog
      • Version 3.2.0
      • Version 3.1.2
      • Version 3.1.1
      • Version 3.1.0
      • Version 3.0.0
      • Version 2.0.1
      • Version 2.0.0
    • Configuration Options
      • Schema Registry Configuration
      • Camus Job Configuration
      • Kafka Configuration
      • Example Configuration
    • Design
    • Operations
      • Important Configuration Options
  • Confluent Proactive Support
    • Confluent Proactive Support
      • What is Proactive Support?
      • How it works
      • Which metadata is collected?
        • Version Collector (default)
        • Confluent Support Metrics (add-on package)
      • Which metadata and data is not being collected?
      • Installing Support Metrics
        • DEB Packages via apt
        • RPM Packages via yum
      • Enabling or disabling the Metrics feature
      • Recommended Proactive Support configuration settings for licensed Confluent customers
      • Proactive Support configuration settings
      • Network ports used by Proactive Support
      • Sharing Proactive Support Metadata with Confluent manually
    • Privacy Policy
  • Previous Versions
  • Docs
  • Kafka Connect »
  • Bundled Connectors »
  • Confluent Replicator
  • View page source

Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Confluent Replicator¶

Contents:

  • Confluent Replicator
    • Features
    • Requirements
    • Quickstart
      • Start the target cluster
      • Start the source cluster
      • Configure and run Replicator
    • Topic Renaming
    • Periodic Metadata Updates
    • Security
  • Changelog
    • Version 3.2.0
  • Configuration Options
    • Confluent Platform
    • Source Topics
    • Source Data Conversion
    • Source Zookeeper
    • Source Kafka
    • Source Kafka: Security
    • Source Kafka: Consumer
    • Destination Topics
    • Destination Zookeeper
    • Destination Data Conversion

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Please report any inaccuracies on this page or suggest an edit.

Next Previous

Last updated on Aug 07, 2020.

Built with Sphinx using a theme provided by Read the Docs.
    Expand Content v.--