Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Kafka Operations

In this document, you will learn how to manage a Apache Kafka® cluster.

Here, we will cover three main topics:

• Deploying your cluster to production, including best practices and important configuration that should (or should not!) be changed
• Performing post-deployment logistics, such as a rolling restart or backup of your cluster
• Monitoring your cluster’s vital statistics, understanding which behaviors are normal and which should be cause for alarm, and interpreting various stats provided by Kafka

• Running Kafka in Production
  • Hardware
    • Memory
    • CPUs
    • Disks
    • Network
    • Filesystem
  • General Considerations
  • JVM
  • Production Configuration Options
    • General configs
    • Replication configs
  • File Descriptors and mmap
  • Multi-node Configuration
• Dynamic Configs
  • Changing Broker Configs Dynamically
  • Updating Password Configs Dynamically
  • Updating Password Configs in ZooKeeper Before Starting Brokers
  • Updating SSL Keystore of an Existing Listener
  • Updating SSL Truststore of an Existing Listener
  • Changing Topic Configs Dynamically
    • cleanup.policy
    • flush.messages
    • flush.ms
    • max.message.bytes
    • min.insync.replicas
    • retention.bytes
    • retention.ms
    • segment.bytes
    • segment.jitter.ms
    • segment.ms
    • unclean.leader.election.enable
• Post Deployment
  • Logging
    • Controller
    • State Change Log
    • Request logging
  • Admin operations
    • Adding topics
    • Modifying topics
    • Deleting topics
    • Graceful shutdown
  • Rolling Restart
    • Scaling the Cluster
    • Increasing replication factor
    • Limiting Bandwidth Usage during Data Migration
    • Balancing Replicas Across Racks
    • Enforcing Client Quotas
  • Performance Tips
    • Picking the number of partitions for a topic
    • Tuning virtual memory
    • Lagging replicas
    • Increasing consumer throughput
  • Software Updates
  • Backup and Restoration
• Auto Data Balancing
  • Requirements
  • Quick Start
    • Start a Kafka cluster
    • Create topics and produce data
    • Execute the rebalancer
    • Check status and finish
    • Leader balance
    • Decommissioning brokers
    • Limiting Bandwidth Usage during Data Migration
  • Licensing
  • Configuration Options
    • Configuration Options for the rebalancer tool
• Monitoring Kafka
  • Server Metrics
    • Broker Metrics
    • ZooKeeper Metrics
  • Producer Metrics
    • Global Request Metrics
    • Global Connection Metrics
    • Per-Broker Metrics
    • Per-Topic Metrics
  • New Consumer Metrics
    • Fetch Metrics
    • Topic-level Fetch Metrics
    • Consumer Group Metrics
    • Global Request Metrics
    • Global Connection Metrics
    • Per-Broker Metrics
  • Old Consumer Metrics
• Confluent Metrics Reporter
  • Installation
  • Configuration
    • Message size
  • Configuration Options
    • Security
    • Authentication
    • Authorization
  • Verification
• kafkacat Utility
  • Consumer Mode
  • Producer Mode
  • Metadata Listing Mode