Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Kafka Operations

In this document, you will learn how to manage an Apache Kafka® cluster.

Here, we will cover three main topics:

• Deploying your cluster to production, including best practices and important configuration that should (or should not!) be changed
• Performing post-deployment logistics, such as a rolling restart or backup of your cluster
• Monitoring your cluster’s vital statistics, understanding which behaviors are normal and which should be cause for alarm, and interpreting various stats provided by Kafka

• Running Kafka in Production
  • Hardware
    • Memory
    • CPUs
    • Disks
    • Network
    • Filesystem
  • General Considerations
    • VMware optimization
  • JVM
  • Production Configuration Options
    • General configs
    • Replication configs
  • File Descriptors and mmap
  • Multi-node Configuration
• Dynamic Configurations
  • Changing Broker Configurations Dynamically
  • Updating Password Configurations Dynamically
  • Updating Password Configurations in ZooKeeper Before Starting Brokers
  • Updating SSL Keystore of an Existing Listener
  • Updating SSL Trust Store of an Existing Listener
  • Changing Topic Configurations Dynamically
    • cleanup.policy
    • flush.messages
    • flush.ms
    • max.message.bytes
    • min.insync.replicas
    • retention.bytes
    • retention.ms
    • segment.bytes
    • segment.jitter.ms
    • segment.ms
    • unclean.leader.election.enable
• Post Kafka Deployment
  • Logging
    • Controller
    • State Change Log
    • Request logging
  • Admin operations
    • Adding topics
    • Modifying topics
    • Deleting topics
    • Graceful shutdown
  • Rolling restart
    • Scaling the Cluster
    • Increasing replication factor
    • Limiting Bandwidth Usage during Data Migration
    • Balancing Replicas Across Racks
    • Enforcing Client Quotas
  • Performance Tips
    • Picking the number of partitions for a topic
    • Tuning virtual memory
    • Lagging replicas
    • Increasing consumer throughput
    • Handling large message sizes
  • Software Updates
  • Backup and Restoration
• Auto Data Balancing
  • Requirements
  • Confluent Auto Data Balancer Quick Start
    • Start a Kafka cluster
    • Create topics and produce data
    • Execute the rebalancer
    • Check status, finish and cancel
    • Leader balance
    • Decommissioning brokers
    • Limiting bandwidth usage during data migration
    • Limiting the scope of the reassignment
    • Incremental reassignment
    • Use JMX for monitoring
  • Licensing
  • Suggested Reading
• Monitoring Kafka
  • Server Metrics
    • Broker Metrics
    • ZooKeeper Metrics
  • Producer Metrics
    • Global Request Metrics
    • Global Connection Metrics
    • Per-Broker Metrics
    • Per-Topic Metrics
  • RBAC and LDAP Health Metrics
  • New Consumer Metrics
    • Fetch Metrics
    • Topic-level Fetch Metrics
    • Consumer Group Metrics
    • Global Request Metrics
    • Global Connection Metrics
    • Per-Broker Metrics
  • Old Consumer Metrics
• Confluent Metrics Reporter
  • Installation
  • Configuration
    • Message size
  • Configuration Options
  • Security
  • Authentication
  • Role-Based Access Control
    • Authorization
    • RBAC with Kerberos enabled
    • RBAC with mTLS running on the same cluster as MDS
  • Verification
• kafkacat Utility
  • Consumer Mode
  • Producer Mode
  • Metadata Listing Mode
  • kafkacat example code
• Tiered Storage
  • Enabling Tiered Storage on a Broker
    • AWS
    • GCS, Azure, JBOD
  • Creating a Topic with Tiered Storage
  • Notable Broker Configs
    • Log Segment Size
    • Topic Deletion Interval
  • Tiered Storage Broker Metrics
    • Tier Archiver Metrics
    • Tier Fetcher Metrics
  • Example Performance Test
  • Known Limitations
  • Suggested Reading