Self-Balancing Clusters

• Self-Balancing Overview
  • What are Self-Balancing Clusters?
  • How Self-Balancing Simplifies Kafka Operations
  • Self-Balancing vs. Auto Data Balancer
  • How it Works
    • Architecture of a Self-Balancing Cluster
    • Enabling Self-Balancing Clusters
    • What defines a “balanced” cluster and what triggers a rebalance?
    • What happens if the lead broker (controller) is removed or lost?
    • How do the brokers leverage Cruise Control?
    • What internal topics does Self-Balancing create and use?
    • Limitations
  • Configuration and Monitoring
    • Using Control Center
    • Kafka Server Properties and Commands
    • Metrics for Monitoring a Rebalance
  • Replica Placement and Rack Configurations
    • Racks
    • Replica Placement and Multi-Region Clusters
    • Capacity
    • Distribution
    • Debugging Rebalance Failures
  • Security Considerations
  • Troubleshooting
    • Self-Balancing options do not show up on Control Center
    • Broker metrics are not displayed on Control Center
    • Consumer lag reflected on Control Center
    • Broker removal attempt fails during Self-Balancing initialization
    • Broker removal cannot complete due to offline partitions
    • Too many excluded topics causes problems with Self-Balancing
  • Suggested Reading
• Quick Start Demo (Docker)
  • About the Demo
  • Setup and What’s Included
  • Start the Services
  • Run the Demo Scenarios
    • Uneven Load
    • Expansion
    • Shrinkage
    • Broker Failure
  • Teardown
  • Suggested Reading
• Tutorial: Adding and Removing Brokers
  • Prerequisites
  • Configure Kafka brokers
    • Enable the Metrics Reporter for Control Center
    • Configure replication factors for Self-Balancing
    • Verify that Self-Balancing is enabled
    • Save the file
    • Create a basic configuration for a five-broker cluster
    • Configure Control Center with REST endpoints and advertised listeners
  • Start Confluent Platform, create topics, and generate test data
  • Use the command line
    • Verify status of brokers and topic data
    • Remove a broker
    • Add a broker (restart)
  • Use Control Center
    • Verify status of brokers and topic data
    • Remove a broker
    • Add a broker (restart)
  • Shutdown and Cleanup Tasks
  • Suggested Reading
• Configuration Options and Commands
  • Self-Balancing Configurations on the Brokers
    • confluent.balancer.enable
    • confluent.balancer.heal.uneven.load.trigger
    • confluent.balancer.heal.broker.failure.threshold.ms
    • confluent.balancer.throttle.bytes.per.second
    • confluent.balancer.disk.max.load
    • confluent.balancer.disk.max.replicas
    • confluent.balancer.exclude.topic.names
    • confluent.balancer.exclude.topic.prefixes
    • confluent.balancer.topic.replication.factor
  • Self-Balancing Internal Topics
  • Required Configurations for Control Center
    • Configure REST Endpoints in the Control Center properties file
    • Configure authentication for REST endpoints on Kafka brokers (Secure Setup)
  • Examples: Update broker configurations on the fly
    • Enable or disable Self-Balancing
    • Set trigger condition for rebalance
    • Set or remove a custom throttle
  • kafka-remove-brokers
    • Flags
    • Examples
    • Broker Removal Phases
    • Self-Balancing Initialization
    • Broker Removal Task Priority
  • Suggested Reading
• Performance and Resource Usage
  • Add brokers to expand a small cluster with a high partition count
    • Test Description
    • Cluster Configurations
    • Performance Results
  • Test scalability of a large cluster with many partitions
    • Test Description
    • Cluster Configurations
    • Performance Results
  • Repeatedly bounce the controller
    • Test Description
    • Cluster Configurations
    • Performance Results