Kafka Design

• Design Overview
  • Topics in this section
  • Learn more
• Kafka and the File System
  • Why the file system?
  • Kafka and the JVM
  • Persistent queue and constant time
  • Related content
• Batch Processing for Efficiency
  • Avoids excessive byte copying
  • Batches I/O operations
  • End-to-end batch compression
  • Related content
• Producer Design
  • What is a producer?
  • Producer load balancing
  • Batching
  • Related content
• Consumer Design
  • What is a consumer?
  • Push versus pull design
  • Consumer groups and group IDs
  • Consumer rebalance protocols and partition assignments
  • Tracking consumer position
  • Consumer offsets
  • Consumer position illustrated
  • Related content
• Kafka Message Delivery Guarantees
  • Semantic guarantees
  • Producer delivery
  • Consumer receipt
  • Exactly once support
  • Related content
• Replication
  • Replication overview
  • In-sync replicas and producer acks
  • Replicated logs: quorums, ISRs, and state machines
  • Unclean leader election and partition loss
  • Replica management
  • Related content
• Kafka Log Compaction
  • Topic compaction
  • Retention example
  • Compaction in action
  • Compaction guarantees
  • Configure compaction
  • Related content
• Quotas
  • Why are quotas necessary?
  • Client groups
  • Quota configuration
  • Network bandwidth quotas
  • Request rate quotas
  • Enforcement
  • Related content