Kafka Design

• Design Overview
  • Topics in this section
  • Learn more
• Kafka and the File System
  • Kafka and the JVM
  • Persistent queue and constant time
• Batch Processing for Efficiency
  • Avoids excessive byte copying
  • Batches I/O operations
  • End-to-end batch compression
• Producer Design
  • Producer load balancing
  • Batching
  • Additional reading
• Consumer Design
  • Push versus pull
  • Track consumer position
  • Learn more
• Message Delivery Guarantees
  • Producer delivery
  • Consumer receipt
  • Exactly once support
  • Learn more
• Replication
  • Committed message guarantees
  • Replicated logs: quorums, ISRs, and state machines
  • Unclean leader election and partition loss
  • Replica management
• Log Compaction
  • Compaction in action
  • Compaction guarantees
  • Configure compaction
• Quotas
  • Why are quotas necessary?
  • Client groups
  • Quota configuration
  • Network bandwidth quotas
  • Request rate quotas
  • Enforcement