# Kafka and the File System Apache Kafka® is an open-source distributed streaming system used for stream processing, real-time data pipelines, and data integration at scale. This topic describes how and why Kafka relies on the file system for storing and caching messages. ## Why the file system? A key reason for Kafka’s reliance on the file system is that hard drive throughput has been diverging from disk seek latency for the several years. For example, a [JBOD](http://en.wikipedia.org/wiki/Non-RAID_drive_architectures) configuration with six 7200rpm SATA RAID-5 array: - Performs linear writes at about 600 MBps. - Performs random writes at about 100 Kbps, over 6000 times slower than linear writes Linear reads and writes provide a predictable usage pattern, and are heavily optimized by the operating system. A modern operating system provides read-ahead and write-behind techniques that prefetch data in large block multiples and group smaller logical writes into large physical writes for these linear operations. To learn more, read the [ACM Queue article](http://queue.acm.org/detail.cfm?id=1563874) . Of note, the article authors find that [sequential disk access can in some cases be faster than random memory access](http://deliveryimages.acm.org/10.1145/1570000/1563874/jacobs3.jpg). Because of the performance divergence between throughput and disk seeks, modern operating systems have become increasingly aggressive in their use of main memory for disk caching. A modern OS will divert *all* free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and writes will go through this unified cache. This feature is hard to turn off without using direct I/O, so even if a process maintains an in-process cache of the data, this data will likely be duplicated in the OS’s page cache, effectively storing everything twice. ## Kafka and the JVM Kafka is built on top of the JVM, and Java memory usage has the following characteristics: 1. The memory overhead of objects is very high, often doubling the size of the data stored (or worse). 2. Java garbage collection becomes increasingly slow as the in-heap data increases. As a result, using the file system and relying on the kernel page cache is better than an in-memory cache or other structure. The page cache provides at least double the available cache by providing automatic access to all free memory, and likely double again by storing a compact byte structure rather than objects. For example, by using the file system, a 32 GB machine can have a cache of up to 28-30GB, without GC penalties. Furthermore, this cache will stay warm even if the service is restarted, whereas the in-process cache will need to be rebuilt in memory, which takes a long time. With Kafka, instead of maintaining as much as possible and flushing it to the file system when memory space runs out, all data is immediately written to a persistent log on the file system without flushing to disk. This means the data is essentially transferred into the kernel’s page cache. All logic for maintaining coherency between the cache and file system is in the OS, which tends to do so more efficiently and more correctly than one-off in-process attempts. ## Persistent queue and constant time A typical messaging system might use per-consumer queues with associated BTrees or similar data structures to maintain message metadata. BTrees are a versatile data structure, but they come with a fairly high cost; Btree operations are O(log N), and O(log N), which is typically considered equivalent to constant time. In contrast, disk operations are slower. Disk seeks are typically 10 MS each, and each disk can do only one seek at a time, so parallelism is limited. This means that even 5-6 disk seeks leads to a lot of overhead. Since storage systems mix fast cached operations with slow physical disk operations, as data increases with a fixed cache, the performance of tree structures is often superlinear, meaning doubling your data makes things worse than twice as slow. Instead, a persistent queue could be built on simple reads and appends to files like you typically see with logging solutions. Then all operations are O(1) and reads do not block writes or each other. This offers performance advantages because performance is decoupled from the data size. One server can use a number of cheap, low-rotational speed 1+TB SATA drives. They have poor seek performance, but these drives perform acceptably for large reads and writes and are inexpensive. Kafka has access to virtually unlimited disk space without any performance penalty, which means that Kafka can provide some features not usually found in a messaging system. For example, in Kafka, instead of attempting to delete messages as soon as they are consumed, Kafka can retain messages for a relatively long period (a week, for example), which enables flexibility for consumers. ## Related content - [Kafka Producer Design](producer-design.md#producer-design) - [Kafka Consumer Design: Consumers, Consumer Groups, and Offsets](consumer-design.md#consumer-design) - [Kafka Message Delivery Guarantees](delivery-semantics.md#delivery-semantics) - [Kafka Log Compaction](log_compaction.md#log-compaction) - [Kafka Replication and Committed Messages](replication.md#replication) - [Kafka Quotas](quotas.md#quotas) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).