Delivery Guarantees and Latency in Confluent Cloud for Apache Flink
Confluent Cloud for Apache Flink® provides exactly-once semantics end-to-end by default, which means that the output of a statement reflects every input message exactly-once and that every output message is delivered exactly once.
To achieve this, Confluent Cloud for Apache Flink relies on Apache Flink®’s checkpointing mechanism and Kafka transactions. While checkpointing and fault tolerance fall into Confluent’s responsibility, you should understand the implications of how Flink reads from and writes to Kafka:
Flink statements write to Kafka by using transactions. Flink commits transactions periodically, approximately every minute.
Flink by default only reads committed messages from Kafka. For more information, see isolation.level.
This implies that depending on the delivery guarantees required by your use case, you can currently achieve different end-to-end latencies with Flink.
Exactly-Once: If you require exactly-once, the latency is roughly one minute and depends on the interval at which Kafka commits transactions. In this case, ensure that all consumers of the output topics of Flink statements use
isolation.level: read_committedor set the Flink table option'kafka.consumer.isolation-level' = 'read-committed'.At-Least-Once: If at-least-once is sufficient for your use case, you can read from the output topics with
isolation-level: read_uncommitted, which is the default in Kafka, or set the Flink table option'kafka.consumer.isolation-level' = 'read-uncommitted'. With this configuration, depending on the logic of your query, you can achieve an end-to-end latency below 100 ms, but you can see some output messages twice. This happens when Flink needs to abort a transaction that your consumer has already read. You don’t see incorrect results, but you can see the same correct result multiple times.
Note
Confluent is actively working on reducing the latency under exactly-once semantics. If your use case requires a lower latency, reach out to Support or your account manager.