General fixes

• Fix compilation without getentropy. glibc versions lacking support are those less than 2.25 (2017). Happening since 2.13.0.

Consumer fixes

• Remove CPU usage regression when a subscription matches no topics. The increased CPU usage (~30%) was seen in particular when there are many topics in the clusters and the given subscription regex doesn’t match any. Happening since 2.10.0.

• Fix rd_kafka_consume_batch_queue incorrectly updating the application position when receiving EOF or error messages, causing the position to move forward and likely be stored and committed. When storing the application offset the leader epoch is also considered for correct offset ordering in case of log truncation. Happening since 2.2.0.

General fixes

• Strip trailing dot of hostname to fix SSL certificate verification issue. Happening since 1.x.

• Fix memory management for interceptors in rd_kafka_conf to prevent double-free errors. In case the client instance fails the users needs to destroy the configuration data structure, it was causing a double-free because the interceptors were already freed in the constructor. Happening since 1.x.

• Fix for the pseudo-random seed on Windows. The function rand_r isn’t present on Windows and the global seed wasn’t based on the current microseconds and thread ID. Also it wasn’t called on every thread as required on this platform but only once per process. The fix allows on this platform the uniqueness of client side member ID generation in next-generation consumer group protocol. Happening since 1.x.

General fixes

• Fix to restore macOS 13 and 14 compatibility in prebuilt binaries present in librdkafka.redist. Happening since 2.12.0.

General fixes

• Fix for KIP-1102 time based re-bootstrap condition. Re-bootstrap is now triggered only after metadata.recovery.rebootstrap.trigger.ms have passed since first metadata refresh request after last successful metadata response. The calculation was since last successful metadata response so it’s possible it did overlap with the periodic topic.metadata.refresh.interval.ms and cause a re-bootstrap even if not needed. Happening since 2.11.0.

• Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry. Happening since 2.3.0.

Telemetry fixes

• Fix issue in GetTelemetrySubscriptions with big-endian architectures where wrong values are read as accepted compression types causing the metrics to be sent uncompressed. Happening since 2.5.0. Since 2.10.1 unit tests are failing when run on big-endian architectures.

Consumer fixes

• Fixed an issue where topic partition errors were not cleared after a successful commit. Previously, a partition could retain a stale error state even though the most recent commit succeeded, causing misleading error reporting. Now, successful commits correctly clear the error state for the affected partitions Happening since 2.4.0.

Producer fixes

• Fix double free of headers in rd_kafka_produceva method in cases where the partition doesn’t exist. Happening since 1.x.

General fixes

• Made the conditions for enabling the features future proof, allowing to remove RPC versions in a subsequent Apache Kafka® version without disabling features. The existing checks were matching a single version instead of a range and were failing if the older version was removed. Happening since 1.x.

• Avoid returning an all brokers down error on planned disconnections. This is done by avoiding to count planned disconnections, such as idle disconnections, broker host change and similar as events that can cause the client to reach the “all brokers down” state, returning an error and since 2.10.0 possibly starting a re-bootstrap sequence. Happening since 1.x.

• An “all brokers down” error isn’t returned when we haven’t tried to connect to all brokers since last successful connection. It happened because the down state is cached and can be stale when a connection isn’t needed to that particular broker. Solved by resetting the cached broker down state when any broker successfully connects, so that broker needs to be tried again. Happening since 1.x.

General fixes

• A data race happened when emptying buffers of a failing broker, in its thread, with the statistics callback in main thread gathering the buffer counts. Solved by resetting the atomic counters instead of initializing them. Happening since 1.x.

• Features BROKER_BALANCED_CONSUMER and SASL_GSSAPI don’t depend on JoinGroup v0 anymore, missing in AK 4.0 and CP 8.0. This PR partially fixes the linked issue, a complete fix for all features will follow. Rest of fixes are necessary only for a subsequent Apache Kafka® major version, such as AK 5.x. Happening since 1.x.

Telemetry fixes

• Fix for poll ratio calculation in case the queues are forwarded. Poll ratio is now calculated per queue instead of per instance and it allows to avoid calculation problems linked to using the same field. Happens since 2.6.0.

General fixes

• Fix for a loop of re-bootstrap sequences in case the client reaches the all brokers down state. The client continues to select the bootstrap brokers given they have no connection attempt and doesn’t reconnect to the learned ones. In case it happens a broker restart can break the loop for the clients using the affected version. Fixed by giving a higher chance to connect to the learned brokers even if there are new ones that never tried to connect. Happens since 2.10.0.

• Fix to the re-bootstrap case when bootstrap.servers is NULL and brokers were added manually through rd_kafka_brokers_add. Avoids a segmentation fault in this case. Happens since 2.10.0.

Producer fixes

• In case of producev or produceva, the producer did not enqueue a leader query metadata request immediately, and rather, waited for the one second timer to kick in. This could cause delays in the sending of the first message by up to one second. Happens since 1.x.

Consumer fixes

• Fix to add locks when updating the metadata cache for the consumer. It can cause memory corruption or use-after-free in case there’s no broker connection and the consumer group metadata needs to be updated. Happens since 2.10.0.

Telemetry fixes

• Fix for frequent disconnections on push telemetry requests with particular metric configurations. A NULL payload is sent in a push telemetry request when an empty one is needed. This causes disconnections every time the push is sent, only when metrics are requested and some metrics are matching the producer but none the consumer or the other way around. Happens since 2.5.0.

• Avoid copy outside boundaries when reading metric names in telemetry subscription. It can cause that some metrics aren’t matched. Happens since 2.5.0.

• Telemetry metrics aren’t duplicated when multiple prefixes match them. Fixed by keeping track of the metrics that already matched. Happens since 2.5.0.

General fixes

• Brokers are now identified exclusively by broker ID, which matches Java client behavior, instead of by hostname. This prevents incorrectly reusing threads and connections when multiple brokers share the same hostname. Happens since 1.x

• Remove brokers and their threads if they are not included in metadata responses. This prevents the client from attempting to connect to unavailable brokers. Because it is not clear if a broker is removed temporarily or permanently, all absent brokers are removed and are re-added automatically if is included in subsequent metadata responses. Happens since 1.x.

• librdkafka code using cnd_timedwait was yielding before a timeout occurred without the condition being fulfilled because of spurious wake-ups. Solved by verifying with a monotonic clock that the expected point in time was reached and calling the function again if needed. Happens since 1.x.

• Doesn’t remove topics from cache on temporary metadata errors but only on metadata cache expiry. It allows the client to continue working in case of temporary problems to the Kafka metadata plane. Happens since 1.x.

• Doesn’t mark the topic as unknown if it had been marked as existent earlier and topic.metadata.propagation.max.ms hasn’t passed still. It achieves this property expected effect even if a different broker had previously reported the topic as existent. Happens since 1.x.

• Doesn’t update partition leaders if the topic in metadata response has errors. It’s in line with what Java client does and allows to avoid segmentation faults for unknown partitions. Happens since 1.x.

• Only topic authorization errors in a metadata response are considered permanent and are returned to the user. It’s in line with what Java client does and avoids returning to the user an error that wasn’t meant to be permanent. Happens since 1.x.

• Prevent metadata cache corruption when topic ID for the same topic name changes. Solved by correctly removing the entry with the old topic ID from metadata cache to prevent subsequent use-after-free. Happens since 2.4.0.

• Fix for the case where a metadata refresh enqueued on an unreachable broker prevents refreshing the controller or the coordinator until that broker becomes reachable again. Given the request continues to be retried on that broker, the counter for refreshing complete broker metadata doesn’t reach zero and prevents the client from obtaining the new controller or group or transactional coordinator. It causes a series of debug messages like: “Skipping metadata request: … full request already in-transit”, until the broker the request is enqueued on is up again. Solved by not retrying these kinds of metadata requests. Happens since 1.x.

• The Nagle’s algorithm (TCP_NODELAY) is now disabled by default. It caused a large increase in latency for some use cases, for example, when using an SSL connection. For efficient batching, the application should use linger.ms, batch.size etc. Happens since: 0.x.

Consumer fixes

• Commits during a cooperative incremental rebalance could cause an assignment lost if the generation ID was bumped by a second join group request. Solved by not rejoining the group in case an illegal generation error happens during a rebalance. Happening since v1.6.0.

• When switching to a different leader a consumer could wait 500 ms (fetch.error.backoff.ms) before starting to fetch again. The fetch backoff wasn’t reset when joining the new broker. Solved by resetting it, given it’s not needed to backoff the first fetch on a different node. This way faster leader switches are possible. Happens since 1.x.

• The function rd_kafka_offsets_for_times refreshes leader information if the error requires it, allowing it to succeed on subsequent manual retries. Similar to the fix done in 2.3.0 in rd_kafka_query_watermark_offsets. Additionally, the partition current leader epoch is taken from metadata cache instead of from passed partitions. Happens since 1.x.

• When consumer is closed before destroying the client, the operations queue isn’t purged anymore as it contains operations unrelated to the consumer group. Happens since 1.x.

• When making multiple changes to the consumer subscription in a short time, no unknown topic error is returned for topics that are in the new subscription but weren’t in previous one. This was due to the metadata request relative to previous subscription. Happens since 1.x.

• Remove a one second wait after a partition fetch is restarted following a leader change and offset validation. This is done by resetting the fetch error backoff and waking up the delegated broker if present. Happens since 2.1.0.

General fixes

• Socket options are now all set before connection, as documentation says it’s needed for socket buffers to take effect, even if in some cases they could have effect even after connection. Happening since v0.9.0.

• Client certificate chain is now sent when using ssl.certificate.pem or ssl_certificate or ssl.keystore.location. Without that, broker must explicitly add any intermediate certification authority certificate to its truststore to be able to accept client certificate. Happens since: 1.x.

Consumer fixes

• Fix to allow to migrate partitions to leaders with NULL leader epoch. NULL leader epoch can happen during a cluster roll with an upgrade to a version supporting KIP-320. Happening since v2.1.0.

• Fix to allow to migrate partitions to leaders with same leader epoch. Same leader epoch can happen when partition is temporarily migrated to the internal broker, or if broker implementation never bumps it, as it’s not needed to validate the offsets. Happening since v2.4.0.

General fixes

• SASL and SCRAM authentication fix: avoid concatenating client side nonce once more, as it’s already prepended in server sent nonce. librdkafka was incorrectly concatenating the client side nonce again, leading to this fix being made on AK side, released with 3.8.1, with endsWith instead of equals. Happening since v0.0.99.

Consumer fixes

• Fix for a fetch regression when connecting to Apache Kafka® versions earlier than 2.7, causing fetches to fail. Happening since v2.6.0.

• A consumer configured with the cooperative-sticky partition assignment strategy could get stuck in an infinite loop, with corresponding spike of main thread CPU usage. That happened with some particular orders of members and potential assignable partitions. Solved by removing the infinite loop cause. Happening since: 1.6.0.

• When retrieving offset metadata, if the binary value contained zeros and librdkafka was configured with strndup, part of the buffer after first zero contained uninitialized data instead of rest of metadata. Solved by avoiding to use strndup for copying metadata. Happening since: 0.9.0.

• When an out of range on a follower caused an offset reset, the corresponding ListOffsets request is made to the follower, causing a repeated “Not leader for partition” error. Fixed by sending the request always to the leader. Happening since 1.5.0 (tested version) or previous ones.

• Issues: Fix to remove fetch queue messages that blocked the destroy of rdkafka instances. Circular dependencies from a partition fetch queue message to the same partition blocked the destroy of an instance, that happened in case the partition was removed from the cluster while it was being consumed. Solved by purging internal partition queue, after being stopped and removed, to allow reference count to reach zero and trigger a destroy. Happening since 2.0.2.

Consumer fixes

• Fix for permanent fetch errors when brokers support a fetch RPC version greater than 12 but cluster is configured to use an inter broker protocol that is less than 2.8. In this case returned topic IDs are zero valued and fetch has to fall back to version 12, using topic names. Happening since v2.5.0.

Telemetry fixes

• Issue: #4833 Fix a regression introduced with KIP-714 support in which an assert is triggered during PushTelemetry call. This happens when no metric is matched on the client side among those requested by broker subscription. Happening since 2.5.0.

General fixes

• Issues: confluentinc/confluent-kafka-dotnet#2084 Fix segfault when a segment is erased and more data is written to the buffer. Happens since 1.x when a portion of the buffer (segment) is erased for FlexVer or compression. More likely to happen since 2.1.0, because of the upgrades to FlexVer, with certain string sizes like a long client ID.

Idempotent producer fixes

• Fix for an idempotent producer error, with a message batch not reconstructed identically when retried. Caused the error message “Local: Inconsistent state: Unable to reconstruct MessageSet”. Happening on large batches. Solved by using the same backoff baseline for all messages in the batch. Happens since 2.2.0.

KIP-848: The next version of the Consumer Rebalance Protocol

• With this new protocol the role of the Group Leader (a member) is removed and the assignment is calculated by the Group Coordinator (a broker) and sent to each member through heartbeats.

The feature is still not production ready. It’s possible to try it in a non-production environment.

General fixes

• Issues: confluentinc/confluent-kafka-go#981. In librdkafka release pipeline a static build containing libsasl2 could be chosen instead of the alternative one without it. That caused the libsasl2 dependency to be required in confluent-kafka-go v2.1.0-linux-musl-arm64 and v2.3.0-linux-musl-arm64. Solved by correctly excluding the binary configured with that library, when targeting a static build. Happening since v2.0.2, with specified platforms, when using static binaries.

• When the main thread loop was awakened less than one ms before the expiration of a timeout, it was serving with a zero timeout, leading to increased CPU usage until the timeout was reached. Happening since 1.x.

• Metadata cache was cleared on full metadata refresh, leading to unnecessary refreshes and occasional UNKNOWN_TOPIC_OR_PART errors. Solved by updating cache for existing or hinted entries instead of clearing them. Happening since 2.1.0.

• A metadata call before member joins consumer group, could lead to an UNKNOWN_TOPIC_OR_PART error. Solved by updating the consumer group following a metadata refresh only in safe states. Happening since 2.1.0.

• Metadata refreshes without partition leader change could lead to a loop of metadata calls at fixed intervals. Solved by stopping metadata refresh when all existing metadata is non-stale. Happening since 2.3.0.

• A partition migration could happen, using stale metadata, when the partition was undergoing a validation and being retried because of an error. Solved by doing a partition migration only with a non-stale leader epoch. Happening since 2.1.0.

Consumer fixes

• In case of subscription change with a consumer using the cooperative assignor it could resume fetching from a previous position. That could also happen if resuming a partition that wasn’t paused. Fixed by ensuring that a resume operation is completely a no-op when the partition isn’t paused. Happening since 1.x.

• While using the cooperative assignor, given an assignment is received while closing the consumer it’s possible that it gets stuck in state WAIT_ASSIGN_CALL, while the method is converted to a full unassign. Solved by changing state from WAIT_ASSIGN_CALL to WAIT_UNASSIGN_CALL while doing this conversion. Happening since 1.x.

General fixes

• An assertion failed with insufficient buffer size when allocating rack information on 32-bit architectures. Solved by aligning all allocations to the maximum allowed word size.

• The timeout for rd_kafka_query_watermark_offsets was not enforced after making the necessary ListOffsets requests, and thus, it never timed out in case of broker or network issues. Fixed by setting an absolute timeout.

Idempotent producer fixes

• After a potentially persisted error, such as a disconnection or timeout, the producer incorrectly incremented the next expected sequence number. This caused a fatal OUT_OF_ORDER_SEQUENCE_NUMBER error when the failed message wasn’t actually persisted and a subsequent message was sent.

  This issue showed up as “sequence desynchronization” after a single potentially persisted error or “rewound sequence number” after multiple errored messages.

  Fixed by treating potentially persisted messages as not persisted. The producer now expects a DUPLICATE_SEQUENCE_NUMBER error if the message was actually persisted or NO_ERROR if the message wasn’t. In both cases, the message is correctly marked as delivered.

Consumer fixes

• Stored offsets were excluded from the commit if the leader epoch was less than committed epoch, as it’s possible if leader epoch is the default -1. This didn’t happen in Python, Go and .NET bindings when stored position was taken from the message. Solved by checking only that the stored offset is greater than committed one, if either stored or committed leader epoch is -1.

• If an OffsetForLeaderEpoch request was being retried, and the leader changed while the retry was in-flight, an infinite loop of requests was triggered, because we weren’t updating the leader epoch correctly. Fixed by updating the leader epoch before sending the request.

• During offset validation a permanent error like host resolution failure would cause an offset reset. This isn’t what’s expected or what the Java implementation does. Solved by retrying even in case of permanent errors.

• If using rd_kafka_poll_set_consumer, along with a consume callback, and then calling rd_kafka_poll to service the callbacks, would not reset max.poll.interval.ms. This was because we were only checking rk_rep for consumer messages, while the method to service the queue internally also services the queue forwarded to from rk_rep, which is rkcg_q. Solved by moving the max.poll.interval.ms check into rd_kafka_q_serve.

• After a leader change a rd_kafka_query_watermark_offsets call would continue trying to call ListOffsets on the old leader, if the topic wasn’t included in the subscription set, so it started querying the new leader only after topic.metadata.refresh.interval.ms.