Kafka .NET Client Changelog

2.13.2

Enhancements

  • Preserve inner exceptions in RestService.

Fixes

  • Support multi-certificate PEM bundles in Schema Registry SslCaLocation.

2.13.1

Enhancements

  • Unify build configurations and remove redundant properties.

  • JSONSchema Validation Errors: Include paths to nested properties.

  • Add Confluent-Client-Version header to requests to SR, .NET Client.

Fixes

  • Fix DeriveKey in Hkdf implementation.

  • Ensure single use of HttpRequestMessage for token fetch.

  • Fix race conditions when retrieving KEK/DEK.

  • Ensure JSON schema access is thread-safe.

  • Use lock on root schema when protecting access to sub-schemas.

  • Processing arrays and maps sequentially to avoid race conditions.

2.13.0

Enhancements

  • References librdkafka.redist 2.13.0. Refer to the librdkafka v2.13.0 release notes for more information.

  • Support Avro schema references for .NET Client.

  • Add Accept-Version header.

  • Handle evolution during field transformation.

  • Re-enable deserialization optimization by removing unnecessary array conversions.

Fixes

  • Ensure schemaId initialization is thread-safe.

  • Ensure all deps are strongly named.

2.12.0

KIP-848 – General Availability

Starting with Confluent.Kafka 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Please refer to the following migration guide for moving from Classic to Consumer protocol.

Note

The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. GroupProtocol configuration property dictates whether to use the new Consumer protocol or older Classic protocol. It defaults to Classic if not provided.

Enhancements

  • References librdkafka.redist 2.12.0. Refer to the librdkafka v2.12.0 release notes for more information.

  • OAuth OIDC method example for Kafka metadata based authentication with an Azure IMDS endpoint using an attached managed identity as principal.

2.11.1

Enhancements

2.11.0

Enhancements

2.10.1

Enhancements

  • References librdkafka.redist 2.10.1. Refer to the librdkafka v2.10.1 release notes for more information.

  • Refactor serializer implementations to remove unnecessary ContinueWith calls.

  • Remove shared mutex from CachedSchemaRegistryClient.

  • Protobuf serialization perf improvements.

  • Refactor AuthenticationHeaderValueProvider construction.

  • Remove redundant ConfigureAwait.

  • Support for schema id in header.

  • Use vectorized null-terminated C-string parsing in PtrToStringUTF8 for .NET 6+.

  • Rework SchemaId encoding in headers to minimize heap allocations.

  • Raise an error if Protobuf deprecated format is specified.

  • Avoid array copy on Avro deserialization.

Fixes

  • Fix JSON Schema validation to use latest schema if needed.

  • Minor check for null client.

  • Fix possible NPE in CSFLE executor.

  • Add missing ConfigureAwait calls.

  • Fix rule transformation of list properties.

2.10.0

Enhancements

  • References librdkafka.redist 2.10.0. Refer to the librdkafka v2.10.0 release notes for more information.

  • [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs.

2.9.0

Enhancements

  • Add utilities to convert decimals to or from Protobuf.

  • Add client credentials OAuth support.

  • Add support for passing schema ID during serialization.

  • Optimize schema registry client by caching results of LookupSchemaAsync.

  • Use ConcurrentDictionary as default cache for Schema Registry client.

Fixes

  • Fix JSON Schema validation to use latest schema if needed.

  • Support Protobuf oneof fields in Data Contract rules.

2.8.0

Enhancements

  • References librdkafka.redist 2.8.0. Refer to the librdkafka v2.8.0 release notes for more information.

  • Add retry logic to RestService for Schema Registry.

  • Add config for validating payloads against JSON Schema.

  • Add ability to override disable flag and actions on a rule.

  • Add AWS AssumeRole support to AWS KMS.

Fixes

  • Retrieve version for writer schema when getting migrations.

  • Add missing JSON Schema validate check.

  • Ensure different key ids use different client instances.

  • DGS-19409 Ensure Avro serde caches per subject.

  • Update hash code for schema references so they can be retrieved correctly.

2.6.1

Enhancements

Fixes

  • Fix to continue supporting .NET Framework 4.6.2+ in core client library.

  • Fix JSON Schema handling to not require use of $id.

  • Update Caching.Memory to 8.0.1 to address CVE.

  • Added Qualified and Custom reference name strategy approaches for Protobuf references.

  • Fix validate of SSL CA certs in Schema Registry client.

  • Skip SSL certs validation when configured in Schema Registry client.

  • Allow proxy to be specified in Schema Registry client.

2.6.0

Enhancements

  • References librdkafka.redist 2.6.0. Refer to the librdkafka v2.6.0 release notes for more information.

  • KIP-848 EA: Admin API for listing consumer groups now has an optional filter to return only groups of given types.

  • KIP-460 Admin Leader Election RPC.

  • .NET 8 support with NJsonSchema 11.

2.5.3

Enhancements

Fixes

  • Properly handle messages with well-known types in Protobuf serializer.

  • Use AES128_GCM in the Local KMS client, for consistency with Java and Go.

  • Include deleted schemas when getting schemas by subject and version.

  • Handle signed integers when transforming Protobuf payloads.

  • Allow null SchemaRegistryClient in AsyncSerde constructor.

2.5.2

Warning

Versions 2.5.0, 2.5.1 and 2.5.2 have a regression in which an assert is triggered during PushTelemetry call. This happens when no metric is matched on the client side among those requested by broker subscription.

You won’t face any problem if:

  • Broker doesn’t support KIP-714.

  • KIP-714 feature is disabled on the broker side.

  • KIP-714 feature is disabled on the client side. This is enabled by default. Set configuration enable.metrics.push to false.

  • If KIP-714 is enabled on the broker side and there is no subscription configured there.

  • If KIP-714 is enabled on the broker side with subscriptions that match the KIP-714 metrics defined on the client.

You should use v2.5.3 and above to avoid this regression.

Fixes

  • Fix client-side field-level encryption (CSFLE) to use the Google Tink format for DEKs for interoperability with clients in other languages (Java, Go, etc.).

  • Improve error when specifying an invalid KMS type for CSFLE.

  • Enhance CSFLE examples with KMS configuration settings.

2.5.1

Fixes

  • Fix client-side field-level encryption (CSFLE) when using Azure Key Vault by specifying RsaOaep256 (instead of RsaOaep) for interoperability with clients in other languages (Java, Go, etc.).

  • Fix AvroSerializer configuration to allow using schema normalization.

  • Upgrade Azure Identity library to 1.11.4 to address a vulnerability in previous versions.

2.5.0

Enhancements

  • References librdkafka.redist 2.5.0. Refer to the librdkafka v2.5.0 release notes for more information.

  • Add support for metadata and ruleSet in the schema registry client, which together support data contracts.

  • Add support for client-side field-level encryption (CSFLE) for AWS, Azure, Google Cloud, and HashiCorp Vault. See the encryption examples in the examples directory.

  • Add support for CEL, CEL_FIELD, and JSONata rules.

Fixes

  • Switch license expression and other repo information.

2.4.0

Enhancements

  • References librdkafka.redist 2.4.0. Refer to the librdkafka v2.4.0 release notes for more information.

  • KIP-848 EA: Added KIP-848 based new consumer group rebalance protocol. Integration tests running with the new consumer group protocol. This feature is in early access and is not suitable for production environments. Please refer detailed doc for more information.

2.3.0

Enhancements

  • References librdkafka.redist 2.3.0. Refer to the librdkafka v2.3.0 release notes for more information.

  • KIP-430: Return authorized operations in describe responses.

  • KIP-396: Added support for ListOffsets Admin API.

  • Add Rack to the Node type, so AdminAPI calls can expose racks for brokers (currently, all Describe Responses).

  • Added support for external JSON schemas in JsonSerializer and JsonDeserializer.

  • Added compatibility methods to CachedSchemaRegistryClient.

  • Add support for AdminAPI DescribeCluster() and DescribeTopics().

2.2.0

Enhancements

Fixes

  • Fix backwards compatibility of TopicPartitionOffset constructor.

  • Fix IConsumer breaking change.

2.1.1

Enhancements

2.1.0

Enhancements

  • References librdkafka.redist 2.1.0. Refer to the librdkafka v2.1.0 release notes for more information.

  • Added SetSaslCredentials. This new method on the Producer, Consumer, and AdminClient allows modifying the stored SASL PLAIN/SCRAM credentials that will be used for subsequent new connections to a broker.

  • Changed the way the _SCHEMA filed is accessed internally from reflecting the static field to accessing it from the instance.

  • KIP-320: add offset leader epoch fields to the TopicPartitionOffset, TopicPartitionOffsetError and ConsumeResult classes.

Fixes

  • Fixed OverflowException thrown intermittently when using the ListGroup method.

2.0.2

Upgrade considerations

OpenSSL 3.0.x upgrade in librdkafka requires a major version bump, as some legacy ciphers need to be explicitly configured to continue working, but it is highly recommended not to use them. The rest of the API remains backward compatible.

Enhancements

  • References librdkafka.redist 2.0.2. Refer to the librdkafka v2.0.0 release notes and later ones for more information.

  • Upgraded NJsonSchema to v10.6.3.

  • Added LatestCompatibilityStrict configuration property to JsonSerializerConfig to check the compatibility with latest schema when UseLatestVersion is set to true.

  • Added DeleteConsumerGroupOffset to AdminClient.

  • KIP-222 Finish remaining implementation: Add Consumer Group operations to Admin API. DeleteGroups is already present.

  • KIP-518 Allow listing consumer groups per state.

  • KIP-396 Partially implemented: support for AlterConsumerGroupOffsets.

  • As result of the above KIPs, added.

    • ListConsumerGroups Admin operation. Supports listing by state.

    • DescribeConsumerGroups Admin operation. Supports multiple groups.

    • ListConsumerGroupOffsets Admin operation. Currently, only supports one group with multiple partitions. Supports the requireStable option.

    • AlterConsumerGroupOffsets Admin operation. Currently, only supports one group with multiple offsets.

Fixes

  • During a group rebalance, partitions are now always revoked as a side effect of a call to Consume, whether or not a partitions revoked handler has been specified. Previously, if no handler was specified, the timing of when the consumer lost ownership of partitions during a rebalance was arbitrary, frequently resulting in an erroneous state exception when committing or storing offsets.

  • Fixed 100% CPU usage with DependentAdminClientBuilder.

Note

There were no 2.0.0 and 2.0.1 releases.

1.9.3

Enhancements

  • Added NormalizeSchemas configuration property to the Avro, Json and Protobuf serdes.

Fixes

  • Schema Registry authentication now works with passwords that contain the ‘:’ character.

  • Added missing librdkafka internal and broker error codes to the ErrorCode enum.

1.9.2

Enhancements

  • References librdkafka.redist 1.9.2 which includes an Apple M1 librdkafka build.

  • Added ACL AdminClient operations for CreateAcls, DescribeAcls, DeleteAcls.

  • Added DeleteGroups to AdminClient.

  • Enhanced the Avro Specific Deserializer to ignore the type namespace.

  • Improved efficiency of the statistics handler.

Fixes

  • The AdminClient poll loop no longer terminates when a request results in an error.

  • Upgraded Newtonsoft.Json to 13.0.1 to address a security vulnerability in 9.0.1.

1.9.1

There was no 1.9.1 release of the .NET Client.

1.9.0

Enhancements

  • References librdkafka.redist 1.9.0. Refer to the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

  • References Apache.Avro 1.11.0. Refer to the release notes for further information.

  • Added support for serializing and deserializing null in Avro serdes.

  • Enhanced CachedSchemaRegistryClient to allow for user-implemented authentication schemes.

  • Reduced memory use when producing with delivery reports disabled.

Fixes

  • Resolved incompatibility with Apple M1 processors.

  • No longer crashes on Alpine Linux when MUSL is installed.

  • JSON validation exception messages now properly include failing paths.

  • Upgraded Google.Protobuf dependency to 3.15.0 for CVE-2021-22570.

  • Resolved memory leak in AdminClient response handler.

  • Resolved memory leak in Producer.SendOffsetsToTransaction.

Upgrade Considerations

  • The earliest supported .NET Framework version is now 4.6.2, previously this was 4.5.

1.8.2

Enhancements and Fixes

  • References librdkafka.redist 1.8.2. Refer to the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

  • Added the SslCaPem configuration property to specify a CA certificate using a PEM string.

1.8.1

Enhancements

  • Updated NJsonSchema to v10.5.2.

1.8.0

Enhancements

  • References librdkafka.redist 1.8.0. Refer to the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

  • Added the UseLatestVersion configuration property to the Protobuf, JSON Schema and Avro serdes.

Fixes

  • Breaking Change: Updated the message framing format used by the Protobuf SerDes in ProtobufSerializer and ProtobufDeserializer to be compatible with the Java Protobuf SerDes. Message indices now use zigzag encoding.

    Note

    This framing encodes schema metadata, enabling integration with Confluent Schema Registry. To disable, set the UseDeprecatedFormat configuration property to true.

Security

  • Upgraded the bundled zlib version from 1.2.8 to 1.2.11 in the librdkafka.redist NuGet package. The updated zlib version fixes CVEs: CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843. See https://github.com/edenhill/librdkafka/issues/2934 for more information.

1.7.0

Enhancements

  • References librdkafka.redist 1.7.0. Refer to the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

  • Added OAuth support to AdminClient

Fixes

  • Resolved a schema caching bug #1587 in CachedSchemaRegistryClient.GetSchemaIdAsync.

  • Fixed a configuration error in the Web example.

Security

1.6.3

Fixes

  • References Apache.Avro v1.10.2, which resolves an issue with large string deserialization AVRO-3005.

1.6.2

Enhancements

  • References librdkafka.redist 1.6.1. Refer to the 1.6.0 and 1.6.1 release notes for more information. Headline features:

    • KIP-429: Incremental rebalancing.

    • KIP-447: Producer scalability for exactly once semantics.

    • KIP-480: Sticky partitioner.

  • KIP-22: Support for custom partitioners.

  • Confluent.Kafka can now be used with Mono on Linux and MacOS.

    Note

    Mono is not a supported runtime.

  • The debian9-librdkafka.so build of librdkafka has been replaced with a more portable one: centos6-librdkafka.so.

    Note

    Debian 9 is still supported.

  • Exceptions thrown by Producer.Produce now include an inner exception with additional context on the error.

  • Added ConfigureAwait(false) to async methods in the Avro Serdes.

  • Added IsInvalid property to Handle class.

Fixes

  • Fixed race condition in ProtobufSerializer.

1.6.0, 1.6.1

Version 1.6.0 and 1.6.1 were not released.

1.5.3

Enhancements

  • References librdkafka 1.5.3. Refer to the release notes for more information.

  • References Apache.Avro v1.10.1, which adds support for enum defaults AVRO-2750.

1.5.2

Enhancements

  • References librdkafka 1.5.2. Refer to the release notes for more information.

  • Avro serializer now supports generic parameter ISpecificRecord. In this case, data is serialized according to the per-message concrete type.

1.5.1

Enhancements

  • Added support for OAuth Authentication via SASL/OAUTHBEARER for KIP-255.

1.5.0

Enhancements

  • References librdkafka 1.5.0 which brings many small improvements and bug fixes. Refer to the release notes for more information.

  • Added support for Schema Registry SSL Authentication.

1.4.4

Fixes

  • Resolved a stack overflow issue in Error(IntPtr error) #1249

  • References librdkafka 1.4.4. Refer to the release notes for more information.

1.4.3

Fixes

  • Subject names are now URL encoded when used in Schema Registry URLs.

  • Fixed a memory leak that occurred when passing a CancellationToken to Producer.ProduceAsync.

1.4.2

Fixes

  • Maintenance release. Refer to the librdkafka release notes for more information.

  • Fixed incorrect content-type header in Schema Registry HTTP requests.

1.4.0

Enhancements

  • References librdkafka v1.4.0. Refer to the release notes for more information. Headline features:

    • KIP-98: Producer support for transactions. This is the final improvement required to enable exactly once stream (EOS) processing in .NET.

    • KIP-345: Static consumer group membership.

    • KIP-511: Client name and version are now provided to brokers.

  • Added Protobuf and JSON serdes including integration with Schema Registry.

  • Switched to the official Apache Avro nuget package, which includes support for logical types, and all fixes from the Confluent fork, which has now been discontinued.

  • Message headers are now exposed to serdes via SerializationContext.

  • Added a CancellationToken parameter to the ProduceAsync methods.

  • Uncaught exceptions thrown in handler methods are now propagated to the initiating function, or in the case of error or log events, ignored. Previously, they would cause the application to terminate.

  • Added a WordCount example demonstrating a streaming map-reduce application with exactly-once processing.

Changes

  • Some internal improvements to the Consumer.

  • Breaking change: net452 is no longer a target framework of Confluent.SchemaRegistry or Confluent.SchemaRegistry.Serdes due to the switch to the official Apache Avro package which only targets netstandard2.0.

  • Marked properties on ConsumeResult that simply delegate to the corresponding properties on ConsumeResult.Message as obsolete.

Fixes

  • Fixed an ArgumentNullException regression in ListGroups.

1.3.0

Enhancements

  • Added support for Subject Name Strategies to Confluent.SchemaRegistry.

  • ConsumeResult now throws MessageNullException, not NullReferenceException when a message property is accessed but no message exists.

  • References librdkafka v1.3.0. Refer to the release notes for more information. Headline feature is support for fetch from follower for KIP-392.

Changes

  • Deprecated properties of SchemaRegistryConfig with the unnecessary prefix SchemaRegistry. Added corresponding properties without this prefix.

Fixes

  • Resolved issue 993 whereby RestService was unable to communicate with Schema Registry hosted on a non-root path.

1.2.2

  • References librdkafka v1.2.2 which upgrades the lz4 dependency to v1.9.2.

1.2.1

Fixes

  • References librdkafka v1.2.1 which resolves an issue that broke GSSAPI authentication on Windows.

1.2.0

Bugs

Warning

There is an issue with SASL GSSAPI authentication on Windows with this release. This is resolved in v1.2.1.

Enhancements

  • References librdkafka v1.2.0. Refer to the release notes for more information. Headline feature is consumer side support for transactions.

  • Added IDictionary overload to Config constructors.

  • Confluent.Kafka, Confluent.SchemaRegistry and Confluent.SchemaRegistry.Serdes are now all signed, and Confluent.Kafka.StrongName deprecated.

Fixes

  • Updated the librdkafka build load order so that the version with the most features is used on any given platform.

1.1.0

Enhancements

  • References librdkafka v1.1.0. Refer to the release notes for more information. Notable improvement for Windows SSL users: You no longer need to specify a CA certificate file/directory (SslCaLocation). Librdkafka will load the CA certs by default from the Windows Root Certificate Store.

1.0.1.1

Changes

  • Applied ConfigureAwait(false) to all internal await calls, which resolves deadlock issues in synchronization contexts with limited numbers of threads #967.

1.0.1

Enhancements

  • Support for Alpine Linux.

  • New LogLevelType enum and functions to convert between different log level type levels.

  • Added netstandard20 as a target.

  • References librdkafka 1.0.1.

1.0.0

Summary

1.0.0 is a major update of the API, introducing many new features and enhancements.

Note

The 1.0 API is not compatible with earlier versions of the library.

Feature highlights:

  • Inherits all of the new features in librdkafka v1.0.0

  • General improvements to client classes:

    • Strongly typed configuration.

    • Construction is by builder classes:

      • Allows or enforces that event handlers are specified at construction time.

      • More extensible.

    • Header support.

    • New Message class abstraction and related changes.

    • Consistency in error reporting across library by exceptions.

    • Support for fatal errors.

  • Added AdminClient:

    • CreateTopics, DeleteTopics, CreatePartitions, AlterConfigs, and DescribeConfigs methods.

    • Moved ListGroups and GetMetadata methods from the Producer and Consumer classes to AdminClient.

  • Producer specific improvements:

    • New serialization interface:

      • Non-blocking support for async serializers.

      • Very flexible. For example, can be easily extended to support header serialization.

    • Capability to specify custom timestamps when producing messages.

    • Message persistence status support.

    • Renamed ProduceAsync variants with a callback to Produce.

  • Consumer improvements:

    • A new rebalance API.

    • New deserialization API analogous to the new serialization API.

    • PartitionEOF notification is via ConsumeResult, not events.

      • EOF notification is now disabled by default. To enable, set the EnablePartitionEof config property to true.

  • Confluent Schema Registry integration.

    • Added support for basic authentication.

    • Added GET subject versions to the cached schema registry client.

    • Renamed Confluent.Kafka.Avro to Confluent.SchemaRegistry.Serdes in preparation for support for additional serialization formats.

1.0.0-RC7

Changes

  • Moved SyncOverAsync functionality to the Confluent.Kafka.SyncOverAsync namespace.

  • Marked DependentProducerBuilder as API-SUBJECT-TO-CHANGE.

  • No-op handlers are no longer registered with librdkafka if corresponding handlers are not specified in client builder classes.

  • Renamed AsyncAvroSerializer to AvroSerializer and AsyncAvroDeserializer to AvroDeserializer.

1.0.0-RC6

New Features

  • Added DependentAdminClientBuilder class.

Changes

  • Reverted RC4 changes.

  • Renamed AvroSerializer to AsyncAvroSerializer and AvroDeserializer to AsyncAvroDeserializer.

  • Added SyncOverAsyncSerializer and SyncOverAsyncDeserializer adapter classes.

  • Added AsSyncOverAsync factory method to AsyncAvroSerializer and AsyncAvroDeserializer.

  • Removed IAsyncDeserializer setter overloads from the ConsumerBuilder class.

  • Renamed Producer.BeginProduce to Producer.Produce.

  • Produce throws an exception if used when async serializers are configured.

  • Made AdminClient, Producer, and Consumer classes internal.

1.0.0-RC4

Changes

  • Removed SerializationContext from non-async serde interfaces.

  • Replaced ISerializer interface with Serializer delegate.

  • Replaced IDeserializer interface with Deserializer delegate.

1.0.0-RC3

New Features

  • Producer.Poll can now be used with producer instances that are in background polling mode.

    • Typically use: Block for a minimal period of time following a ErrorCode.Local_QueueFull error.

Changes

  • Removed the Confluent.Kafka.Serdes namespace.

Fixes

  • Added CompressionType property to ProducerConfig class.

1.0.0-RC2

New Features

  • References librdkafka.redist v1.0.0

Changes

  • Moved API docs from the client classes to their respective interfaces.

  • Update formatting of client API docs so they display well in Visual Studio Code intellisense.

1.0.0-RC1

New Features

  • Added GET subject versions to the cached schema registry client.

  • References librdkafka.redist 1.0.0-RC9

    • Supports Alpine Linux out-of-the-box.

    • Fallback support that excludes security features for most Linux distributions previously unsupported out-of-the-box.

    • Fixed a dependency issue on MacOS.

Changes

  • A new rebalance API. ISerializer<T>, IAsyncSerializer<T>, IDeserializer<T>, and IAsyncDeserializer<T>

    • SetRebalanceHandler has been split into SetPartitionsAssignedHandler and SetPartitionsRevokedHandler.

    • Calling of Assign or Unassign in these handlers is prohibited.

    • Partitions to read from or start offsets can be optionally specified manually via the return value from these handlers.

  • The Message.PersistenceStatus property name has changed to Message.Status.

  • Moved the GetWatermarkOffsets and QueryWatermarkOffsets methods from admin client to consumer.

  • Context is now provided to serdes via a SerializationContext class instance.

Fixes

  • Corrected an error in the rd_kafka_event_type method signature which was causing incompatibility with mono.

  • Audited exception use across the library and made changes in various places where appropriate.

  • Removed unused CancellationToken parameters.

  • Builder classes now return interfaces, not concrete classes.

  • Removed the dependency on CompilerServices.Unsafe which was causing ProduceAsync to hang in some scenarios.

  • Fixed a deadlock-on-dispose issue in AdminClient.

  • Made Producer.ProduceAsync async.

1.0.0-beta3

New Features

  • Revamped producer and consumer serialization functionality.

    • There are now two types of serializer and deserializer: ISerializer<T>, IAsyncSerializer<T>, IDeserializer<T>, and IAsyncDeserializer<T>.

      • ISerializer<T> and IDeserializer<T> are appropriate for most use cases.

      • IAsyncSerializer<T> and IAsyncDeserializer<T> are async friendly, but less performant, returning Task.

    • Changed the name of Confluent.Kafka.Avro to Confluent.SchemaRegistry.Serdes.

    • Added an example demonstrating working with Protobuf serialized data.

  • The Consumer, Producer, and AdminClient classes are now constructed using builder classes.

    • This is more verbose, but provides a sufficiently flexible and future proof API for specifying serdes and other configuration information.

    • All event classes on the client classes have been replaced with corresponding Set...Handler methods on the builder classes.

      • This allows or enforces handlers are set on librdkafka initialization, which is important for some handlers, particularly the log handler.

      • event allows for more than one handler to be set, but this is often not appropriate, such as OnPartitionsAssigned, and never necessary. This is no longer possible.

      • event is also not async friendly. Handlers can’t return Task. The Set…Handler approach can be extend in such a way that it is.

  • Avro serdes no longer make blocking calls to ICachedSchemaRegistryClient. Everything is await.

    Note

    The Consumer implementation still calls async deserializers synchronously because the Consumer API is still otherwise fully synchronous.

  • Reference librdkafka.redist 1.0.0-RC7

    • Notable features include idempotent producer, sparse connections, max.poll.interval.ms for KIP-62.

    Note

    End of partition notification is now disabled by default. Enable it using the EnablePartitionEof configuration property.

  • Removed the Consumer.OnPartitionEOF event in favor notifying of partition eof via ConsumeResult.IsPartitionEOF.

  • Removed ErrorEvent class and added IsFatal to Error class.

    • The IsFatal flag is now set appropriately for all errors. This was previously always set to false.

  • Added PersistenceStatus property to DeliveryResult, which provides information on the persistence status of the message.

Fixes

  • Added Close method to IConsumer interface.

  • Changed the name of ProduceException.DeliveryReport to ProduceException.DeliveryResult.

  • Fixed bug where enum config property couldn’t be read after setting it.

  • Added SchemaRegistryBasicAuthCredentialsSource back into SchemaRegistryConfig.

  • Fixed Schema Registry client failover connection issue.

  • Improvements to librdkafka dependency discovery.

1.0.0-beta2

New Features

  • References librdkafka 1.0.0-PRE1. Highlights:

    • Idempotent producer.

    • Sparse connections, so broker connections are only held open when in use.

Enhancements and Fixes

  • Fixed a memory leak in ProduceAsync #640, which was a regression from 0.11.x.

1.0.0-beta

New Features

  • Added an AdminClient, providing CreateTopics, DeleteTopics, CreatePartitions, DescribeConfigs and AlterConfigs.

  • Can now produce and consume message headers.

  • Can now produce user defined timestamps.

  • Added IClient, IProducer and IConsumer interfaces, which is useful for dependency injection and mocking when writing tests.

  • Added a Handle property to all clients classes:

    • Producers can utilize the underlying librdkafka handle from other Producers, which replaces the 0.11.x GetSerializingProducer method on the Producer class.

    • AdminClient can utilize the underlying librdkafka handle from AdminClient, Producer, or Consumer.

  • IDeserializer now exposes message data via ReadOnlySpan<byte>, directly referencing librdkafka allocated memory. This results in a considerable performance increase of up to 2x and reduced memory.

  • Most blocking operations now accept a CancellationToken parameter.

  • .NET Specific configuration parameters are all specified and documented in the ConfigPropertyNames class.

Major Breaking API Changes

  • The Message class has been re-purposed and now encapsulates specifically the message payload only.

    • ProduceAsync and BeginProduce now return a DeliveryReport object and Consumer.Consume returns a ConsumeResult object.

  • The methods used to produce messages have changed:

    • Methods that accept a callback are now named BeginProduce instead of ProduceAsync, analogous to similar methods in the standard library.

    • Callbacks are now specified as Action<DeliveryReportResult<TKey, TValue>> delegates, not implementations of IDeliveryHandler.

    • The IDeliveryHandler interface has been deprecated.

    • There are two variants of ProduceAsync and BeginProduce, the first takes a topic name and a Message. The second takes a TopicPartition and a message.

      • In other words, when producing, there is now clear separation between what is produced and where it is produced to.

    • The new API is more future proof.

    • ProduceAsync now calls SetException instead of SetResult on the returned Task, making error checking more convenient and less prone to developer mistakes.

  • The feature to block ProduceAsync calls on local queue full has been removed, which results in Local_QueueFull error. This should be implemented at the application layer if required.

  • The non-serializing Producer and non-deserializing Consumer types have been removed, which uses generic types with byte[] instead, considerably reducing API surface area.

  • The ISerializingProducer interface has been removed - you can achieve the same functionality by sharing client handles instead.

  • The Consumer.Poll method and corresponding OnMessage event have been removed. You should use Consumer.Consume instead.

  • The Consumer.OnConsumeError has been removed. Consume errors are now exposed via a ConsumeException.

  • The Consumer.Consume method now returns a ConsumeResult object rather than a Message from an out parameter.

  • CommitAsync has been removed. Use Commit instead.

  • Commit errors are reported via an exception and method return values have correspondingly changed.

  • ListGroups, ListGroup, GetWatermarkOffsets, QueryWatermarkOffsets, and GetMetadata have been removed from Producer and Consumer and exposed only from AdminClient.

  • Added Consumer.Close.

  • Various methods that formerly returned TopicPartitionOffsetError or TopicPartitionError now return TopicPartitionOffset or TopicPartition and throw an exception in case of error with a Result property of type TopicPartitionOffsetError or TopicPartitionError.

Minor Breaking API Changes

  • Removed cast from Error to bool.

  • Consumer.OffsetsForTimes if provided an empty collection will return an empty collection, instead of throwing an exception.

  • manualPoll argument has been removed from the Producer constructor and is now a configuration option.

  • enableDeliveryReports argument has been removed from the Producer constructor and is now a configuration option.

  • Removed methods with a millisecondsTimeout parameter, always preferring a TimeSpan parameter.

  • Added Consumer.Consume variants with a CancellationToken parameter.

  • Added A Producer.Flush method variant without a timeout parameter but with a CancellationToken parameter that is observed.

  • Added the SyslogLevel enumeration, which is used by the log handler delegate.

Minor Enhancements and Fixes

  • When delivery reports are disabled, ProduceAsync will return any completed Task rather than a Task that will never complete.

  • Avro serializers and deserializer now handle null values.

  • Examples upgraded to target 2.1.

  • Changed name of HasError to IsError

  • Configuration options have been added to allow fine-grained control over of marshalling of values to and from librdkafka for high performance usage.

    • Headers, message keys and values, timestamps, and the topic name.

  • Improved XML API documentation.

0.11.5 and previous

Refer to the release notes