Kafka Go Client Changelog

v2.13.3

This is a maintenance release:

Fixes

  • Allow compatibility with librdkafka 2.13.0+.

v2.13.2

This is a maintenance release:

Fixes

  • Allow Azure KMS key version to be optional.

  • schemaregistry/rules/encryption: fix dropped errors.

  • Update frozen semantics for associations in mock Schema Registry client.

confluent-kafka-go is based on librdkafka v2.13.2, see the librdkafka v2.13.2 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.13.0

This is a feature release:

Enhancements

  • Add support for forwarding librdkafka log events to a Go channel in AdminClient.

  • Add support for FIPS 140-3 on Schema Registry operations.

  • Add Schema Registry Associations support with mock client implementation.

  • Add associations example.

  • Handle evolution during field transformation.

  • Add Accept-Version header to Schema Registry requests.

Fixes

  • Fix support for wrapped Avro unions.

  • Fix Avro bytes serialization.

confluent-kafka-go is based on librdkafka v2.13.0, see the librdkafka v2.13.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.12.0

This is a feature release:

KIP-848 General Availability

Starting with confluent-kafka-go 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Refer to the following migration guide for moving from classic to consumer protocol.

Note

The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. group.protocol configuration property dictates whether to use the new consumer protocol or older classic protocol. It defaults to classic if not provided.

Enhancements

  • OAuth OIDC method example for Kafka metadata based authentication with an Azure IMDS endpoint using an attached managed identity as principal.

confluent-kafka-go is based on librdkafka v2.12.0, see the librdkafka v2.12.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.11.1

This is a maintenance release:

confluent-kafka-go is based on librdkafka v2.11.1, see the librdkafka v2.11.1 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.11.0

This is a feature release:

confluent-kafka-go is based on librdkafka v2.11.0, see the librdkafka v2.11.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.10.1

This is a maintenance release:

confluent-kafka-go is based on librdkafka v2.10.1, see the librdkafka v2.10.1 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

Enhancements

  • Support for schema id in header.

  • Use hamba encoder schema function if available.

Fixes

  • Prevent panic when Schema Registry is configured without an auth provider.

  • Fix NPE in CSFLE executor.

v2.10.0

This is a feature release:

confluent-kafka-go is based on librdkafka v2.10.0, see the librdkafka v2.10.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

There was no v2.9.0 release.

Enhancements

  • KIP-848: Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs.

  • KIP-848: DescribeConsumerGroups() now supports KIP-848 introduced consumer groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a classic or consumer group. Target assignment is only valid for the consumer protocol and it defaults to NULL.

v2.8.0

This is a feature release:

confluent-kafka-go is based on librdkafka v2.8.0, see the librdkafka v2.8.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

There was no v2.7.0 release.

Enhancements

  • Add retry logic to RestService for Schema Registry.

  • Add ability to override disable flag and actions on a rule.

  • Add AWS AssumeRole support to AWS KMS.

  • Add support for comma-separated URLs.

Fixes

  • Add deleted query param when looking up schema under subject.

  • For Protobuf, copy the result into the target if necessary.

  • Don’t register maps as native types in CEL rules.

  • Lookup local secret from env if needed.

  • Ensure different key ids use different client instances.

  • Fix handling of nested maps for Avro and JSON Schema.

  • Ensure algorithm query param is passed for CSFLE.

v2.6.1

This is a maintenance release:

confluent-kafka-go is based on librdkafka v2.6.1, see the librdkafka v2.6.1 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

Fixes

  • Minor optimization to reduce schema ID lookups.

  • Support transforming maps in Data Contract rules.

  • Minor optimization to not derive schema when using existing schema.

v2.6.0

This is a feature release:

  • KIP-848 EA: Admin API for listing consumer groups now has an optional filter to return only groups of given types.

  • KIP-460: Admin Leader Election RPC.

confluent-kafka-go is based on librdkafka v2.6.0, see the librdkafka v2.6.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.5.4

v2.5.4 is a maintenance release with the following fixes and enhancements:

Fixes

  • Upgrade azidentity library to 1.6.0.

  • Upgrade vault library to 1.15.0.

  • Handle primitives in avrov2 library.

  • Allow RuleRegistry to be set in serdes.

confluent-kafka-go is based on librdkafka v2.5.3, see the librdkafka v2.5.3 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.5.3

v2.5.3 is a maintenance release with the following fixes and enhancements:

Fixes

  • Properly handle 409 conflicts when registering KEKs/DEKs.

  • Run rule actions when a rule condition fails.

  • Include deleted schemas when getting schemas by subject and version.

  • Handle signed ints when transforming Protobuf payloads.

  • Use correct URL when calling DEK Registry to retrieve a DEK by version.

  • Upgrade hamba Avro library to 2.24.0.

  • Perform Avro schema resolution in the Avro deserializer if necessary.

  • Add some missing APIs to the Schema Registry client.

confluent-kafka-go is based on librdkafka v2.5.3, see the librdkafka v2.5.3 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

There were no v2.5.1 or v2.5.2 releases.

v2.5.0

Warning

This version has introduced a regression in which an assert is triggered during PushTelemetry call. This happens when no metric is matched on the client side among those requested by broker subscription.

You won’t face any problem if:

  • Broker doesn’t support KIP-714.

  • KIP-714 feature is disabled on the broker side.

  • KIP-714 feature is disabled on the client side. This is enabled by default. Set configuration enable.metrics.push to false.

  • If KIP-714 is enabled on the broker side and there is no subscription configured there.

  • If KIP-714 is enabled on the broker side with subscriptions that match the KIP-714 metrics defined on the client.

Having said this, we strongly recommend using v2.5.3 and above to not face this regression at all.

This is a feature release.

  • Adds an AdminAPI DeleteRecords().

  • Add support for metadata and ruleSet in the Schema Registry client, which together support data contracts.

  • Add a new Avro package “avrov2” which uses the Avro hamba library. The old package “avro” uses Avro libraries which are no longer maintained and should not be used in new code.

  • Move rest service for Schema Registry client into internal package for use by both the Schema Registry client and the DEK Registry client.

  • Add support for CSFLE (client-side field-level encryption) for AWS, Azure, Google Cloud, and HashiCorp Vault. See the encryption examples in the examples directory.

  • Add support for CEL, CEL_FIELD, and JSONata rules.

Fixes

  • Windows builds are linked to libssp in addition to other libraries, since the bundled zstd and zlib are built with -fstack-protector, and not linking causes build failures. Happening since 2.0.0.

confluent-kafka-go is based on librdkafka v2.5.0, see the librdkafka v2.5.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.4.0

This is a feature release.

  • KIP-848: Integration tests running with the new consumer group protocol. The feature is an Early Access: not production ready.

  • Allow custom http.Client to be passed to Schema Registry client.

  • Cache schemas when setting UseLatestVersion=true.

  • Add CacheLatestTtlSecs to allow expiration of latest schemas.

  • Cache parsed file descriptors in Protobuf deserializer.

  • Add CacheSchemas option to Protobuf serializer.

  • Add tags field to Confluent metadata.

Fixes

  • The version of Go in go.mod has been changed from 1.17 to 1.21. This is necessary to update test dependencies with security vulnerabilities. Code using the library will still work with Go 1.17.

  • Fix JSON validation during serialization.

  • Fix counter in mock Schema Registry client.

confluent-kafka-go is based on librdkafka v2.4.0, see the librdkafka v2.4.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.3.0

This is a feature release.

  • Adds support for AdminAPI DescribeCluster() and DescribeTopics().

  • KIP-430: Return authorized operations in Describe Responses.

  • Adds Rack to the Node type, so AdminAPI calls can expose racks for brokers (currently, all Describe Responses).

  • KIP-396: completed the implementation with the addition of ListOffsets.

  • Adds cache for Schema Registry client’s GetSchemaMetadata.

  • MockCluster can now be shutdown and started again to test broker availability problems.

  • Adds CreateTopic method to the MockCluster.

  • Honor HTTPS_PROXY environment variable, if set, for the Schema Registry client.

  • KIP-516: Partial support of topic identifiers. Topic identifiers in metadata response are available through the new DescribeTopics function.

Fixes

  • Fixes a bug in the mock Schema Registry client where the wrong ID was being returned for pre-registered schema.

  • The minimum version of Go supported has been changed from 1.16 to 1.17.

  • Fixes an issue where testing was being imported by a non-test file, testhelpers.go.

  • Fixes the optional Coordinator field in ConsumerGroupDescription in case it’s not known. It now contains a Node with ID -1 in that case. Avoids a C segmentation fault.

  • Fixes an issue with Producer.Flush. It was waiting for queue.buffering.max.ms while flushing.

  • Fixes an issue where consumer methods would not be allowed to run while the consumer was closing, and during the final partition revoke.

confluent-kafka-go is based on librdkafka v2.3.0, see the librdkafka v2.3.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.2.0

This is a feature release.

  • KIP-339: IncrementalAlterConfigs API.

  • KIP-554: User SASL/SCRAM credentials alteration and description.

Fixes

  • Fixes a nil pointer bug in the Protobuf Serializer.Serialize(), caused due to an unchecked error.

  • Fixes incorrect Protobuf FileDescriptor references.

  • Allow fetching all partition offsets for a consumer group by passing a nil slice in AdminClient.ListConsumerGroupOffsets, when earlier it was not processing that correctly.

  • Deprecate m.LeaderEpoch in favor of m.TopicPartition.LeaderEpoch.

confluent-kafka-go is based on librdkafka v2.2.0, see the librdkafka v2.2.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.1.1

This is a maintenance release.

It is strongly recommended to update to v2.1.1 if v2.1.0 is being used, as it fixes a critical issue in the consumer.

confluent-kafka-go is based on librdkafka v2.1.1, see the librdkafka v2.1.1 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.1.0

This is a feature release:

  • Added Consumer SeekPartitions() method to seek multiple partitions at once and deprecated Seek().

  • KIP-320: add offset leader epoch to the TopicPartition and Message structs.

  • The minimum version of Go supported has been changed from 1.14 to 1.16.

  • Add validation on the Producer, the Consumer and the AdminClient to prevent panic when they are used after close.

  • Fix bug causing Schema Registry URL with existing path to not be parsed correctly.

  • Support for Offset types on Offset.Set().

  • Added example for using rebalance callback with manual commit.

confluent-kafka-go is based on librdkafka v2.1.0, see the librdkafka v2.1.0 release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v2.0.2

This is a feature release:

  • Added SetSaslCredentials. This new method (on the Producer, Consumer, and AdminClient) allows modifying the stored SASL PLAIN/SCRAM credentials that will be used for subsequent (new) connections to a broker.

  • Channel-based producer (Producer ProduceChannel()) and channel-based consumer (Consumer Events()) are deprecated.

  • Added IsTimeout() on Error type. This is a convenience method that checks if the error is due to a timeout.

  • The timeout parameter on Seek() is now ignored and an infinite timeout is used, the method will block until the fetcher state is updated (typically within microseconds).

  • The minimum version of Go supported has been changed from 1.11 to 1.14.

  • KIP-222: Add Consumer Group operations to Admin API.

  • KIP-518: Allow listing consumer groups per state.

  • KIP-396: Partially implemented: support for AlterConsumerGroupOffsets.

  • As result of the above KIPs, added.

    • ListConsumerGroups Admin operation. Supports listing by state.

    • DescribeConsumerGroups Admin operation. Supports multiple groups.

    • DeleteConsumerGroups Admin operation. Supports multiple groups.

    • ListConsumerGroupOffsets Admin operation. Currently, only supports one group with multiple partitions. Supports the requireStable option.

    • AlterConsumerGroupOffsets Admin operation. Currently, only supports one group with multiple offsets.

  • Added SetRoundtripDuration to the mock broker for setting RTT delay for a given mock broker.

  • Built-in support for Linux/ arm64.

Fixes

  • The SpecificDeserializer.Deserialize method was not returning its result result correctly, and was hence unusable. The return has been fixed.

  • The schema ID to use during serialization, specified in SerializerConfig, was ignored. It is now used as expected.

  • Creating a new Schema Registry client with an SSL CA Certificate led to a panic. This was due to a nil pointer, fixed with proper initialization.

Upgrade considerations

  • OpenSSL 3.0.x upgrade in librdkafka requires a major version bump, as some legacy ciphers need to be explicitly configured to continue working, but it is highly recommended not to use them. The rest of the API remains backward compatible, see the librdkafka release notes below for details.

  • As required by the Go module system, a suffix with the new major version has been added to the module name, and package imports must reflect this change.

confluent-kafka-go is based on librdkafka v2.0.2, see the librdkafka v2.0.0 release notes and later ones for a complete list of changes, enhancements, fixes, and upgrade considerations.

Note

There were no confluent-kafka-go v2.0.0 or v2.0.1 releases.

v1.9.2

This is a maintenance release:

  • Bundles librdkafka v1.9.2.

  • Example for using go clients with AWS Lambda.

  • OAUTHBEARER unsecured producer, consumer and OIDC examples.

confluent-kafka-go is based on librdkafka v1.9.2, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v1.9.1

This is a feature release:

  • Schema Registry support for Avro Generic and Specific, Protocol Buffers and JSON Schema.

  • Built-in support for macOS M1 / arm64.

confluent-kafka-go is based on librdkafka v1.9.1, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v1.9.0

This is a feature release:

  • OAUTHBEARER OIDC support.

  • KIP-140 Admin API ACL support.

  • Added MockCluster for functional testing of applications without the need for a real Kafka cluster. See examples/mock_cluster.

Fixes

  • Fix Rebalance events behavior for static membership.

  • Fix consumer close taking 10 seconds when there’s no rebalance needed.

confluent-kafka-go is based on librdkafka v1.9.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v1.8.2

This is a maintenance release:

  • Bundles librdkafka v1.8.2.

  • Check termination channel while reading delivery reports.

  • Added convenience method Consumer.StoreMessage().

confluent-kafka-go is based on librdkafka v1.8.2, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

Note

There were no confluent-kafka-go v1.8.0 and v1.8.1 releases.

v1.7.0

Enhancements

  • Experimental Windows support.

  • The produced message headers are now available in the delivery report Message.Headers if the Producer’s go.delivery.report.fields configuration property is set to include headers, such as "go.delivery.report.fields": "key,value,headers". This comes at a performance cost and are thus disabled by default.

Fixes

  • AdminClient.CreateTopics() previously did not accept default value(-1) of ReplicationFactor without specifying an explicit ReplicaAssignment, this is now fixed.

confluent-kafka-go is based on librdkafka v1.7.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

v1.6.1

v1.6.1 is a feature release:

  • KIP-429: Incremental consumer rebalancing. See cooperative_consumer_example.go for an example how to use the new incremental rebalancing consumer.

  • KIP-480: Sticky producer partitioner. Increase throughput and decrease latency by sticking to a single random partition for some time.

  • KIP-447: Scalable transactional producer. A single transaction producer can now be used for multiple input partitions.

confluent-kafka-go is based on and bundles librdkafka v1.6.1, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.

Enhancements

  • go.delivery.report.fields=all,key,value,none can now be used to avoid copying message key and/or value to the delivery report, improving performance in high-throughput applications.

Fixes

  • Consumer.Close() previously did not trigger the final RevokePartitions callback, this is now fixed.

v1.5.2

v1.5.2 is a maintenance release with the following fixes and enhancements:

  • Bundles librdkafka v1.5.2 - see release notes for all enhancements and fixes.

  • Documentation fixes.

confluent-kafka-go is based on librdkafka v1.5.2, see the librdkafka release notes for a complete list of changes, enhancements, fixes, and upgrade considerations.