Important
You are viewing documentation for an older version of Confluent Platform. For the latest, click here.
Release Notes¶
Confluent Platform 4.1.3¶
This is a bugfix release of Confluent Platform that provides you with Apache Kafka 1.1.1, the latest stable version of Kafka and additional bug fixes.
You are encouraged to upgrade to Confluent Platform 4.1.3 as it includes important bug fixes. The technical details of this release are summarized below.
Highlights¶
Enterprise Features¶
Confluent Rebalancer¶
- Relax rack constraint when already violated.
Control Center Front-End¶
- Upgrade redux-search
- Provide an end date that does not crash the ui
Kafka Broker Plugins¶
- Add authenticated connections and authentication rate metrics per tenant
- Block altering non-topic configs to fix AlterConfig policy not updated for dynamic broker configs
Kafka Connect JMS¶
- Backport changes for ‘Unsafe use of javax.jms.Session in JMS Connector’ fix to 4.1.x and update corresponding docs
- Remove the unused HornetQ-specific connector
Kafka Connect Replicator¶
- Corrected how Replicator instantiates and configures source converters and header converters
Community Features¶
Apache Kafka 1.1.1-cp2¶
- PR-4978 - KAFKA-6878: NPE when querying global state store not in READY state (#4978)
- PR-6298 - KAFKA-7959: Delete leader epoch cache files with old message format versions (#6298)
- PR-6220 - KAFKA-7401: Fix inconsistent range exception on segment recovery (#6220)
- PR-6253 - KAFKA-7897: Do not write epoch start offset for older message format versions (#6253)
- PR-5989 - KAFKA-7693: Fix SequenceNumber overflow in producer (#5989)
- PR-5990 - KAFKA-7692: Fix ProducerStateManager SequenceNumber overflow (#5990)
- PR-5946 - KAFKA-7443: OffsetOutOfRangeException in restoring state store from changelog topic when start offset of local checkpoint is smaller than that of changelog topic (#5946)
- PR-6006 - KAFKA-6388: Recover from rolling an empty segment that already exists (#6006)
- PR-5981 - KAFKA-7660: fix streams and Metrics memory leaks (#5981)
- PR-5993 - KAFKA-7678: Avoid NPE when closing the RecordCollector (#5993)
- PR-5959 - KAFKA-7671: Stream-Global Table join should not reset repartition flag (#5959)
- PR-5767 - KAFKA-7192: Wipe out state store if EOS is turned on and checkpoint file does not exist (#5767)
- PR-5875 - KAFKA-7576: Fix shutdown of replica fetcher threads (#5875)
- PR-5833 - KAFKA-7534: Error in flush calling close may prevent underlying store from closing (#5833)
- PR-5515 - KAFKA-7299: Batch LeaderAndIsr requests for AutoLeaderRebalance (#5515)
- PR-5749 - KAFKA-7415: Persist leader epoch and start offset on becoming a leader (#5678) (#5749)
- PR-5720 - KAFKA-6914: Set parent classloader of DelegatingClassLoader same as the worker’s (#5720)
- PR-5727 - KAFKA-7467: NoSuchElementException is raised because controlBatch is empty (#5727)
- PR-5713 - KAFKA-7454: Use lazy allocation for SslTransportLayer buffers and null them on close (#5713)
- PR-5712 - KAFKA-7453: Expire registered channels not selected within idle timeout (#5712)
- PR-5654 - KAFKA-7414: Out of range errors should never be fatal for follower (#5654)
- PR-5680 - KAFKA-7216: Ignore unknown ResourceTypes while loading acl cache (#5680)
- PR-5495 - KAFKA-7280: Synchronize consumer fetch request/response handling (#5495)
- PR-5627 - KAFKA-7044: Fix Fetcher.fetchOffsetsByTimes and NPE in describe consumer group (#5627)
- PR-5585 - KAFKA-7287: Set open ACL for old consumer znode path (#5503) (#5585)
- PR-5557 - KAFKA-7128: Follower has to catch up to offset within current leader epoch to join ISR (#5557)
- PR-5436 - KAFKA-7164: Follower should truncate after every missed leader epoch change (#5436)
- PR-4633 - KAFKA-5891: Proper handling of LogicalTypes in Cast (#4633)
- PR-5509 - KAFKA-7119: Handle transient Kerberos errors on server side (#5509)
- PR-5501 - KAFKA-7285: Create new producer on each rebalance if EOS enabled (#5501)
- PR-5513 - KAFKA-7284: streams should unwrap fenced exception (#5513)
- PR-5487 - KAFKA-7119: Handle transient Kerberos errors as non-fatal exceptions (#5487)
- PR-5484 - KAFKA-7261: Record 1.0 for total metric when Count stat is used for rate (#5484)
- PR-3907 - KAFKA-4950: Fix ConcurrentModificationException on assigned-partitions metric update (#3907)
- PR-5478 - KAFKA-7255: Fix timing issue with create/update in SimpleAclAuthorizer (#5478)
Schema Registry¶
- PR-1008 - CC-3740: Add support for ignoring metadata during toConnectMetadata
- PR-1003 - CC-3732: Prevent SR from registering duplicate IDs
- PR-856 - https://github.com/confluentinc/schema-registry/issues/799
Kafka Rest¶
Kafka Connect HDFS¶
Kafka Connect Storage Coud¶
Confluent Platform 4.1.2¶
This is a bugfix release of the Confluent Platform that provides you with Apache Kafka 1.1.1, the latest stable version of Kafka and additional bug fixes.
You are encouraged to upgrade to Confluent Platform 4.1.2 as it includes important bug fixes. The technical details of this release are summarized below.
Highlights¶
Open Source Features¶
Apache Kafka 1.1.1-cp1¶
- PR-4928 - KAFKA-6292; Improve FileLogInputStream batch position checks to avoid type overflow (#4928)
- PR-5341 - KAFKA-7136: Avoid deadlocks in synchronized metrics reporters (#5341)
- PR-5305 - KAFKA-7104: More consistent leader’s state in fetch response (#5305)
- PR-5259 - KAFKA-7082: Concurrent create topics may throw NodeExistsException (#5259)
- PR-5237 - KAFKA-7012: Don’t process SSL channels without data to process (#5237)
- PR-5229 - KAFKA-6975; Fix replica fetching from non-batch-aligned log start offset (#5133) (#5229)
- PR-5219 - KAFKA-6711: GlobalStateManagerImpl should not write offsets of in-memory stores in checkpoint file (#5219)
- PR-5207 - KAFKA-7021: checkpoint offsets from committed (#5207)
- PR-5187 - KAFKA-6860: Fix NPE in Kafka Streams with EOS enabled (#5187)
- PR-4912 - KAFKA-6749: Fixed TopologyTestDriver to process stream processing guarantee as exactly once (#4912)
- PR-4900 - KAFKA-6782: solved the bug of restoration of aborted messages for GlobalStateStore and KGlobalTable (#4900)
- PR-5105 - KAFKA-6906: Fixed to commit transactions if data is produced via wall clock punctuation (#5105)
- PR-4834 - KAFKA-6760: Fix response logging in the Controller (#4834)
- PR-5108 - KAFKA-6925: fix parentSensors memory leak (#5108)
- PR-5074 - KAFKA-6937: In-sync replica delayed during fetch if replica throttle is exceeded (#5074)
- PR-5036 - KAFKA-6917; Process txn completion asynchronously to avoid deadlock (#5036)
- PR-4999 - KAFKA-6893; Create processors before starting acceptor in SocketServer (#4999)
- PR-4985 - KAFKA-6870 Concurrency conflicts in SampledStat (#4985)
- PR-4869 - KAFKA-6765: Handle exception while reading throttle metric value in test (#4869)
- PR-4978 - KAFKA-6878: NPE when querying global state store not in READY state (#4978)
Confluent CLI¶
JDBC Source Connector¶
JDBC Sink Connector¶
Enterprise Features¶
Replicator¶
- PR-113 - CC-2099: Extend replicator binary to run from source repo and system tests
- PR-114 - CC-2106 Don’t prefix interceptors with src.kafka and dest.kafka
- PR-108 - CC-1263: refresh Replicator quickstart configuration
- PR-110 - CC-2061: Handle NotLeaderException gracefully.
- PR-101 - CC-1458: Update configuration options
Confluent Platform 4.1.1¶
This is a bugfix release of the Confluent Platform that provides Confluent users with Apache Kafka 1.1.0, the latest stable version of Kafka and additional bug fixes.
Users are encouraged to upgrade to Confluent Platform 4.1.1 as it includes important bug fixes. The technical details of this release are summarized below.
Highlights¶
Enterprise Features¶
Confluent Control Center¶
- Fixed race condition for expert mode topic creation.
- Fixed chart rendering bug causing UI to blank out.
- Made topic name field freeform in trigger create form.
- Hide no data modal for Streams Monitoring when data is received.
- Made connector status requests more resilient.
- Set X-Frame-Options: DENY to prevent clickjacking.
- More robust time range checks for metric queries.
JDBC Source Connector¶
- Added configuration option to enable extended handling of
NUMERICdata type. - JDBC URL validation is now more flexible.
- Missing columns are discovered in a case insensitive way.
Confluent CLI¶
- Fixed issue with starting and stopping individual services.
Open Source Features¶
Apache Kafka 1.1.1-cp1¶
- KAFKA-6879: Controller deadlock following session expiration
- KAFKA-6857: Leader should reply with undefined offset if undefined leader epoch requested
- KAFKA-6772: Load credentials from ZooKeeper before accepting connections
- KAFKA-6854: Handle batches deleted during cleaning of logs with transactions
- KAFKA-6752: Unclean leader election metric no longer working
- KAFKA-6853: ResponseMetadata calculates latency incorrectly (and therefore
ZooKeeperRequestLatencyMsis incorrect) - KAFKA-6855: Kafka fails to start with Java 10 due to faulty Java version detection
Confluent Platform 4.1.0¶
This is a major release of the Confluent Platform that provides Confluent users with Apache Kafka 1.1.0, the latest stable version of Kafka.
Confluent Platform users are encouraged to upgrade to 4.1.0. The technical details of this release are summarized below.
Highlights¶
Enterprise Features¶
Control Center¶
This release includes an updated Control Center with several new capabilities. You can now create and edit Kafka topics in multiple Kafka clusters from within Control Center. Simply navigate to the Topics page under Management, select the desired managed Cluster in the dropdown menu at the top-right, and click the Create button. When creating a new topic, you can either use default settings or customize them using predefined availability models, cleanup policies, and more.
A topic search and filtering capability has also been introduced and allows you to hide Apache Kafka/Confluent Platform internal topics from the Control Center UI. You can toggle topic visibility by enabling and disabling the “Show internal topics” checkbox on the System health and Topics pages.
This release also introduces a restricted, read-only user role that prevents the creating or editing of topics, alerts, or connectors. You can restrict access to these features by adding restricted roles to the confluent.controlcenter.auth.restricted.roles parameter.
This release also adds a new alert to warn when Control Center stops receiving data from a Kafka cluster.
For more information, see the documentation.
Replicator¶
This release includes an updated Replicator that does not have to connect to ZooKeeper and can instead connect only to the origin and destination Kafka clusters.
Simply remove the src.zookeeper.* properties from the Replicator connector configuration, or include those properties to use the old behavior of directly connecting to the source (origin) ZooKeeper instance.
The dest.zookeeper.* properties work the same way for the destination ZooKeeper behavior.
Also introduced in this release is a new Replicator executable that is easier to run and requires less knowledge about Kafka Connect. This new executable is self-contained and runs a Replicator process with configuration files defining the producer and consumer properties for the origin and destination Kafka clusters and the replication options. The Replicator process is elastically scalable, so simply start more processes and the replication tasks are immediately distributed across the Replicator cluster. If any Replicator process leaves the cluster for any reason, the tasks that had been assigned to that process will be automatically transferred to the remaining processes in the cluster.
Other than how you configure and run it, the new Replicator executable has all of the same features and capabilities of the Replicator connector, and you can continue to run either.
JMS Connectors¶
This release adds three new connectors for JMS brokers. The general JMS Source Connector can read messages from any JMS-compliant broker queue and write them to a Kafka topic. It uses JNDI-style configuration properties, and you must provide the client libraries before it can be used.
The IBM MQ Source Connector is based upon the general JMS source connector but is configured using IBM MQ specific settings and properties. However, due to licensing restrictions, you must provide the IBM MQ JMS client libraries for this connector.
Finally, the ActiveMQ Source Connector is also based upon the JMS source connector but is configured using ActiveMQ-specific settings and properties. This connector already includes the ActiveMQ client libraries, so you can immediately use this connector.
All three of these connectors require a Confluent license, but can be used for trial purposes for 30 days.
Open Source Features¶
Scalability¶
With Apache Kafka 1.1.0, the Controller supports more partitions on a single cluster. Controlled shutdown and restart with a large number of partitions is significantly faster. Here are some examples:
- If you have a multi-rack Kafka cluster with five nodes and 25000 single-partition topics with two replicas each, the elapsed time for controlled shutdown is reduced from 390 seconds to 3 seconds.
- If you have a five node cluster with 100000 partitions, the amount of time to reload the Controller is reduced from 28 seconds to 14 seconds.
KIP-48 added delegation token based authentication to Kafka brokers to support large number of clients without overloading Kerberos KDCs or other authentication servers.
Reduced Producer Latency¶
KIP-227
provides more efficient replication when the number of partitions is large by introducing incremental fetch requests.
As a result, produce latency is reduced when the producer waits for all in-sync replicas to acknowledge (acks=all)
the request before considering it successful.
Availability and Resilience¶
ZooKeeper session expiration edge cases are fixed, making brokers more resilient.
Operational Improvements¶
Several frequently updated broker configuration options, like SSL keystores, can now be updated dynamically without restarting the broker. For details and the full list of dynamic configs, see KIP-226. The kafka-configs tool has been updated to enable broker configuration options to be altered and described.
KSQL¶
This release includes a production ready version of KSQL. KSQL is the streaming SQL engine for Apache Kafka. It provides a simple and completely interactive SQL interface for stream processing on Kafka. You don’t need to write code in a programming language such as Java or Python. KSQL is open-source (Apache 2.0 licensed), distributed, scalable, reliable, and real-time. It supports a wide range of powerful stream processing operations including aggregations, joins, windowing, sessionization, and much more.
This version of KSQL is not backward compatible with the previous developer preview releases. In particular, you must manually migrate queries running in the older preview releases of KSQL to the
new version by issuing statements like CREATE STREAM and CREATE TABLE again.
To learn about KSQL you can:
Kafka Connect¶
Kafka Connect includes a number of features and improvements. First, the Kafka Connect REST API can be secured with SSL / TLS encryption and SSL / TLS client authentication.
Second, the REST API includes in the response for the worker’s root resource the ID of the Kafka cluster.
Third, the Connect framework now has support for exposing message headers
to connectors and transforms, allowing source connectors to add records to generated source records and sink connectors to read headers.
A new HeaderConverter API is available for customizing how headers are serialized and deserialized, and a new SimpleHeaderConverter (the default) serializes them as string values.
The new Connect Header API makes it possible to get header values in terms of Connect types, and can even cast between primitive types to make it easier for transforms and sink connectors to get header values in the desired type.
Fourth, sink connector configurations can use regular expressions instead of the topic list to identify the topics to be consumed by the connector.
Finally, there are also a number of important fixes and changes in Kafka Connect. Connect now enforces the legal set of characters allowed in connector names, although nearly any character can be URL escaped. Several problems with metrics have been fixed, including the “metric XX already exists” bug (also fixed in 4.0.1). If Connect attempts to create internal topics upon startup and ACLs prevent it from doing so, it does not fail but will let the broker auto-create the topic if so configured.
Elasticsearch Connector¶
The Elasticsearch connector has a number of features and improvements. The connector works with Elasticsearch 2.x, 5.x, and 6.x,
and automatically uses the text type with Elasticsearch 5.x and above rather than the string type
deprecated in ES 5 and removed in ES 6.0.
You can use the new conn.timeout.ms property to set the timeout for establishing connections to Elasticsearch, and
the read.timeout.ms property to set the max time to wait for an Elasticsearch response,
The new behavior.on.null.values property controls how the connector handles records with null values (e.g., tombstone messages), with three options: ignore, delete, and fail (the default).
Finally, the behavior.on.malformed.documents property controls how the connector handles records that are rejected by Elasticsearch due to malformed documents, with three options: ignore, warn, and fail (the default).
HDFS Connector¶
The HDFS connector has a new configuration option to apply a specific Avro compression codec (e.g., Snappy, Deflate, Bzip2) used for files written to HDFS.
The partition.field.name now supports multiple comma-separated fields on which the output should be partitioned.
Also, when using the TimeBasedPartitioner the connector now supports extracting the timestamp from nested fields.
A new storage format is available for writing messages to HDFS as simple text files with a .txt extension, where records are stored one per line.
Also, when the HDFS connector starts and is assigned new topics or topic partitions for which there is no prior output in HDFS, rather than always starting from
the beginning of the topic the connector now defers to the consumer to start based upon the consumer.auto.offset.reset property in the Connect worker config
(which defaults to earliest).
This gives the user more control and flexibility over where the HDFS connector starts.
However, note that the worker’s consumer.auto.offset.reset property applies to all connectors in that worker cluster.
The consumer for the HDFS connector no longer commits offsets, since they are never used and because the HDFS connector always determines the offsets based upon what’s been written to HDFS.
S3 Connector¶
The S3 connector has a new configuration option to apply a specific Avro compression codec (e.g., Snappy, Deflate, Bzip2) used for files written to S3.
The partition.field.name now supports multiple comma-separated fields on which the output should be partitioned.
Also, when using the TimeBasedPartitioner the connector now supports extracting the timestamp from nested fields.
A new storage format is available for writing messages to S3 as simple text files with a .txt extension, where records are stored one per line.
Also, the S3 connector has a new s3.compression.type property that controls the type of compression used for files written to S3, with two options: none and gzip.
The new s3.sse.kms.key.id property sets the name of the AWS Key Management Service (AWS-KMS) key used for server-side encryption of the S3 objects.
The default of no key name results in no encryption. To enable encryption, just set the server-side encryption algorithm
to aws:kms for the s3.ssea.name property and use a valid key name for the s3.sse.kms.key.id property.
Finally, the connector has several other bug fixes and minor improvements.
JDBC Source Connector¶
The JDBC source connector has a number of small improvements and bug fixes, including the proper closing of result sets and statements during shutdown and getting the current timestamp correctly on IBM DB2.
JDBC Sink Connector¶
The JDBC sink connector has a number of small improvements, including the logging of actual SQL statements.
Confluent clients¶
Confluent provides high quality clients for the most popular non-Java languages, these clients are all based on the underlying librdkafka C/C++ implementation which provides high performance and stability. The non-Java clients follow their own versioning scheme slightly trailing the Apache Kafka releases. The current clients release is v0.11.4.
Highlights from v0.11.4:
* Avro serializers and deserializers and Schema Registry integration added to the .NET client.
* The Python client on PyPi is now prebuilt and packaged with the required dependencies (librdkafka, OpenSSL, etc). It is no longer necessary to install librdkafka, et.al, manually. Install with pip install confluent_kafka.
* Message Headers support added to the C/C++, Python and Go clients.
* Java compatible Murmur2 partitioner added to all clients. Set the partitioner configuration property to murmur2_random
* Improved producer retry handling in all clients.
* Many enhancements and bug fixes.
See the client release notes for more information: * confluent-kafka-dotnet * confluent-kafka-go * confluent-kafka-python * librdkafka C/C++
Confluent CLI¶
Confluent CLI adds two additional services: ksql-server available in Confluent Enterprise and Confluent Open Source
and control-center present in Confluent Enterprise. It also adds confluent version, a command that
returns the version and flavor of Confluent Platform as well as the version of an individual service.
Kafka Streams: New Test Utility Artifact¶
A new artifact named kafka-streams-test-utils has been added to help users easily write unit tests for their streams applications’ business logic.
For more details, please read KIP-247.
Kafka Streams: Embedded Admin Client Configuration¶
Users can now configure parameters for the embedded admin client created by Kafka Streams library, similar to the other embedded clients such as producer and consumer.
This configuration can be provided via StreamsConfig with the prefix admin. or via the function StreamsConfig#adminClientPrefix(String) to distinguish from other configurations
of producer and consumer clients. For more details, please read the Kafka Streams Upgrade Guide.
Apache Kafka 1.1.0-cp1¶
KIP 113
enables data balancing with JBOD by adding support for replica movement between log directories.
Administrators can now specify the log directory that a replica should be moved to by using the kafka-reassignment-partitions
tool. You can use the new kafka-log-dirs tool to get the list of replicas per log directory on a broker.
25 new features / config changes has been added along with over 160 bug fixes and performance improvements. For a full list of changes in this release of Apache Kafka, see the 1.1.0 Release Notes.
Confluent Platform 4.0.1¶
This is a bugfix release of the Confluent Platform that provides Confluent users with Apache Kafka 1.0.1, the latest stable version of Kafka and additional bug fixes.
Users are encouraged to upgrade to Confluent Platform 4.0.1 as it includes important bug fixes. The technical details of this release are summarized below.
Highlights¶
Open Source Features¶
Apache Kafka 1.0.1-cp1¶
- KAFKA-6185: Selector memory leak with high likelihood of OOM in case of down conversion
- KAFKA-6174: Add methods in Options classes to keep binary compatibility with 0.11
- KAFKA-6261: Request logging throws exception if acks=0
- KAFKA-6319: kafka-acls regression for comma characters (and maybe other characters as well)
- KAFKA-6164: ClientQuotaManager threads prevent shutdown when encountering an error loading logs
- KAFKA-6172: Cache lastEntry in TimeIndex to avoid unnecessary disk access
- KAFKA-6175: AbstractIndex should cache index file to avoid unnecessary disk access during resize()
- KAFKA-6210: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version
- KAFKA-6238: Issues with protocol version when applying a rolling upgrade to 1.0.0
- KAFKA-6258: SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data
- KAFKA-6260: AbstractCoordinator not clearly handles NULL Exception
- KAFKA-6289: NetworkClient should not return internal failed api version responses from poll
- KAFKA-6307: mBeanName should be removed before returning from JmxReporter#removeAttribute()
- KAFKA-6529: Broker leaks memory and file descriptors after sudden client disconnects
- KAFKA-6366: StackOverflowError in kafka-coordinator-heartbeat-thread
- KAFKA-6457: Error: NOT_LEADER_FOR_PARTITION leads to NPE
- KAFKA-4767: KafkaProducer is not joining its IO thread properly
- KAFKA-6287: Inconsistent protocol type for empty consumer groups
- KAFKA-6422: When enable trace level log in mirror maker, it will throw null pointer exception and the mirror maker will shutdown
- KAFKA-6748: Allow the punctuate() call to cancel its scheduled tasks
- KAFKA-6252: A metric named ‘XX’ already exists, can’t register another one
- KAFKA-6148: Correct classpath isolation due to ClassCastException
- KAFKA-6288: Broken symlink interrupts scanning the plugin path
- KAFKA-6277: Make loadClass thread-safe for class loaders of Connect plugins
- KAFKA-6467: Enforce layout of dependencies within a Connect plugin to be deterministic
- KAFKA-6504: Connect: Some per-task-metrics not working
- KAFKA-4827: Kafka connect: error with special characters in connector name
- KAFKA-6308: Connect: Struct equals/hashCode method should use Arrays#deep* methods
Confluent Platform 4.0.0¶
This is a major release of the Confluent Platform that provides Confluent users with Apache Kafka 1.0.0, the latest stable version of Kafka.
Confluent Platform users are encouraged to upgrade to CP 4.0.0. The technical details of this release are summarized below.
Highlights¶
Enterprise Features¶
Control Center: Backward Compatibility¶
The 4.0.0 release of Confluent Control Center is backward compatible and fully tested with releases of Confluent Platform 3.2.0 and higher and Apache Kafka releases 0.10.2.0 and higher. We encourage you to install and use Control Center before upgrading Apache Kafka so you can use Control Center to monitor broker health and application availability when performing the rolling upgrade of the system.
Control Center: Improvements¶
- Control Center can now monitor significantly larger clusters. Our internal tests show that we can now successfully monitor Kafka clusters with 1 million partitions, up from 30,000 partitions in previous releases. If you are interested in running more than 10,000 partitions, please contact Confluent at contact@confluent.io for support setting up your Confluent Platform installation.
- Control Center now shows a highly visible notification bar when the Kafka cluster it is monitoring is not available.
- Bug fixes including smaller metric messages, improved stability, handling misconfigured clients and misformated events and large number of UI improvements.
Open Source Features¶
JBOD Disk Failure Handling¶
Apache Kafka can now tolerate disk failures better. With KIP-112, Kafka brokers handle JBOD disk failures more gracefully. A single disk failure in a JBOD broker will not bring the entire broker down; rather, the broker will continue serving any log files that remain on functioning disks.
Metrics Improvements¶
Operating Apache Kafka at scale requires that the system remains observable, and to make that easier, several new metrics have been added to improve health monitoring. See KIP-164, KIP-168, KIP-187 and KIP-188 for details.
Kafka Connect¶
Kafka Connect includes a number of features and improvements. First, Kafka Connect workers now have metrics for connectors, tasks, rebalances, and worker information.
The metrics can be accessed through JMX and with custom metrics reporters, just like with Kafka producers and consumers.
Second, the plugin.path now supports relative symlinks.
Finally, Connect will now use a separate Converter for every task, which allows for more flexibility and performance tuning.
Kafka Connect also includes a number of bug fixes. Connect classloader isolation now correctly finds JDBC drivers when they are included in a plugin. Also, topic creation now works correctly when Connect uses pre-0.11 Kafka clusters by delegating to auto-topic creation. Transformations now properly handle tombstone events with non-null keys and null values. And, Connect will commit the original topic-partition offsets even when transformation mutate the topic-partitions.
Elasticsearch Connector¶
The Elasticsearch connector now has the ability to drop invalid messages when needed.
HDFS Connector¶
The HDFS connector aligns its partitioners and formatters with the S3 connector. In particular, the timestamp partitioners are more capable and can extract the timestamp from the records rather than just using wallclock time to extract a timestamp for the records. Also, the JSON formatter can now be used with the HDFS connector.
Existing connector configurations that use the older partitioners and formatters are still valid and will continue to work. However, the older partitioners and formatters are deprecated and will be removed in a future release, so users are encouraged to begin evaluating the new formatters and partitioners and converting existing connector configurations.
Several fixes are also included and a number of dependent libraries were upgraded. All configurations that specify a class now have recommenders, making it much easier to configure HDFS connectors using C3.
S3 Connector¶
The S3 connector can now use S3 proxies, optionally write records using a provided ACL header, can write records at the root of the S3 bucket, and will retry part uploads when needed.
A new ByteArrayFormat formatter makes it very easy to store the binary form of the records in S3. All configurations that specify a class now have recommenders, making it much easier to
configure S3 connectors using C3. And finally, the connector has several other minor fixes and logging improvements.
JDBC Source Connector¶
The JDBC source connector has a number of small improvements. First, the connector now retries to establish JDBC connections to the database when needed. Second, the JDBC source connector is now able to get the current timestamp when using DB2 UDB on AS/400.
Confluent CLI¶
Adds support for setting ACLs for services, and properly sets the plugin.path when the Connect distributed worker configuration uses a relative path.
Kafka Streams: Extended Punctuation Semantics¶
Applications that use the Processor API in Kafka Streams can now customize their punctuation methods based on either event-time or wall-clock-time, to make the punctuation either data-driven or wall-clock time driven. For more details about this change, please read the Kafka Streams Developer Guide.
Kafka Streams: Deserialization Error Handling¶
Managing records that are corrupted or malformatted is a common scenario in stream processing. You can now customize the respective error handling logic of your applications. For example, your applications can now decide wether to skip over corrupted records. In previous versions, corrupted records triggered deserialization exceptions that would always cause application instances to fail-fast and terminate. Please read the Streams FAQ section on handling corrupted records for more details.
Kafka Streams: State Restoration Listener¶
In this release, Kafka Streams has added a new interface that users can instantiate to be able to monitor their state restoration progress during the application (re-)initialization process. Users can be notified when 1) the restoration process has started, 2) the restoration process has ended, 3) a batch of changelog records has been applied during the restoration process, for each state store. It also allows users to inject their own processing logic during the restoration process for their customized state store engines. For more details about this change, please read this KIP.
Apache Kafka 1.0.0-cp1¶
Authentication error handling has been improved to provide feedback to clients. Previously, some authentication error conditions were indistinguishable from broker failures and were not logged in a clear way. Authentication failures are now handled as non-retriable errors that are logged and propagated to clients.
Apache Kafka now supports Java 9, enabling significantly faster TLS and CRC32C implementations. Over-the-wire encryption will be faster now, which will keep Kafka fast and compute costs low when encryption is enabled.
Important
Java 1.9 is currently not supported in Confluent Platform. Later versions of Confluent Platform will support Java 1.9.
Throughput of idempotent producers (which are also used for exactly-once processing) have been significantly improved by relaxing the restriction on max.in.flight.requests.per.connection being limited to one. Now this can be as large as five, relaxing the constraint on maximum throughput.
29 new features / config changes has been added along with over 250 bug fixes and performance improvements. For a full list of changes in this release of Apache Kafka, see the 1.0.0 Release Notes.
How to Download¶
Confluent Platform is available for download at https://www.confluent.io/download/. See the Installing and Upgrading section for detailed information.
To upgrade Confluent Platform to a newer version, check the Upgrade documentation.
Supported Versions and Interoperability¶
For the supported versions and interoperability of Confluent Platform and its components, see Supported Versions and Interoperability.
Questions?¶
If you have questions regarding this release, feel free to reach out via the |cp| mailing list. Confluent customers are encouraged to contact our support directly.