Changelog

Version 10.0.37

  • PR-807 - CC-30442 Bumped hadoop-common version to 3.4.0.

  • PR-805 - S3 sink connector should handle transformed tombstone records gracefully

  • PR-801 - CC-29764 Add timeout to sink put cycle

  • PR-799 - CC-29287: Write actual record in DLQ instead of projected record

  • PR-793 - CC-29553 Fix connector to work with SSE-S3

  • PR-794 - CC-28527 Close only the revoked topic partitions on close

Version 10.0.36

  • PR-792 - CC-29502: Bumped kakfa-connect-storage-cloud

Version 10.0.35

  • PR-787 - Bumped protobuf-java and common-io version

  • PR-715 - Stopped rotating files on partition change when rotate.interval.ms is set

Version 10.0.34

  • PR-783 - CC-29196 Bumped protobuf-java version

Version 10.0.33

  • PR-774 - CC-26094 Added config to disable writing null records to dlq

Version 10.0.32

  • PR-768 - CC-28258 Bumped dnsjava version to 3.6.1.

  • PR-752 - CC-27029 Bumped kafka-connect-storage-common-parent version to 11.2.15.

Version 10.0.31

  • PR-748 - CC-26925 Added validation for s3.path.style.access.enabled and s3.wan.mode configs

  • PR-745 - CC-24623 Handled Avro SchemaParseException and Parquet InvalidSchemaException according to errors.tolerance

Version 10.0.30

  • PR-741 - CC-26516 CC-26360 Bumped aws version and jackson dependency versions

  • PR-740 - CC-26361 Bumped commons-configuration2 dependency version

Version 10.0.29

  • PR-735 - CC-26046 Pinned zookeeper version to 3.8.4

Version 10.0.28

  • PR-730 - CC-25322 Bumped kafka-connect-storage-common-parent

  • PR-722 - Updated common version to fix record reporting

Version 10.0.27

Version 10.0.26

  • PR-684 - Removed x-amz-server-side-encryption to get SEE-C working

Version 10.0.25

  • PR-680 - CC-22247 | Fixed S3 Sink’s Parquet conversion exception.

Version 10.0.24

  • PR-673 - CC-21917 | Added support for decimal.format config to be passed to json formatter.

Version 10.0.23

  • PR-660 - ESC-581: Pin dependencies to resolve CVEs for Citi escalation

Version 10.0.22

  • PR-659 - [CC-20979] [CC-20329] Updated CVEs (jackson and snappy package)

Version 10.0.21

  • PR-651 - CC-19464 | Added logs to track the count for each of the rotation criteria. This helps debug at runtime the number of rotations that occurred for each criteria.

Version 10.0.20

  • PR-641 - CCMSG-2432 Updated SnakeYaml to address CVE-2022-1471.

  • PR-638 - CCMSG-2416 CCMSG-2417 Updated json-smart and jettison.

  • PR-637 - CC-19528 | Add Integration tests for SR backed types.

Version 10.0.19

No changes

Version 10.0.18

No changes

Version 10.0.17

No changes

Version 10.0.16

No changes

Version 10.0.15

No changes

Version 10.0.14

  • PR-566 - CCMSG-2233: Fixed compression validation for custom format and compression type none

Version 10.0.13

  • PR-564 - Bumped storage common version

  • PR-551 - Bumped S3 SDK to 1.12.268

Version 10.0.12

  • PR-542 - CCMSG-2014: CVE fix - Upgrade storage-common version to remove parquet format CVEs

  • PR-536 - CCMSG-1974: CVE fix - Remove dependency on slf4j-log4j

Version 10.0.11

  • PR-522 - CCMSG-1688: CVE fix for Guava dependency

  • PR-533 - Update parent pom to 11.0.9

Version 10.0.10

  • PR-530 - CCMSG-2050: Added cross validation for compression() and format() class

Version 10.0.9

No changes

Version 10.0.8

  • PR-518 - CCMSG-1577 label change to compression level for s3 sink connector

Version 10.0.7

  • PR-504 <https://github.com/confluentinc/kafka-connect-storage-cloud/pull/504> - CCMSG-1743: Upgraded storage-common to resolve jackson-databind 2.13.2 CVE

Version 10.0.6

  • PR-497 - Fix regions list being one item of a list

  • PR-495 - Don’t log full proxy config and don’t log configuration at info level

  • PR-494 - Generalize S3 record writes with wrapper to possibly convert to RetriableException

  • PR-492 - CCMSG-1481: Retriable exception is possible to be thrown from close()

  • PR-461 - Fix retries for S3 transient errors

  • PR-490 - MINOR: Suppress warning which causes IntelliJ to error after Maven import (#489)

  • PR-485 - CCMSG-1531: Support null items within arrays with Parquet writer

Version 10.0.5

  • PR-473 - CCMSG-1582 Fix bug that generates small files when a scheduled rotation has expired

  • PR-470 - MINOR: Remove guava usage and superfluous template arguments

  • PR-451 - MINOR: Remove toString() call in trace log to resolve a performance issue

Version 10.0.4

  • PR-466 - Allow string match of formatting file extension to exist in the topic name (or elsewhere in the file besides the end)

  • PR-456 - CCMSG-1420 Exclude unused netty-codec dependency

Version 10.0.3

  • PR-453 - CCMSG-1373 use lastIndexOf for deriving adjusted filename when the extension is present in the filename.

  • PR-450 - Moving to OpenJDK 8 build image.

Version 10.0.2

  • PR-448 - CC-15510: Add guava dependency for standalone functionality

  • PR-445 - CCMSG-1259: Upgrade commons-compress to 1.21

Version 10.0.1

  • PR-441 - CCMSG-1146: Upgrade json-smart to 2.4.7

  • PR-432 - MINOR: Remove unused dependency on kafka-connect-storage-hive

  • PR-432 - CC-14987: Fix serialization when headers.format.class is json

  • PR-429 - Upgrade jackson-dataformat-cbor to 2.11.4

  • PR-414 - CCMSG-1021: Upgrade AWS SDK to 1.11.1015 and exclude jackson-dataformat-cbor

  • PR-399 - SEC-1996: Replace commons-httpclient with apache httpclient

  • PR-394 - Upgrade to hadoop common 2.10.1 for parquet support

Version 10.0.0

  • PR-393 - MINOR: Upgrade storage common to 10.0.5 and catch PartitionException in TopicPartitionWriter’s error handling

  • PR-392 - CCMSG-861: Catch additional exception to report in DLQ

  • PR-391 - CC-13830: Upgraded S3 connector to use HttpClient defined in kafka-connect-storage-common

  • PR-349 - CC-9723: DLQ support in S3 sink

  • PR-379 - CC-12772: Upgraded testcontainers to 1.15.0 to address commons-compress CVE

  • PR-377 - CC-12825: Removed unintentional proprietary dependencies

  • PR-378 - CC-12695: Stop throwing retriable exceptions on non-retriable S3 errors

  • PR-325 - CC-7933: Support writing key and headers to S3

  • PR-262 - MINOR: Restore the complete hierarchy of exceptions during failures in part upload

  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3

  • PR-346 - SEC-1034: log4j migration to confluent repackaged version

  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms

  • PR-328 - CC-9198 Move null value handling earlier in SinkTask put

  • PR-291 - S3 Object Level Tagging for Offsets & Record Counts

Version 5.5.10

No changes

Version 5.5.9

No changes

Version 5.5.8

  • PR-485 - CCMSG-1531 Support null items within arrays with Parquet writer

  • PR-473 - CCMSG-1582 Fix bug that generates a small files when scheduled rotation has expired

  • PR-451 - Minor: Remove toString() call in trace log

Version 5.5.7

  • PR-456 - CCMSG-1420 Exclude unused netty-codec dependency for fix CVE

Version 5.5.6

No changes

Version 5.5.5

  • PR-394 - Upgrade to hadoop common 2.10.1 for parquet support

  • PR-432 - MINOR: Remove unused dependency on kafka-connect-storage-hive in the 5.2.x release branch

  • PR-429 - Pin version of jackson-dataformat-cbor to 2.11.4

  • PR-414 - CCMSG-1021: Upgrade AWS SDK to 1.11.1015 and exclude jackson-dataformat-cbor

Version 5.5.4

  • PR-399 - SEC-1996: Replace commons-httpclient with apache httpclient

  • PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common

Version 5.5.3

  • PR-379 - CC-12772: Upgrade testcontainers to 1.15.0 to address commons-compress CVE

  • PR-377 - CC-12825: Remove unintentional proprietary dependencies

  • PR-378 - CC-12695: Stop throwing retriable exceptions on non-retriable S3 errors

Version 5.5.2

  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3

  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms

Version 5.5.1

  • PR-328 - CC-9198: Move null value handling earlier in SinkTask put

Version 5.5.0

  • PR-318 - CC-8170: Exclude jackson-mapper-asl and commons-beanutils from transient dependencies of hadoop-common

  • PR-310 - MINOR: Fix log message when tagging an S3 object

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM AssumeRole credentials provider

  • PR-305 - CC-7892: Introduce configuration properties for key and secret to easily define credentials per connector instance

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

  • PR-291 - S3 object level tagging for offsets and record counts

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.4.8

No changes

Version 5.4.7

  • PR-473 - CCMSG-1582 Fix bug that generates a small files when scheduled rotation has expired

Version 5.4.6

  • PR-456 - CCMSG-1420 Exclude unused netty-codec dependency

Version 5.4.5

  • PR-394 - Upgrade to hadoop common 2.10.1 for parquet support

  • PR-432 - MINOR: Remove unused dependency on kafka-connect-storage-hive in the 5.2.x release branch

  • PR-429 - Pin version of jackson-dataformat-cbor to 2.11.4

Version 5.4.4

  • PR-414 - CCMSG-1021: Upgrade AWS SDK to 1.11.1015 and exclude jackson-dataformat-cbor

  • PR-399 - SEC-1996: Replace commons-httpclient with apache httpclient

  • PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common

  • PR-378 - CC-12695: stop throwing retriable exceptions on non-retriable S3 errors

Version 5.4.3

  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3

  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms

  • PR-328 - CC-9198: Move null value handling earlier in SinkTask put

Version 5.4.2

  • PR-318 - CC-8170: Exclude jackson-mapper-asl and commons-beanutils from transient dependencies of hadoop-common

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM AssumeRole credentials provider

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

Version 5.4.1

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.4.0

  • PR-289 - MINOR: Upgrade bugfix versions of dependencies to avoid CVEs

  • PR-286 - MINOR: Pin hadoop-common version to 2.7.7 and exclude htrace-core dependency

  • PR-282 - CC-6565: Use parquet.codec and add compression options for Parquet format

  • PR-281 - CC-6801: Retry effectively on retriable failures while closing files

  • PR-280 - CC-6766: Handle null-valued records in all three output types

  • PR-241 - Add Parquet Format

  • PR-264 - MINOR: New log for SSE

  • PR-263 - MINOR: Remove jackson databind version override

  • PR-250 - CC-4969 MINOR: Remove Unused S3Storage object from TopicPartitionWriter

  • PR-114 - CC-147: logical types support for HiveSchemaConverter

  • PR-116 - CC-6565: Add parquet.codec property to enable compression in storage sink connectors

  • PR-110 - CC-5963: Improved ability to handle schema evolution with different names

Version 5.3.8

No changes

Version 5.3.7

No changes

Version 5.3.6

No changes

Version 5.3.5

  • PR-432 - MINOR: Remove unused dependency on kafka-connect-storage-hive in the 5.2.x release branch

  • PR-429 - Pin version of jackson-dataformat-cbor to 2.11.4

  • PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common

  • PR-378 - CC-12695: stop throwing retriable exceptions on non-retriable S3 errors

Version 5.3.4

  • PR-357 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3 (back port to 5.0.x)

Version 5.3.3

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM Assume Role credentials provider

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.3.2

No changes

Version 5.3.1

No changes

Version 5.3.0

  • PR-244 - CC-4433c: Remove unused dependency on Hive.

  • PR-243 - CC-4228: Add configuration option to override expect-continue in upload protocol.

Version 5.2.7

No changes

Version 5.2.6

No changes

Version 5.2.5

  • PR-432 - MINOR: Remove unused dependency on kafka-connect-storage-hive in the 5.2.x release branch

  • PR-429 - Pin version of jackson-dataformat-cbor to 2.11.4

  • PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common

  • PR-378 - CC-12695: stop throwing retriable exceptions on non-retriable S3 errors

  • PR-357 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3 (back port to 5.0.x)

Version 5.2.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM AssumeRole credentials provider

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.2.3

No changes

Version 5.2.2

No changes

Version 5.2.1

No changes

Version 5.2.0

  • PR-220 - CC-3035: Upgrade the S3Mock library so the module builds on Java 8 and Java 11

Version 5.1.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM AssumeRole credentials provider

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.1.3

No changes

Version 5.1.2

No changes

Version 5.1.1

No changes

Version 5.1.0

No changes

Version 5.0.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-311 - Add AWS IAM AssumeRole credentials provider

  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.0.3

No changes

Version 5.0.2

No changes

Version 5.0.1

  • PR-214 - CC-2617: Accept timestamp for wallclock instances to assure consistency

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions

  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs

  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 5.0.0

  • PR-181 - MINOR: Update config options to include SSE-C

  • PR-173 - Extend the existing SSE support to enable Customer (provided) Keys SSE-C

  • PR-171 - MINOR: Remove MaxPermSize since we dropped support for Java 7

  • PR-160 - CC-1793. Used full jitter backoff strategy from AWS SDK.

  • PR-156 - CC-1497. Added logic for passing a subset of configs down to configurable credentials provider

  • PR-63 - DataUtils::getField returns field

Version 4.1.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 4.1.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions

  • PR-198 - Add autogenerated configs that were missed before

  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs

  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 4.1.2

No changes

Version 4.1.1

  • PR-155 - CC-918: Update build to generate plugin package

  • PR-159 - CC-1791: Added compression options to raw value example

Version 4.1.0

  • PR-150 - Make confluent CLI command output shared

  • PR-148 - Rely on jackson.version from common pom instead of defining it ourselves

  • PR-149 - Bump Jackson version

  • PR-140 - Fix for BufferOverflowException in S3OutputStream

  • PR-106 - Add s3.sse.kms.key.id to specify a particular key in AWS KMS

  • PR-98 - Add gzip compression support for json and byte array formats

  • PR-143 - Support for multiple fields in field partitioner

  • PR-138 - CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner

  • PR-132 - CC-1214: Import all the AvroData configs to storage sink connectors

  • PR-123 - CC-1299: Refactor this code to use the AVRO_CODEC config key from storage-common

  • PR-114 - Update to PowerMock 2.0.x and Mockito 2

  • PR-48 - RecordField nested timestamp extraction

  • PR-57 - Issue-53. Multiple field support for partitioning.

  • PR-55 - CC-1489: NPE on records with null timestamp.

  • PR-45 - HOTFIX: Consider all the properties in AvroDataConfig.

  • PR-52 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core

  • PR-51 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core

  • PR-50 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core

  • PR-46 - CC-1299: Create AVRO_CODEC ConfigKey

Version 4.0.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions

  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs

  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 4.0.2

No changes

Version 4.0.1

  • PR-148 - Rely on jackson.version from common pom instead of defining it ourselves

  • PR-149 - Bump Jackson version

  • PR-140 - Fix for BufferOverflowException in S3OutputStream

  • PR-138 - CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner

  • PR-132 - CC-1214: Import all the AvroData configs to storage sink connectors

  • PR-55 - CC-1489: NPE on records with null timestamp.

  • PR-45 - HOTFIX: Consider all the properties in AvroDataConfig.

Version 4.0.0

  • PR-102 - Allow to write records at the root of the bucket

  • PR-105 - Add instructions about upstream builds

  • PR-97 - MINOR: S3 Proxy Settings revisited

  • PR-85 - Add ByteArrayFormat

  • PR-87 - Add s3.canned.acl config

  • PR-84 - Create CONTRIBUTING.md

  • PR-92 - CC-1114: Switch to common pom and fix checkstyle issues.

  • PR-78 - Retry s3 part upload

  • PR-80 - HOTFIX: Remove refs to schema-generator config.

  • PR-60 - Add logging when creating a new RecordWriter

  • PR-44 - HOTFIX: Remove unused parameter in newConfigDef for PartitionerConfig.

  • PR-41 - HOTFIX: Update dependencies

  • PR-40 - HOTFIX: Make specific dependencies explicit.

  • PR-35 - Add missing modules to the dependencyManagement pom section so downstream projects will inherit the right version automatically.

  • PR-31 - Remove schema.generator.class config and have Formats specify their own SchemaGenerator internally

Version 3.3.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions

  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 3.3.2

No changes

Version 3.3.1

  • PR-116 - HOTFIX: Add recommender for schema generator classes only on 3.3.x.

  • PR-112 - CC-1244: 3.3.1 release notes

  • PR-108 - HOTFIX: Handle primitive types in AvroFormat

  • PR-104 - Fix new client creation to cater for URLs and region together

  • PR-100 - CC-1167: Add recommenders for class type properties

  • PR-90 - Add upstream project so build are triggered automatically

  • PR-82 - CC-872: Update default part.size number in S3 docs and more.

  • PR-77 - Update quickstart to use Confluent CLI.

  • PR-43 - HOTFIX: Add recommender for schema generator classes only on 3.3.x.

  • PR-39 - HOTFIX: Check simple class names in recommender.

  • PR-38 - CC-1153: Isolate config instances and recommenders.

  • PR-36 - CC-1153: Storage connectors can add recommendations for Class types

Version 3.3.0

  • PR-28 - HOTFIX: Explicitly deactivate licensing profiles by default.

  • PR-34 - CC-524: Reduce the default s3.part.size to 25MB to avoid OOM with current default jvm heap size.

  • PR-33 - CC-513: Newline is not appended between records when using JsonFormat.

  • PR-37 - Enable compression of output Avro files.

  • PR-38 - CC-530: Exclude storage-common jars when packaging S3 connector.

  • PR-19 - Move to using io.confluent:common for deps.

  • PR-22 - HOTFIX: Import recent changes from the hdfs connector.

  • PR-24 - Convert - to _ in Hive table names.

  • PR-25 - Include causes with ConfigExceptions caused by catching other exceptions.

  • PR-26 - Add constructors to SchemaGenerators to support no parameters and a Map config.

  • PR-29 - Timestamp information should be copied to projected SinkRecord.

Version 3.2.4

No changes

Version 3.2.3

No changes

Version 3.2.2

  • PR-19 - CC-500: Provide exactly-once time-based partitioning in S3

  • PR-45 - HOTFIX: S3SinkConnector should extend SinkConnector

  • PR-51 - Allow custom partitioners to have their own configs

  • PR-27 - HOTFIX: Include a trailing delimiter when verifying data format for Hive

  • PR-23 - HOTFIX: Add test schema and record builder for records with timestamp field

  • PR-18 - CC-497: Add timestamp based partitioners.

Version 3.2.1

  • PR-33 - Separate JSON records using line separator instead of single white space.

  • PR-34 - Reduce the default s3.part.size to 25MB to avoid OOM exceptions with the current default java heap size settings for Connect.

  • PR-32 - Add s3.region property to quickstart config and docs.

Version 3.2.0

Initial Version