Changelog

Version 10.0.0

  • PR-393 - MINOR: Upgrade storage common to 10.0.5 and catch PartitionException in TopicPartitionWriter’s error handling
  • PR-392 - CCMSG-861: Catch additional exception to report in DLQ
  • PR-391 - CC-13830: Upgrade S3 connector to use HttpClient defined in kafka-connect-storage-common
  • PR-349 - CC-9723: DLQ support in S3 sink
  • PR-379 - CC-12772: Upgrade testcontainers to 1.15.0 to address commons-compress CVE
  • PR-377 - CC-12825: Remove unintentional proprietary dependencies
  • PR-378 - CC-12695: Stop throwing retriable exceptions on non-retriable S3 errors
  • PR-325 - CC-7933: Support writing key and headers to S3
  • PR-262 - MINOR: Restore the complete hierarchy of exceptions during failures in part upload
  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3
  • PR-346 - SEC-1034: log4j migration to confluent repackaged version
  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms
  • PR-328 - CC-9198 Move null value handling earlier in SinkTask put
  • PR-291 - S3 Object Level Tagging for Offsets & Record Counts

Version 5.5.3

  • PR-379 - CC-12772: Upgrade testcontainers to 1.15.0 to address commons-compress CVE
  • PR-377 - CC-12825: Remove unintentional proprietary dependencies
  • PR-378 - CC-12695: Stop throwing retriable exceptions on non-retriable S3 errors

Version 5.5.2

  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3
  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms

Version 5.5.1

  • PR-328 - CC-9198: Move null value handling earlier in SinkTask put

Version 5.5.0

  • PR-318 - CC-8170: Exclude jackson-mapper-asl and commons-beanutils from transient dependencies of hadoop-common
  • PR-310 - MINOR: Fix log message when tagging an S3 object
  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM AssumeRole credentials provider
  • PR-305 - CC-7892: Introduce configuration properties for key and secret to easily define credentials per connector instance
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
  • PR-291 - S3 object level tagging for offsets and record counts
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.4.3

  • PR-356 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3
  • PR-340 - CC-9753: Validate timezone configuration with rotate.schedule.interval.ms
  • PR-328 - CC-9198: Move null value handling earlier in SinkTask put

Version 5.4.2

  • PR-318 - CC-8170: Exclude jackson-mapper-asl and commons-beanutils from transient dependencies of hadoop-common
  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM AssumeRole credentials provider
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats

Version 5.4.1

  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.4.0

  • PR-289 - MINOR: Upgrade bugfix versions of dependencies to avoid CVEs
  • PR-286 - MINOR: Pin hadoop-common version to 2.7.7 and exclude htrace-core dependency
  • PR-282 - CC-6565: Use parquet.codec and add compression options for Parquet format
  • PR-281 - CC-6801: Retry effectively on retriable failures while closing files
  • PR-280 - CC-6766: Handle null-valued records in all three output types
  • PR-241 - Add Parquet Format
  • PR-264 - MINOR: New log for SSE
  • PR-263 - MINOR: Remove jackson databind version override
  • PR-250 - CC-4969 MINOR: Remove Unused S3Storage object from TopicPartitionWriter
  • PR-114 - CC-147: logical types support for HiveSchemaConverter
  • PR-116 - CC-6565: Add parquet.codec property to enable compression in storage sink connectors
  • PR-110 - CC-5963: Improved ability to handle schema evolution with different names

Version 5.3.4

  • PR-357 - CC-11480: Add configuration property to choose between path style and virtual hosted bucket access to S3 (back port to 5.0.x)

Version 5.3.3

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM Assume Role credentials provider
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.3.2

No changes

Version 5.3.1

No changes

Version 5.3.0

  • PR-244 - CC-4433c: Remove unused dependency on Hive.
  • PR-243 - CC-4228: Add configuration option to override expect-continue in upload protocol.

Version 5.2.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM AssumeRole credentials provider
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.2.3

No changes

Version 5.2.2

No changes

Version 5.2.1

No changes

Version 5.2.0

  • PR-220 - CC-3035: Upgrade the S3Mock library so the module builds on Java 8 and Java 11

Version 5.1.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM AssumeRole credentials provider
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.1.3

No changes

Version 5.1.2

No changes

Version 5.1.1

No changes

Version 5.1.0

No changes

Version 5.0.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-311 - Add AWS IAM AssumeRole credentials provider
  • PR-308 - CC-8085: Allow setting compression level for gzip in Json and ByteArray formats
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 5.0.3

No changes

Version 5.0.2

No changes

Version 5.0.1

  • PR-214 - CC-2617: Accept timestamp for wallclock instances to assure consistency
  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions
  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs
  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 5.0.0

  • PR-181 - MINOR: Update config options to include SSE-C
  • PR-173 - Extend the existing SSE support to enable Customer (provided) Keys SSE-C
  • PR-171 - MINOR: Remove MaxPermSize since we dropped support for Java 7
  • PR-160 - CC-1793. Used full jitter backoff strategy from AWS SDK.
  • PR-156 - CC-1497. Added logic for passing a subset of configs down to configurable credentials provider
  • PR-63 - DataUtils::getField returns field

Version 4.1.4

  • PR-309 - MINOR: Pin httpclient dependency to 4.5.9 to silence warning by the AWS SDK
  • PR-306 - MINOR: Upgrade AWS SDK to 1.11.725

Version 4.1.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions
  • PR-198 - Add autogenerated configs that were missed before
  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs
  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 4.1.2

No changes

Version 4.1.1

  • PR-155 - CC-918: Update build to generate plugin package
  • PR-159 - CC-1791: Added compression options to raw value example

Version 4.1.0

  • PR-150 - Make confluent CLI command output shared
  • PR-148 - Rely on jackson.version from common pom instead of defining it ourselves
  • PR-149 - Bump Jackson version
  • PR-140 - Fix for BufferOverflowException in S3OutputStream
  • PR-106 - Add s3.sse.kms.key.id to specify a particular key in AWS KMS
  • PR-98 - Add gzip compression support for json and byte array formats
  • PR-143 - Support for multiple fields in field partitioner
  • PR-138 - CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner
  • PR-132 - CC-1214: Import all the AvroData configs to storage sink connectors
  • PR-123 - CC-1299: Refactor this code to use the AVRO_CODEC config key from storage-common
  • PR-114 - Update to PowerMock 2.0.x and Mockito 2
  • PR-48 - RecordField nested timestamp extraction
  • PR-57 - Issue-53. Multiple field support for partitioning.
  • PR-55 - CC-1489: NPE on records with null timestamp.
  • PR-45 - HOTFIX: Consider all the properties in AvroDataConfig.
  • PR-52 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
  • PR-51 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
  • PR-50 - CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
  • PR-46 - CC-1299: Create AVRO_CODEC ConfigKey

Version 4.0.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions
  • PR-196 - CC-2601: Add S3 Append-Late-Data option to configs
  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 4.0.2

No changes

Version 4.0.1

  • PR-148 - Rely on jackson.version from common pom instead of defining it ourselves
  • PR-149 - Bump Jackson version
  • PR-140 - Fix for BufferOverflowException in S3OutputStream
  • PR-138 - CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner
  • PR-132 - CC-1214: Import all the AvroData configs to storage sink connectors
  • PR-55 - CC-1489: NPE on records with null timestamp.
  • PR-45 - HOTFIX: Consider all the properties in AvroDataConfig.

Version 4.0.0

  • PR-102 - Allow to write records at the root of the bucket
  • PR-105 - Add instructions about upstream builds
  • PR-97 - MINOR: S3 Proxy Settings revisited
  • PR-85 - Add ByteArrayFormat
  • PR-87 - Add s3.canned.acl config
  • PR-84 - Create CONTRIBUTING.md
  • PR-92 - CC-1114: Switch to common pom and fix checkstyle issues.
  • PR-78 - Retry s3 part upload
  • PR-80 - HOTFIX: Remove refs to schema-generator config.
  • PR-60 - Add logging when creating a new RecordWriter
  • PR-44 - HOTFIX: Remove unused parameter in newConfigDef for PartitionerConfig.
  • PR-41 - HOTFIX: Update dependencies
  • PR-40 - HOTFIX: Make specific dependencies explicit.
  • PR-35 - Add missing modules to the dependencyManagement pom section so downstream projects will inherit the right version automatically.
  • PR-31 - Remove schema.generator.class config and have Formats specify their own SchemaGenerator internally

Version 3.3.3

  • PR-200 - CC-2619: Bugfix for append.late.data configuration rotating partitions
  • PR-187 - CC-2313: Handle late arriving records in storage cloud sink connectors

Version 3.3.2

No changes

Version 3.3.1

  • PR-116 - HOTFIX: Add recommender for schema generator classes only on 3.3.x.
  • PR-112 - CC-1244: 3.3.1 release notes
  • PR-108 - HOTFIX: Handle primitive types in AvroFormat
  • PR-104 - Fix new client creation to cater for URLs and region together
  • PR-100 - CC-1167: Add recommenders for class type properties
  • PR-90 - Add upstream project so build are triggered automatically
  • PR-82 - CC-872: Update default part.size number in S3 docs and more.
  • PR-77 - Update quickstart to use Confluent CLI.
  • PR-43 - HOTFIX: Add recommender for schema generator classes only on 3.3.x.
  • PR-39 - HOTFIX: Check simple class names in recommender.
  • PR-38 - CC-1153: Isolate config instances and recommenders.
  • PR-36 - CC-1153: Storage connectors can add recommendations for Class types

Version 3.3.0

  • PR-28 - HOTFIX: Explicitly deactivate licensing profiles by default.
  • PR-34 - CC-524: Reduce the default s3.part.size to 25MB to avoid OOM with current default jvm heap size.
  • PR-33 - CC-513: Newline is not appended between records when using JsonFormat.
  • PR-37 - Enable compression of output Avro files.
  • PR-38 - CC-530: Exclude storage-common jars when packaging S3 connector.
  • PR-19 - Move to using io.confluent:common for deps.
  • PR-22 - HOTFIX: Import recent changes from the hdfs connector.
  • PR-24 - Convert - to _ in Hive table names.
  • PR-25 - Include causes with ConfigExceptions caused by catching other exceptions.
  • PR-26 - Add constructors to SchemaGenerators to support no parameters and a Map config.
  • PR-29 - Timestamp information should be copied to projected SinkRecord.

Version 3.2.4

No changes

Version 3.2.3

No changes

Version 3.2.2

  • PR-19 - CC-500: Provide exactly-once time-based partitioning in S3
  • PR-45 - HOTFIX: S3SinkConnector should extend SinkConnector
  • PR-51 - Allow custom partitioners to have their own configs
  • PR-27 - HOTFIX: Include a trailing delimiter when verifying data format for Hive
  • PR-23 - HOTFIX: Add test schema and record builder for records with timestamp field
  • PR-18 - CC-497: Add timestamp based partitioners.

Version 3.2.1

  • PR-33 - Separate JSON records using line separator instead of single white space.
  • PR-34 - Reduce the default s3.part.size to 25MB to avoid OOM exceptions with the current default java heap size settings for Connect.
  • PR-32 - Add s3.region property to quickstart config and docs.

Version 3.2.0

Initial Version