Changelog

Version 2.4.0

  • Adding support for topic2TableMap config

Version 2.3.4

  • Fixing CVEs: CCDB-4936, CCDB-4836 & CCDB-4835.

Version 2.3.3

  • CCDB-4929: Adding error messages for union operation failure cases.

Version 2.3.0

  • Added user-agent header for GCP calls

Version 2.2.2

  • Retry on IOExceptions thrown during table writes

Version 2.2.1

  • CCDB-4518: Sanitize field names when converting to BQ schema
  • GH-164: Remove dependency on ConfigDef.CaseInsensitiveValidString
  • Support keywords by enclosing column identifiers by backtick characters

Version 2.2.0

  • Support Application Default Credentials (ADC) to find credentials automatically in GCP

Version 2.1.10

  • GH-139: Improve preflight config validation
  • GH-149: Improve error messages on write thread failure

Version 2.1.9

  • GH-138: Fail faster when attempting to write to non-day-partitioned tables with decorator syntax
  • MINOR: Handle new “table is deleted” error from BigQuery

Version 2.1.8

  • GH-161 Fixed cast error of Long in debezium time converter
  • GH-139: Move all user-visible configuration properties to single configuration class
  • GH-146: Parameterize BigQuerySinkConnectorIT tests
  • Adding table and column information to help debugging
  • GH-144: Isolate error classification logic into single, testable class

Version 2.1.7

  • Register TimestampConverter in DebeziumLogicalConverters.

Version 2.1.6

  • CCDB-4187: Recursively unionize subfields

Version 2.1.5

  • Skip record with null valueSchema when proposing new BigQuery schema.

Version 2.1.4

  • Prevent records with null value schema from causing NPE when delete is enabled
  • GH-65: Add support for Kafka logical Time type
  • Update supportSummary for BigQuery Sink
  • MINOR: Remove outdated properties

Version 2.1.3

  • GH-96 (follow-up): Include REPEATED fields in unionized schemas if missing from later schemas

Version 2.1.2

  • GH-96: Fix unionization/update logic for REPEATED fields

Version 2.1.1

  • MINOR: Increase retry period for writes after table creates/updates to 15 minutes
  • Prevent NPE thrown at SchemaManager.isFieldRelaxation() by specifying unspecified field mode as NULLABLE

Version 2.1.0

  • GH-80 (PR-84): Add support for configuring time partitioning type on created tables. See the timePartitioningTypeconfiguration properties.
  • PR-61: Expand support for time-partitioned tables to allow connector to write to HOUR-, MONTH-, and YEAR-partitioned tables.

Version 2.0.6

  • Skip record with null valueSchema when proposing new BigQuery schema.

Version 2.0.5

  • Prevent records with null value schema from causing NPE when delete is enabled
  • GH-65: Add support for Kafka logical Time type
  • MINOR: Remove outdated properties

Version 2.0.4

  • GH-96 (follow-up): Include REPEATED fields in unionized schemas if missing from later schemas

Version 2.0.3

  • GH-96: Fix unionization/update logic for REPEATED fields

Version 2.0.2

  • MINOR: Increase retry period for writes after table creates/updates to 15 minutes

Version 2.0.1

  • Prevent NPE thrown at SchemaManager.isFieldRelaxation() by specifying unspecified field mode as NULLABLE

Version 2.0.0

  • CCDB-3674: Make IdentitySchemaRetriever the default schema retriever for BigQuery 2.0.0
  • CC-8803 - Add support for upsert/delete through periodic merge flushes. See the following configuration properties:
    • upsertEnabled
    • deleteEnabled
    • intermediateTableSuffix
    • mergeIntervalMs
    • mergeRecordsThreshold
  • PR-291 - Generalize schema retriever (see Upgrading to 2.0.0)
  • PR-28 - Add allowSchemaUnionization config property

Version 1.6.5

No changes

Version 1.6.4

No changes

Version 1.6.3

No changes

Version 1.6.2

No changes

Version 1.6.1

  • PR-246 - CC-7246: add ability to partition based on timestamp of a record value field
  • PR-244 - Added configuration to add/remove partition decorator for BigQuery inserts
  • PR-237 - Add support for handling empty schemas and records
  • PR-231 - change log level for BigQuery exception to WARN

Version 1.5.2

  • PR-220 - CC-7399: Handle null value if schema-less record has a map with a null value
  • PR-215 - Use batch delete operation when cleaning up blob ids

Version 1.6.0

No changes

Version 1.5.1

No changes

Version 1.4.1

No changes

Version 1.3.0

  • PR-198 - Log errors encountered when writing batches
  • PR-192 - Log topic and table name when a row fails to be inserted into BQ
  • PR-191 - Added keyFileType field to connector configs
  • PR-188 - Add Support for Schemaless Record
  • PR-187 - Add field name Sanitization
  • PR-172 - Handle xargs and basename on linux
  • PR-170 - Fix gcs load retry logic
  • PR-168 - Support Java 11

Version 1.2.0

  • PR-167 - Upgrade google-cloud version
  • PR-164 - Add prefix/folder related configuration
  • PR-159 - Await termination on correct executor

Version 1.1.1

No changes

Version 1.1.2

  • PR-161 - Prevent context update after/while rebalancing

Version 1.1.0

Version 1.6.6

  • 1.6.x backports
  • MINOR: Catch ‘request payload exceeds size’ errors from BigQuery and reduce batch size in response
  • CC-12157: Throw nicer exception on detecting cycle in schema
  • GH-41: Fix concurrent integration test runs on Jenkins
  • MINOR: Fix scope for test dependencies (Avro converter, Schema Registry, etc.)
  • GH-31: Port Dockerized integration tests to embedded integration tests
  • GH-27: Add allowSchemaUnionization config property
  • MINOR: Remove PR template
  • MINOR: Fix integration test to use new datasets config property
  • MINOR: Add a brief summary of the WePay->Confluent migration to the README
  • RCCA-1483: Bump up ioConfluentVersion to 5.5.1 to include the latest kafka-connect-avro-converter.
  • CC-9208: Include original trace in worker trace when task fails.
  • CC-8804: Integration tests for upsert/delete