Horizontal Scaling

The Oracle CDC Source connector scales horizontally using the existing Kafka Connect framework.

The connector is configured with three tasks in the following graphic.

  • Task 1: Reads records from the Oracle Database Redo Log, then writes these records to a redo log topic in Apache Kafka®.
  • Task 2: Creates a snapshot for the DEPARTMENTS and EMPLOYEES tables.
  • Task 3: Creates a snapshot for the JOBS table.

Once Task 2 and Task 3 are done with snapshotting, they read records from the Redo Log Topic in Kafka rather than from the Oracle database Redo Log Topic. The three tasks populate Kafka with table-specific topics from the snapshot forward.

Tip

If you need to create snapshots for many tables, you can add more tasks to get better snapshot performance.

Oracle CDC Source Architecture

Horizontal Scaling Example