<a id="connect-azure-sql-dw"></a>

# Azure Synapse Analytics Sink Connector for Confluent Platform

The Azure Synapse Analytics Sink connector allows you to export data from Apache Kafka®
topics to Azure Synapse Analytics. The connector polls data from Kafka to write to
the data warehouse based on the topics subscription. Auto-creation of tables and
limited auto-evolution are also supported. This connector is compatible with
[Azure Synapse Analytics SQL pool](https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/create-data-warehouse-portal)

#### IMPORTANT
<!-- WARNING: THIS IS A SHARED FILE AND THE SOURCE IS LOCATED IN DOCS-COMMON. DO NOT ADD TO ANY OTHER REPO. -->

Effective July 6, 2025, only self-managed connector versions that meet or exceed the minimum version listed on the
[Supported Connector Versions](https://docs.confluent.io/platform/7.8/connect/supported-connector-version.html#supported-connector-versions-till-cp-7-8)
page receive support from Confluent. Older, unsupported connector versions have been removed from Confluent Marketplace and
are no longer available for download.

## Features

The Azure Synapse Analytics Sink connector for Confluent Platform includes the following
features:

- [At least once delivery](#synapse-sink-at-least-once-delivery)
- [Dead Letter Queue](#synapse-sink-dead-letter-queue)
- [Multiple tasks](#synapse-sink-multiple-tasks)
- [Data mapping](#synapse-sink-data-mapping)
- [Auto-creation and auto-evolution](#synapse-sink-auto-creation)
- [Client-side encryption](#synapse-sink-csfle-sm)

<a id="synapse-sink-at-least-once-delivery"></a>

### At least once delivery

This connector guarantees that records are delivered at least once to the Kafka
topic. If the connector restarts, there may be some duplicate
records in the Kafka topic.

<a id="synapse-sink-dead-letter-queue"></a>

### Dead Letter Queue

This connector supports the Dead Letter Queue (DLQ) functionality. For
information about accessing and using the DLQ, see [Confluent Platform
Dead Letter Queue](/platform/current/connect/concepts.html#dead-letter-queue).

<a id="synapse-sink-multiple-tasks"></a>

### Multiple tasks

The Azure Synapse Analytics Sink connector supports running one or more tasks.
You can specify the number of tasks in the `tasks.max` configuration
parameter. Multiple tasks may improve performance when moving a large amount of
data.

<a id="synapse-sink-data-mapping"></a>

### Data mapping

The sink connector requires knowledge of schemas, so you should use a suitable
converter. For example, the Avro converter that comes with Schema Registry, or the JSON
converter with schemas enabled. Kafka record keys if present can be primitive
types or a Connect struct, and the record value must be a Connect struct. Fields
selected from Connect structs must be primitive types. You may need to implement
a custom `Converter` if the data in the topic is not in a compatible format.

<a id="synapse-sink-auto-creation"></a>

### Auto-creation and auto-evolution

If `auto.create` is enabled, the connector can create the destination table if
it is found to be missing. The connector uses the record schema as a basis for
the table definition, so the creation takes place online with records consumed
from the topic.

If `auto.evolve` is enabled, the connector can perform limited auto-evolution
by issuing alter on the destination table when it encounters a record for which
a column is found to be missing. Since data-type changes and removal of columns
can be dangerous, the connector does not attempt to perform such evolutions on
the table.

#### IMPORTANT
For backwards-compatible table schema evolution, new fields in record schemas
must be optional or have a default value.

<a id="synapse-sink-csfle-sm"></a>

### Client-side encryption

This connector supports Client-Side Field Level Encryption (CSFLE) and Client-Side Payload Encryption (CSPE). For more information, see [Manage Client-Side Encryption](https://docs.confluent.io/platform/current/connect/manage-csfle.html).

## Limitations

- This connector can only insert data into Azure Synapse Analytics. Azure Synapse
  Analytics does not support primary keys, and because updates, upserts, and
  deletes are all performed on the primary keys, these queries are not supported
  for this connector.
- When `auto.evolve` is enabled, if a new column with a default value is
  added, that default value is only used for new records. Existing records will
  have “null” as the value for the new column.

<a id="install-connector"></a>

## Install the Azure Synapse Analytics Sink Connector

You can install this connector by using the [confluent connect
plugin
install](https://docs.confluent.io/confluent-cli/current/command-reference/connect/plugin/confluent_connect_plugin_install.html)
command, or by manually downloading the ZIP file.

### Prerequisites

- You must install the connector on every machine where Connect will run.
- Confluent Platform 4.0.0 or later, or Kafka 1.0.0 or later.
- Java 1.8.
- At minimum, the `INSERT` permission is required for this connector. For
  more details, see [Permissions: GRANT, DENY, REVOKE (Azure Synapse Analytics,
  Parallel Data Warehouse)](https://docs.microsoft.com/en-us/sql/t-sql/statements/permissions-grant-deny-revoke-azure-sql-data-warehouse-parallel-data-warehouse?view=aps-pdw-2016-au7#permission-list).
- If `auto.create=true`, the `CREATE TABLE` and `CREATE SCHEMA`
  permissions are required. Additionally, if `auto.evolve=true`, the `ALTER
  ANY SCHEMA` permissions are required.
- An installation of the latest (`latest`) connector version.

  To install the `latest` connector version, navigate to your Confluent Platform
  installation directory and run the following command:
  ```bash
  confluent connect plugin install confluentinc/kafka-connect-azure-sql-dw:latest
  ```

  You can install a specific version by replacing `latest` with a version
  number as shown in the following example:
  ```bash
  confluent connect plugin install confluentinc/kafka-connect-azure-sql-dw:1.0.3
  ```

### Install the connector manually

[Download and extract the ZIP file](https://www.confluent.io/hub/confluentinc/kafka-connect-azure-sql-dw) for
your connector and then follow the manual connector installation
[instructions](/kafka-connectors/self-managed/userguide.html#connect-installing-plugins).

## License

You can use this connector for a 30-day trial period without a license key.

After 30 days, you must purchase a connector subscription which includes [Confluent enterprise license](/platform/current/installation/license.html#enterprise-subscription-license) keys to subscribers, along with [enterprise-level support](https://www.confluent.io/subscription/) for Confluent Platform and your connectors. If you are a subscriber, you can contact [Confluent Support](https://support.confluent.io/) for more information.

See [Confluent Platform license](azure_sql_dw_configuration_options.md#az-sql-dw-sink-connector-license-config) for license properties and
[License topic configuration](azure_sql_dw_configuration_options.md#azure-sql-dw-license-topic-configuration)
for information about the license topic.

## Configuration Properties

For a complete list of configuration properties for this connector, see
[Configuration Reference for Azure Synapse Analytics Sink Connector for Confluent Platform](azure_sql_dw_configuration_options.md#azure-sql-dw-configuration-options).

<a id="azure-sql-dw-sink-quickstart"></a>

## Quick Start

In this quick start, the Azure Synapse Analytics Sink connector is used to
export data produced by the Avro console producer to an Azure Synapse Analytics
instance.

Azure Prerequisites
: - [Azure Account](https://azure.microsoft.com/en-us/free/)
  - [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
  - [mssql-cli](https://docs.microsoft.com/en-us/sql/tools/mssql-cli?view=sql-server-ver15)

Confluent Prerequisites
: - [Confluent Platform](/platform/current/installation/index.html)
  - [Confluent CLI](https://docs.confluent.io/confluent-cli/current/installing.html) (requires separate installation)
  - [Azure Synapse Analytics Sink connector](#install-connector)

For an example of how to get Kafka Connect connected to [Confluent Cloud](/cloud/current/index.html), see
[Connect Self-Managed Kafka Connect to Confluent Cloud](/cloud/current/cp-component/connect-cloud-config.html#distributed-cluster).

<a id="create-azure-sql-dw"></a>

### Create an Azure Synapse Analytics instance

For this section, use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest)
to create the necessary resources for this Quick Start.

1. Login with your Azure account.
   ```bash
   az login
   ```

   The above command will open your default browser and load a sign-in page
   where you can login into your Azure account.
2. Create a resource group.
   ```bash
   az group create \
       --name quickstartResourceGroup \
       --location eastus2
   ```
3. Create a SQL server instance.

   Choose a unique SQL Server name, username and password and supply them for
   the `name`, `admin-user`, and `admin-password` arguments.
   ```bash
   az sql server create \
       --name <your-sql-server-name> \
       --resource-group quickstartResourceGroup \
       --location eastus2  \
       --admin-user <your-username> \
       --admin-password <your-password>
   ```
4. Enable a server-level firewall rule.

   Pass your IP address for the `start-ip-address` and
   `end-ip-address` argument to enable connectivity
   to the server.
   ```bash
   az sql server firewall-rule create \
       --name quickstartFirewallRule \
       --resource-group quickstartResourceGroup \
       --server <your-sql-server-name> \
       --start-ip-address <your-ip-address> --end-ip-address <your-ip-address>
   ```
5. Create a Synapse Analytics instance.
   ```bash
   az sql dw create \
       --name quickstartDataWarehouse \
       --resource-group quickstartResourceGroup \
       --server <your-sql-server-name>
   ```

   This can take a couple of minutes.

<a id="load-connector"></a>

### Load the connector

1. Adding a new connector plugin requires restarting Kafka Connect. Use the
   Confluent CLI to restart Connect.
   ```bash
   confluent local services connect stop && confluent local services connect start
   ```
2. Create an `azure-sql-dw-quickstart.properties` file and add the following properties.

   Make sure to substitute your SQL server name, username, and password in the
   `azure.sql.dw.url`, `azure.sql.dw.user` and `azure.sql.dw.password`
   arguments respectively.
   ```properties
   name=AzureSqlDwSinkConnector
   topics=products
   tasks.max=1
   connector.class=io.confluent.connect.azuresqldw.AzureSqlDwSinkConnector

   azure.sql.dw.url=jdbc:sqlserver://<your-sql-server-name>.database.windows.net:1433;
   azure.sql.dw.user=<your-username>
   azure.sql.dw.password=<your-password>
   azure.sql.dw.database.name=quickstartDataWarehouse
   auto.create=true
   auto.evolve=true
   table.name.format=kafka_${topic}

   # The following configs define the Confluent license stored in Kafka, so you need the Kafka bootstrap addresses.
   # `replication.factor` may not be larger than the number of Kafka brokers in the destination cluster,
   # so here we set this to '1' for demonstration purposes. Always use at least '3' in production configurations.
   confluent.license=
   confluent.topic.bootstrap.servers=localhost:9092
   confluent.topic.replication.factor=1
   ```
3. Start the Azure Synapse Analytics Sink connector by loading the connector’s
   configuration with the following command:
   ```bash
   confluent local load azure-sql-dw --config azure-sql-dw-quickstart.properties
   ```
4. Confirm that the connector is in a `RUNNING` state.
   ```bash
   confluent local status azure-sql-dw
   ```

<a id="send-data"></a>

### Send data to Kafka

1. To produce some records into the `products` topic, first start a Kafka
   producer.
   ```bash
   kafka-avro-console-producer \
   --broker-list localhost:9092 --topic products \
   --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"name","type":"string"}, {"name":"price","type":"float"}, {"name":"quantity","type":"int"}]}'
   ```
2. The console producer is now waiting for input, so you can go ahead and
   insert some records into the topic.
   ```bash
   {"name": "scissors", "price": 2.75, "quantity": 3}
   {"name": "tape", "price": 0.99, "quantity": 10}
   {"name": "notebooks", "price": 1.99, "quantity": 5}
   ```

<a id="verify-data-sent"></a>

### Check Azure Synapse Analytics for data

For this section, use the [mssql-cli](https://github.com/dbcli/mssql-cli/blob/master/doc/usage_guide.md) to query
the data in the data warehouse and validate that the connector was able to
export the data from Kafka to the data warehouse.

1. Connect to the remote data warehouse, supplying the SQL Server name,
   username, and password as arguments.
   ```bash
   mssql-cli -S <your-sql-server-name>.database.windows.net -U <your-username> -P <your-password> -d quickstartDataWarehouse
   ```
2. Now, query the `kafka_products` table to see its contents.
   ```bash
   select * from kafka_products;
   ```

   Your output should resemble the one below (the rows and columns may possibly
   be in a different order):
   ```bash
   +------------+---------+-----------+
   | quantity   | price   | name      |
   |------------+---------+-----------|
   | 10         | 0.99    | tape      |
   | 3          | 2.75    | scissors  |
   | 5          | 1.99    | notebooks |
   +------------+---------+-----------+
   ```

<a id="closing-resources"></a>

### Delete unnecessary resources

Once you’ve finished the quick start, go ahead and delete the resources created
here to avoid incurring additional charges.

```bash
az group delete --name quickstartResourceGroup
```

This command will delete the resource group and all the resources within it.