Cluster Linking using AWS PrivateLink¶

When a dedicated Confluent Cloud cluster is deployed in a private network (Confluent Cloud Network), the cluster is isolated in its own network for network-level security. However, this security prevents Cluster Linking on another Confluent Cloud cluster, from accessing the Kafka cluster and its data. When using AWS PrivateLink, Network Linking can create a secure path between two isolated Confluent Cloud clusters for Cluster Linking.

A Network Link is a unidirectional link that establishes one-way connectivity from the origin network to the target network, either in the same or different cloud regions. To create bidirectional connectivity and data replication, two separate Network Links are needed.

Network Links do not incur additional charges in Confluent Cloud.

A Network Link is composed of two components: a Network Link Service and a Network Link Endpoint.

../_images/cluster-linking-network-linking.png

The target Confluent Cloud Network contains:

A Network Link Service

A Network Link Service contains a list of network ID(s) and/or environment ID(s) that are allowed to establish Network Links to the target network.

A Network Link Service can be updated with additional, different, or fewer, environment ID(s) and network ID(s) to change the allowed network topologies. If an environment or network id is removed from the list, all network links from that environment or network are terminated. A terminated Network Link will cause its Network Link Endpoint to enter the disconnected state and its cluster link(s) to stop replicating data.

The origin Confluent Cloud Network contains:

A Network Link Endpoint

A Network Link Endpoint refers to one specific Network Link Service. It must be created after the Network Link Service.

When the Network Link Endpoint’s status changes to READY, the Network Link has been successfully established.

Requirements¶

Network Links and their Cluster Links must stay within the bounds of one Confluent Cloud Organization that is authorized to incur billing charges and contains:
- User(s) with the following RBAC roles: OrganizationAdmin, EnvironmentAdmin, or NetworkAdmin.
- Two Dedicated Confluent Cloud clusters with AWS PrivateLink configuration in different Confluent Cloud networks (any regions).
- Connectivity to Confluent REST API endpoints (https://api.confluent.cloud/), available over public internet) for one-time REST API calls. This can be from a personal computer, SSH session, or development instance.
An organization in Confluent Cloud can have a maximum of 20 Network Links.
1 Confluent Cloud Network can have a maximum of 1 Network Link service and 5 Network Link Endpoints.
The REST API calls to manage Network Linking must be authenticated using the Cloud API key and the secret.

Create a Network Link¶

Note

Currently, you can use Confluent REST API or Terraform to create and manage Network Linking. Confluent Cloud Console is not supported to create and manage Network Linking.

You need the following information to create a Network Link:

Environment ID and Network ID of the origin Confluent Cloud Network
Environment ID and Network ID of the target Confluent Cloud Network

For creating and managing Network Linking using Terraform, see:

The high-level workflow to create a Network Link via Confluent REST API is:

Create a Network Link Service in the target network.
Create a Network Link Endpoint in the origin network.
Verify that the Network Link is in the READY state.

Create a Network Link Service¶

For details about the Network Link Service REST request, see Network Link Services.

In the target network, create a Network Link Service with an accept list.

An accept list lets what Environments or Networks are allowed to connect to this target network using the Network Link.

REST request

POST https://api.confluent.cloud/networking/v1/network-link-services

REST request body

{
    "spec":
    {
        "display_name": "<display-name>",
        "description": "<optional-description>",
        "environment":
        {
            "id": "<environment-id-this-network-link-belongs-to>"
        },
        "network":
        {
            "id": "<network-id-this-network-link-belongs-to>"
        },
        "accept": {
            "networks": [ "<list-of-allowed-network-ids>" ],
            "environments": [ "<list-of-allowed-environment-ids>" ]
        }
    }
 }

Take note of the ID of the network_link_service in the REST API response.

Create a Network Link Endpoint¶

For details about the Network Link Endpoint REST request, see Network Link Endpoint.

In the origin network, create a Network Link Endpoint, using the network_link_service.id from the previous step.

If you don’t have this ID from the previous step, it can be retrieved by listing the Network Link Services in the target Environment or target Confluent Cloud Network with the this request.

REST request

POST https://api.confluent.cloud/networking/v1/network-link-endpoints

REST request body

{
   "spec":
   {
       "display_name": "<display-name>",
       "description": "<optional-description>",
       "environment":
       {
           "id": "<environment-id-this-network-link-belongs-to>"
       },
       "network":
       {
           "id": "<network-id-this-network-link-belongs-to>"
       },
       "network_link_service":
       {
           "id": "<ID-of-created-network-link-service>"
       }
   }
}

Check Network Link status¶

Check that the Network Link is in the READY state.

For details about the Network Link Endpoint REST request, see Network Link Endpoint Status.

After several minutes, the Network Link will be ready, and its status will change from PROVISIONING to READY at this endpoint:

REST request

GET https://api.confluent.cloud/networking/v1/network-link-endpoints?environment=<target-environment-id>

REST response example

{
  "api_version": "networking/v1",
  "id": "nle-61ozeo",
  "kind": "NetworkLinkEndpoint",
  "metadata": {
    "created_at": "2023-03-23T16:32:29.851017Z",
    "resource_name": "crn://stag.cpdev.cloud/organization=94b47aa0-ed3a-44da-9460-6cec0c710e84/environment=env-y9z9k/network=n-pr1ml2/network-link-endpoint=nle-61ozeo",
    "self": "https://api.stag.cpdev.cloud/networking/v1/network-link-endpoints/nle-61ozeo?environment=env-y9z9k",
    "updated_at": "2023-03-23T16:36:16.46025Z"
  },

  ...

  "status": {
    "phase": "READY"
  }
}

Update a Network Link¶

You can update the following fields in a Network Link.

Network Link Service: display_name, description, accept

Updating the accept policy to remove an existing environment or network will result in the termination of the connection.
Network Link Endpoint: display_name, description

To update a Network Link Service:

REST request

PATCH https://api.confluent.cloud/networking/v1/network-link-services/<network-link-service-id>

REST request body

{
   "spec":
   {
      "display_name": "<new-display-name>",
      "description": "<new-description>",
      "accept":
      {
         "networks": [ "<updated-list-of-allowed-network-ids>" ],
         "environments": [ "<updated-list-of-allowed-environment-ids>" ]
       },
      "environment":
      {
         "id": "<environment-id-this-network-link-belongs-to>"
      }
   }
}

To update a Network Link Endpoint:

REST request

PATCH https://api.confluent.cloud/networking/v1/network-link-endpoints/<network-link-endpoint-id>

REST request body

{
   "spec":
   {
      "display_name": "<new-display-name>",
      "description": "<new-description>",
      "environment":
      {
         "id": "<environment-id-this-network-link-belongs-to>"
      }
   }
}

Delete a Network Link¶

Deleting a Network Link will stop data replication from any cluster links that it contains (this is expected to take several minutes).

You can delete Network Links that are in the READY state.

To delete a Network Link:

Delete the Network Link endpoint:

REST request

DELETE https://api.confluent.cloud/networking/v1/network-link-endpoints/<network-link-endpoint-id>?environment=<target-environment-id>

Delete the Network Link Service. The Network Link Service cannot be deleted if any active, pending provisioning, or pending de-provisioning Network Link Endpoints are connected.

REST request
```
DELETE https://api.confluent.cloud/networking/v1/network-services/<network-link-service-id>?environment=<target-environment-id>
```

Alternatively, you can terminate the connection between two networks by updating the allowlist on the Network Link Service. Remove the ID of the Network or Environment containing the Network Link Endpoint. This is useful if the owner of the Network Link Service wants to delete the Network Link but doesn’t have access to the Network Link Endpoint. The Network Link Endpoint will enter the DISCONNECTED state, and will need to be deleted.

You cannot recover a Network Link Endpoint that is in the DISCONNECTED state. You must delete the DISCONNECTED Network Link Endpoint before you can establish a new Network Link between this permutation of networks.

Cluster Linking using Network Link¶

After the Network Link is created, you can create Cluster Links for privately networked, Dedicated Confluent Cloud clusters on AWS.

The origin network hosts the destination Kafka cluster.
The target network hosts the source Kafka cluster.

For a tutorial on Cluster Linking in privately networked dedicated Confluent Cloud clusters, see here.

Source-initiated cluster links are not available with Network Links.

To get started, see Cluster Linking between AWS PrivateLink clusters.