Use Cluster Linking on Confluent Cloud with AWS PrivateLink¶

This topic describes Network Links in Confluent Cloud. The feature is soon to be deprecated and is being replaced in AWS regions in phases.

Important

Cluster Linking between Dedicated clusters over AWS PrivateLinks no longer requires Network Links if your clusters are in the following AWS regions. And it is no longer necessary or recommended to set up Network Linking to use Cluster Linking in those regions.

eu-central-1
eu-west-1
eu-west-2
us-east-1
us-east-2
us-west-2
ap-south-1
ap-southeast-1
ap-southeast-2

If you are currently using the now legacy Network Links feature, and if you choose to continue using Network Links, you cannot set up and use Cluster Linking between AWS and Azure.

To upgrade existing Network Links, contact Confluent Support.

When you set up topic replication between two Dedicated Confluent Cloud clusters deployed with PrivateLink configuration on AWS, you need to set up and operate a replicator in your network. With Network Linking, you can enable managed topic mirroring using Cluster Linking. You can use Network Linking to create a secure path between two Dedicated Confluent Cloud clusters with private link configurations, enabling Cluster Linking operations.

A Network Link is a unidirectional link that establishes one-way connectivity from the origin network to the target network, either in the same or different cloud regions. To create bidirectional connectivity and data replication, two separate Network Links are needed.

Network Links do not incur additional charges in Confluent Cloud.

A Network Link is composed of two components: a Network Link Service and a Network Link Endpoint.

../_images/cluster-linking-network-linking.png

The target Confluent Cloud Network contains:

A Network Link Service

A Network Link Service contains a list of network ID(s) and/or environment ID(s) that are allowed to establish Network Links to the target network.

A Network Link Service can be updated with additional, different, or fewer, environment ID(s) and network ID(s) to change the allowed network topologies. If an environment or network id is removed from the list, all network links from that environment or network are terminated. A terminated Network Link will cause its Network Link Endpoint to enter the disconnected state and its cluster link(s) to stop replicating data.
Network Link Service Association

A Network Link Service Association contains a list of incoming Network Link Endpoints associated with a Network Link Service.

The origin Confluent Cloud Network contains:

A Network Link Endpoint

A Network Link Endpoint refers to one specific Network Link Service. It must be created after the Network Link Service.

When the Network Link Endpoint’s status changes to READY, the Network Link has been successfully established.

Requirements¶

Network Links and their Cluster Links must stay within the bounds of one Confluent Cloud Organization that is authorized to incur billing charges and contains:
- User(s) with the following RBAC roles: OrganizationAdmin, EnvironmentAdmin, or NetworkAdmin.
- Two Dedicated Confluent Cloud clusters with AWS PrivateLink configuration in different Confluent Cloud networks (any regions).
- Connectivity to Confluent REST API endpoints (https://api.confluent.cloud/), available over secure public endpoints) for one-time REST API calls. This can be from a personal computer, SSH session, or development instance.
An organization in Confluent Cloud can have a maximum of 20 Network Links.
Each Confluent Cloud network can have a maximum of 1 Network Link Service.
Each Confluent Cloud network can have a maximum of 5 Network Link Endpoints (outgoing links).
Each Network Link Service can have a maximum of 5 Network Link Endpoints that are referenced by Network Link Service Associations (incoming links).

The 5 incoming Network Link Endpoints for a Network Link Service must originate from different Confluent Cloud networks; they cannot all come from the same network.
Cluster Linking mirroring throughput (the bandwidth used to read data or write data to your cluster) is counted against your Fixed limits and recommended guidelines.
The REST API calls to manage Network Linking must be authenticated using the Cloud API key and the secret.

Create a Network Link¶

Note

Currently, you can use Confluent REST API, Confluent CLI, or Terraform to create and manage Network Linking. Confluent Cloud Console is not supported to create and manage Network Linking.

You need the following information to create a Network Link:

Environment ID and Network ID of the origin Confluent Cloud network
Environment ID and Network ID of the target Confluent Cloud network

The high-level workflow to create a Network Link is:

Create a Network Link Service in the target network.
Verify that the Network Link Service is in the READY state.
Create a Network Link Endpoint in the origin network.
Verify that the Network Link Endpoint is in the READY state.

See an end-to-end example of creating and managing Network Linking using Terraform.

Create a Network Link Service¶

Create a Network Link Service in Confluent Cloud.

For details about the Network Link Service REST request, see Network Link Services.

In the target network, create a Network Link Service with an accept list.

An accept list lets what Environments or Networks are allowed to connect to this target network using the Network Link.

REST request

POST https://api.confluent.cloud/networking/v1/network-link-services

REST request body

{
    "spec":
    {
        "display_name": "<display-name>",
        "description": "<optional-description>",
        "environment":
        {
            "id": "<environment-id-this-network-link-belongs-to>"
        },
        "network":
        {
            "id": "<network-id-this-network-link-belongs-to>"
        },
        "accept": {
            "networks": [ "<list-of-allowed-network-ids>" ],
            "environments": [ "<list-of-allowed-environment-ids>" ]
        }
    }
 }

Take note of the ID of the network_link_service in the REST API response.

Use the confluent network link service create Confluent CLI command to create a Network Link Service:

confluent network link service create  [name] [flags]

The following are the command-specific flags:

--network: Required. The ID of the Confluent Cloud network this service belongs to.
--description: The network link service description.
--accepted-networks: A comma-separated list of networks from which connections can be accepted.
--accepted-environments: A comma-separated list of environments from which connections can be accepted.

The following example command creates a Network Link Service for network “n-123456” with accepted environments “env-111111” and “env-222222”:

confluent network link service create \
  --network n-123456 \
  --description "example network link service" \
  --accepted-environments env-111111,env-222222

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Use the confluent_network_link-service resource to create a Network Link Service.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_network" "aws-private-link" {
  display_name     = "AWS Private Link Network"
  cloud            = "AWS"
  region           = "us-east-1"
  connection_types = ["PRIVATELINK"]
  zones            = ["use1-az1", "use1-az2", "use1-az6"]
  environment {
    id = confluent_environment.development.id
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "confluent_network_link_service" "aws_nls" {
  display_name = "AWS Private Link network link service"
  environment {
    id = confluent_environment.development.id
  }
  network {
    id = confluent_network.aws-private-link.id
  }
  description = "Test NL service"
  accept {
    environments = ["env-5678", "env-0923"]
    networks = ["n-1234"]
  }

  lifecycle {
    prevent_destroy = true
  }
}

See an end-to-end example of creating and managing Network Linking using Terraform.

Check Network Link Service status¶

Check that the Network Link Service is in the READY state.

For details about the Network Link Service REST request, see Network Link Service Status.

REST request

GET https://api.confluent.cloud/networking/v1/network-link-services/<network-link-service-id>

Use the confluent network link service describe Confluent CLI command to create a Network Link Service:

confluent network link service describe <id> [flags]

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following example command describe network link service “nls-123456”.

confluent network link service describe nls-123456

Create a Network Link Endpoint¶

Create a Network Link Service Endpoint in Confluent Cloud.

For details about the Network Link Endpoint REST request, see Network Link Endpoint.

In the origin network, create a Network Link Endpoint, using the network_link_service.id from the previous step.

If you don’t have this ID from the previous step, it can be retrieved by listing the Network Link Services in the target Environment or target Confluent Cloud network with the this request.

REST request

POST https://api.confluent.cloud/networking/v1/network-link-endpoints

REST request body

{
   "spec":
   {
       "display_name": "<display-name>",
       "description": "<optional-description>",
       "environment":
       {
           "id": "<environment-id-this-network-link-belongs-to>"
       },
       "network":
       {
           "id": "<network-id-this-network-link-belongs-to>"
       },
       "network_link_service":
       {
           "id": "<ID-of-created-network-link-service>"
       }
   }
}

Use the confluent network link endpoint create Confluent CLI command to create a Network Link Endpoint:

confluent network link endpoint create [name] [flags]

The following are the command-specific flags:

--network: Required. The ID of the Confluent Cloud network this endpoint belongs to.
--network-link-service: Required. The Network Link Service ID.
--description: The description of this Network Link Endpoint.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create an AWS Network Link Endpoint for network “n-123456” and network link service “nls-abcde1”:

confluent network link endpoint create \
  --network n-123456 \
  --description "example network link endpoint" \
  --network-link-service nls-abcde1

Use the confluent_network_link_endpoint resource to create a Network Link Endpoint.

An example snippet of Terraform configuration:

resource "confluent_network_link_endpoint" "nle" {
  display_name = "nle1"
  description  = "TEST-NLE1"
  environment {
    id = "env-xyz456"
  }
  network {
    id = "n-abc123"
  }
  network_link_service {
    id = "nls-g3e1ox"
  }
}

output "network_link_endpoint" {
  value = confluent_network_link_endpoint.nle
}

See an end-to-end example of creating and managing Network Linking using Terraform.

Check Network Link Endpoint status¶

Check that the Network Link is in the READY state.

For details about the Network Link Endpoint REST request, see Network Link Endpoint Status.

REST request

GET https://api.confluent.cloud/networking/v1/network-link-endpoints/<network-link-endpoint-id>

REST response example

{
  "api_version": "networking/v1",
  "id": "nle-61ozeo",
  "kind": "NetworkLinkEndpoint",
  "metadata": {
    "created_at": "2023-03-23T16:32:29.851017Z",
    "resource_name": "crn://confluent.cloud/organization=94b47aa0-ed3a-44da-9460-6cec0c710e84/environment=env-y9z9k/network=n-pr1ml2/network-link-endpoint=nle-61ozeo",
    "self": "https://api.confluent.cloud/networking/v1/network-link-endpoints/nle-61ozeo?environment=env-y9z9k",
    "updated_at": "2023-03-23T16:36:16.46025Z"
  },

  ...

  "status": {
    "phase": "READY"
  }
}

Use the confluent network link endpoint describe Confluent CLI command to check the status of a Network Link Endpoint:

confluent network link endpoint describe <id> [flags]

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Check Network Link Service Associations¶

You can fetch the list of incoming network link endpoints to a Network Link Service using the Network Link Service Association API.

For details, see Cloud API reference.

REST request

GET https://api.confluent.cloud/networking/v1/network-link-service-associations

REST query parameters

spec.network_link_service: Required. ID of the Network Link Service.
environment: Required. ID of the Confluent Cloud environment.

REST response example

{
  "api_version": "networking/v1",
  "id": "nle-gqz3wo",
  "kind": "NetworkLinkServiceAssociation",
  "spec": {
    ...
    "network_link_endpoint": "nle-gqz3wo",
    "network_link_service":
    {
      "environment": "env-domzy7",
      "id": "nls-6v8x5j",
    }
  }
}

Use the confluent network link service association describe Confluent CLI command to create a Network Link Service:

confluent network link service association describe <id> [flags]

The following is a command-specific flag:

--network-link-service: Required. The Network Link Service ID.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Update a Network Link¶

You can update the following fields in a Network Link.

Network Link Service: display_name, description, accept

Updating the accept policy to remove an existing environment or network will result in the termination of the connection.
Network Link Endpoint: display_name, description

Update a Network Link Service¶

Update a Network Link Service in Confluent Cloud.

REST request

PATCH https://api.confluent.cloud/networking/v1/network-link-services/<network-link-service-id>

REST request body

{
   "spec":
   {
      "display_name": "<new-display-name>",
      "description": "<new-description>",
      "accept":
      {
         "networks": [ "<updated-list-of-allowed-network-ids>" ],
         "environments": [ "<updated-list-of-allowed-environment-ids>" ]
       },
      "environment":
      {
         "id": "<environment-id-this-network-link-belongs-to>"
      }
   }
}

Use the confluent network link service update Confluent CLI command to update a Network Link Service:

confluent network link service update  [id] [flags]

The following are the command-specific flags:

--name: The name of the Network Link Service.
--description: The description of the Network Link Service.
--accepted-networks: A comma-separated list of networks from which connections can be accepted.
--accepted-environments: A comma-separated list of environments from which connections can be accepted.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following example Confluent CLI command updates the name and description of network link service “nls-123456”:

confluent network link service update nls-123456 \
  --name my-network-link-service \
  --description "example network link service"

The following example Confluent CLI command update the accepted environments and accepted networks of network link service “nls-123456”:

confluent network link service update nls-123456 \
  --description "example network link service" \
  --accepted-environments env-111111 \
  --accepted-networks n-111111,n-222222

Update a Network Link Endpoint¶

Create a Network Link Service Endpoint in Confluent Cloud.

REST request

PATCH https://api.confluent.cloud/networking/v1/network-link-endpoints/<network-link-endpoint-id>

REST request body

{
   "spec":
   {
      "display_name": "<new-display-name>",
      "description": "<new-description>",
      "environment":
      {
         "id": "<environment-id-this-network-link-belongs-to>"
      }
   }
}

Use the confluent network link endpoint update Confluent CLI command to update a Network Link Endpoint:

confluent network link endpoint update <id> [flags]

The following are the command-specific flags:

--name: The name of the Network Link Endpoint.
--description: The description of the Network Link Endpoint.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following example command updates the name and description of network link endpoint “nle-123456”:

confluent network link endpoint update nle-123456 \
  --name my-network-link-endpoint \
  --description "example network link endpoint"

Delete a Network Link¶

Deleting a Network Link will stop data replication from any cluster links that it contains (this is expected to take several minutes).

To delete a Network Link:

Delete the Network Link Endpoint.
Delete the Network Link Service.

Delete a Network Link Service¶

You can delete a Network Link Service that is in the READY state.

You can delete a Network Link Service only if there is no active, pending provisioning, or pending de-provisioning Network Link Endpoints is connected.

REST request

DELETE https://api.confluent.cloud/networking/v1/network-services/<network-link-service-id>

Use the confluent network link service delete Confluent CLI command to delete a Network Link Service:

confluent network link service delete <id-1> [id-2] ... [id-n] [flags]

The following is the command-specific flag:

--force: Skip the deletion confirmation prompt.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Delete a Network Link Endpoint¶

You can delete a Network Link Endpoint that is in the READY state.

REST request

DELETE https://api.confluent.cloud/networking/v1/network-link-endpoints/<network-link-endpoint-id>

Use the confluent network link endpoint delete Confluent CLI command to delete a Network Link Endpoint:

confluent network link endpoint delete <id-1> [id-2] ... [id-n] [flags]

--force: Skip the deletion confirmation prompt.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Alternatively, you can terminate a connection between two networks by updating the allowlist on the Network Link Service. Remove the ID of the Network or Environment containing the Network Link Endpoint. This is useful if the owner of the Network Link Service wants to delete the Network Link but doesn’t have access to the Network Link Endpoint. The Network Link Endpoint will enter the DISCONNECTED state, and will need to be deleted.

You cannot recover a Network Link Endpoint that is in the DISCONNECTED state. You must delete the DISCONNECTED Network Link Endpoint before you can establish a new Network Link between this permutation of networks.

Cluster Linking using Network Link¶

After the Network Link is created, you can create Cluster Links for privately networked, Dedicated Confluent Cloud clusters on AWS.

The origin network hosts the destination Kafka cluster.
The target network hosts the source Kafka cluster.

Source-initiated cluster links are not available with Network Links.

To get started, see Cluster Linking between AWS PrivateLink clusters.