Manage Observability Integrations on Confluent Cloud

Connect DataDog as a third-party integration in Confluent Cloud to simplify your observability activities.

Note

Observability integrations are an Early Access Program feature in Confluent Cloud.

An Early Access feature is a component of Confluent Cloud introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent, particularly as it becomes more widely available in follow-on preview editions.

Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. Early Access Program features are provided: (a) without support; (b) “AS IS”; and (c) without indemnification, warranty, or condition of any kind. No service level commitment will apply to Early Access Program features. Early Access Program features are considered to be a Proof of Concept as defined in the Confluent Cloud Terms of Service. Confluent may discontinue providing preview releases of the Early Access Program features at any time in Confluent’s sole discretion.

If you would like to participate in the Early Access Program, contact Confluent at obs.integration@confluent.io.

With observability integrations, you can send observability data from audit, connect, and Confluent Cloud for Apache Flink logs to DataDog. This empowers you to centralize all of your observability data and to store your data for longer durations to comply with your internal data and security policies.

Manage destinations

Destinations are a third-party integration where you can send your observability activities.

Destinations can be in a Ready or Error status.

Create destination

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click Add destination.

  3. Select DataDog as the provider for storing your logs. Provide the following details to configure DataDog as your observability destination. For more information, see DataDog Logs documentation.

    Name

    A human-readable name for the destination, such as my_datadog_destination.

    Description

    A description of the destination, such as “Send observability data to DataDog”.

    Datadog Site

    Your DataDog site parameter. Select the appropriate site based on your region:

    • datadoghq.com for US1

    • datadoghq.eu for EU

    • us3.datadoghq.com for US3

    • us5.datadoghq.com for US5

    • ap1.datadoghq.com for AP1

    • ddog-gov.com for US1-FED

    API Key

    Your DataDog API key ID for authentication. This key authenticates requests and sends data to your DataDog organization. For information on managing API keys, see DataDog API Keys.

  4. After you have entered the details, click Complete.

You will receive confirmation that your destination was created. You can then set up your observability integration.

Edit destination

You can edit the details for your existing destination. This is helpful if you need to update your API key or any additional details.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your destination name.

  3. Click Edit.

  4. In the Edit destination panel, you can update the same details you entered when creating your destination.

    Note

    If you are changing the configuration details, you must also re-enter your destination credentials. You can change the Name and Description without re-entering your credentials.

  5. Click Save changes.

Your changes to the destination are updated immediately.

Delete destination

You can delete your existing destination.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your destination name.

  3. Click Delete destination.

  4. Enter your destination name, which you can copy from the Delete window.

  5. Click Delete.

Your destination will no longer show up on the Observability integrations page.

Manage observability integrations

Observability integrations are the connection from Confluent Cloud to a destination.

Observability integrations can be in one of these statuses:

  • Active

  • Error

  • Provisioning

  • Paused

  • Resuming

  • Updating

Create observability integration

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click Add observability integration.

  3. To set up your observability integration, enter the required details:

    Name

    A name for the observability integration that describes the integration, such as production_logs_to_datadog.

    Description

    A description of the observability integration, such as “Stream production cluster logs to DataDog for long-term storage”.

    Destination

    Select the destination where logs will be sent. Select the dropdown to see a list of your available configured destinations.

    Log Source

    Select the log sources to include in the data streamed to your destination. Select the dropdown to see a list of your available sources.

  4. Click Create integration.

You can now view your observability integration on the Observability integrations page, along with their destination and status. From the Observability integrations page, you can also edit your observability integration to revise any of these details.

Observability data is not transferred while your observability integration is Provisioning. Once the integration is in Active state, your data starts to transfer.

Edit observability integration

You can edit the details for your existing observability integrations. This is helpful if you need to update any integration details, including log sources.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your observability integration name.

  3. Click Edit.

  4. In the Edit panel, you can update the same details you entered when creating your observability integration.

  5. Click Save changes.

Your changes to the observability integration are updated almost immediately. While your observability integration is Updating, no observability data is transferred to your destination from this observability integration.

Pause observability integration

If you want to temporarily stop transferring observability data to a destination, you can pause your observability integration.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your observability integration name.

  3. On the observability integration page, select Pause.

  4. Confirm you want to pause your integration.

While your observability integration is Paused, no observability data is transferred to your destination from this observability integration.

Resume observability integration

If you want to start transferring observability data to a destination again, you can resume your observability integration. When you resume your integration, Confluent Cloud transfers only new observability data. Confluent Cloud does not transfer the data that was created while the integration was paused.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your observability integration name.

  3. On the observability integration page, select Resume.

  4. Confirm you want to resume your integration.

Observability data is not transferred while your observability integration is Resuming. Once the integration is in Active state, your data starts to transfer again.

Delete observability integration

You can delete your existing observability integration.

  1. In Confluent Cloud Console, open the sidebar menu. Below the current user name, click Observability integrations.

  2. Click your observability integration name.

  3. Click Delete observability integration.

  4. Enter your observability integration name, which you can copy from the Delete window.

  5. Click Delete.

Your observability integration will no longer show up on the Observability integrations page.

Related content