Example triggers and actions

Follow these examples to set up triggers and actions.

Example triggers

This section provides trigger examples using the Control Center web UI for alerts.

Create a trigger for Control Center cluster down

Follow these steps to create an alert trigger for cluster down using the UI rather than the properties file configurations.

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Trigger tab.

  3. Enter a unique name for the trigger. For example, Cluster down.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. In the Components pane, set fields as follows:

    • For Component type, select Cluster.
    • For Cluster id, select the cluster ID for Control Center from the list.
  5. In the Criteria pane, set fields as follows:

    • For Metric, select Cluster down.

    • For Condition, select Yes.

      ../../_images/c3-cluster-down-alert.png

      Cluster down trigger example

  6. Click Submit. The trigger saved message displays and prompts you to create or select an action now or do it later.

    Create a corresponding action or actions for the trigger as applicable to your environment.

    For next steps, see the cluster down action examples:

Create a cluster trigger for under-replicated partitions

Follow these steps to create an alert trigger for any under-replicated topic partitions (URP) in a cluster. Alert on topic partitions when the number of in-sync replicas is less than replication factor. The number of in sync replicas should equal the total number of replicas. Investigate whenever the under replicated partitions value is greater than zero (0).

This example creates a cluster trigger. The under-replicated topic partitions metric is also available for a topic trigger.

To create a cluster trigger for URP:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Triggers tab -> + New trigger. The New trigger form appears.

  3. Enter a unique name for the trigger. For example, URP my_cluster.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. In the Components pane, set fields as follows:

    • For Component type, select Cluster.
    • For Cluster id, select the cluster.

    Warning

    There is a known issue when multiple clusters are selected for a broker or cluster trigger. As a recommended best practice, only select a single cluster for the trigger. For more information, see the known issues section in the release notes.

  5. In the Criteria pane, set fields as follows:

    • For Metric, select Under-replicated topic partitions.

    • For Condition, select Greater than.

    • For Value, type 0 (zero).

      ../../_images/c3-trigger-cluster-urp.png

      Under-replicated topic partitions cluster trigger example

  6. Click Submit. The trigger saved message displays and prompts you to create or select an action now or do it later.

    Create a corresponding action or actions for the trigger as applicable to your environment.

Create a topic trigger for under-replicated partitions

Follow these steps to create an alert trigger for under-replicated partitions (URP) in a specific topic.

This example creates a topic trigger. The under-replicated topic partitions metric is also available for a cluster trigger.

To create a topic trigger for URP:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Triggers tab -> + New trigger. The New trigger form appears.

  3. Enter a unique name for the trigger. For example, URP users topic.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. In the Components pane, set fields as follows:

    • For Component type, select Topic.
    • For Cluster id, select the cluster.
    • For Condition, select Equals (to select one specific topic).
    • For Topic name, select the name of the topic to alert for under-replicated partitions. Enter some text of the topic name to filter the list if necessary.
  5. In the Criteria pane, set fields as follows:

    • For Metric, select Under-replicated topic partitions.

    • For Condition, select Greater than.

    • For Value, type 0 (zero).

      ../../_images/c3-trigger-topic-urp.png

      Under-replicated topic partitions topic trigger example

  6. Click Submit. The trigger saved message displays and prompts you to create or select an action now or do it later.

    Create a corresponding action or actions for the trigger as applicable to your environment.

Create a cluster trigger for active controller count

Follow these steps to create an alert trigger for active controller count in a specific cluster. There should be only one active controller at any given time. The active controller count metric represents the number of brokers in the cluster reporting as the active controller in the last interval.

The number of active controllers is displayed in the Broker uptime pane of the Brokers overview page for a cluster. The active controllers count is also displayed in the deprecated System health monitoring page.

To create a cluster trigger for active controller count:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Triggers tab -> + New trigger. The New trigger form appears.

  3. Enter a unique name for the trigger. For example, Active controller count.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. In the Components pane, set fields as follows:

    • For Component type, select Cluster.
    • For Cluster id, select the cluster.
  5. In the Criteria pane, set fields as follows:

    • For Metric, select Active controller count.

    • For Condition, select Not equal to.

    • For Value, type 1.

      ../../_images/c3-cluster-trigger-active-controller.png

      Active controller count cluster trigger example

  6. Click Submit. The trigger saved message displays and prompts you to create or select an action now or do it later.

    Create a corresponding action or actions for the trigger as applicable to your environment.

Create a consumer group trigger for consumer lag

Follow these steps to create an alert trigger for consumer lag in a specific consumer group. Consumer lag measures how many messages a consumer group is behind. This example sets up the alert from the context of the Consumer groups page. Most of the fields are already pre-populated. Just enter a trigger name and the lag value you want to alert on as appropriate for your environment. This example triggers an alert when the consumer lag is greater than 100.

To create a consumer lag trigger:

  1. From the navigation menu, click Cluster > Consumers.

  2. Navigate to the Consumer Group. Select the Consumer Group from the list if the consumer group you want is not already displayed.

    ../../_images/c3-consumer-group-nav.png
  3. Click the Set up an alert context menu option. The Consumer Trigger form appears with most fields pre-populated.

    ../../_images/c3-trigger-consumer-lag-prepop.png
  4. Enter a unique name for the trigger. For example, Consumer group lag.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  5. In the Criteria pane, set fields as follows:

    • For Value, type 100.
    • For Condition, select Greater than.
    ../../_images/c3-cgroup-trigger-clag.png

    Consumer lag consumer group trigger example

  6. Click Submit. The trigger saved message displays and prompts you to create or select an action now or do it later.

    Create a corresponding action or actions for the trigger as applicable to your environment.

Example actions

This section provides action examples using the web UI for alerts.

Create an email action for cluster down

Follow these steps to create an email action that can be assigned to one or more triggers. This example creates an action to send an email alert for cluster down once per hour.

Prerequisites:

To create an email action:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Actions tab -> + New action. The New action form appears.

  3. Enter a unique name for the Action Name. For example: Send email to DevOps.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. (Optional) The action is enabled by default. Click the Enabled toggle to Disabled if you are not ready to activate the action. The action remains disabled regardless of the pause or resume all actions status.

  5. From the Triggers list, select one or more triggers to assign to the Action.

  6. In the Actions pane, set fields as follows:

    • Select Send email for the Action field.

    • Enter one or more email addresses separated by a comma in the Recipient email address field.

    • Enter a subject for the email. For example: Attention DevOps: Control Center Cluster Down Alert!

    • Enter a Max send rate value. For example, 1 for once.

    • Select the frequency option. For example, Per hour.

      ../../_images/c3-send-alert-email-action.png
  7. Click Save. The action is added to the Actions management page.

Create a Slack notification action for cluster down

Follow these steps to create a Slack webhook action that can be assigned to one or more triggers. This example creates a Slack notification to post to a designated Slack channel once per day until the issue is acknowledged and resolved.

Prerequisites:

To create a Slack action:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Actions tab -> + New action. The New action form appears.

  3. Enter a unique name for the Action Name. For example, Control Center Cluster down Slack webhook.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. (Optional) The action is enabled by default. Click the Enabled toggle to Disabled if you are not ready to activate the action. The action remains disabled regardless of the pause or resume all actions status.

  5. From the Triggers list, select one or more triggers to assign to the action.

  6. In the Actions pane, set fields as follows:

    • Select Send Slack notification for the Action field.

    • Enter the Slack webhook URL in the Webhook URL field.

    • Enter a brief subject for the notification. For example: Attention DevOps: Control Center Cluster Down Alert!

    • Enter a value for Max send rate. For example, 1 for once.

    • Select the frequency option. For example, Per day.

    • (Optional) To override the username originally configured in Slack, enter the Slack handle. For example, @otherslackusername.

    • (Optional) To override the slack channel originally configured in Slack, enter the alternative channel. For example, #DevOpsOnCall.

      ../../_images/c3-down-slack-action.png
  7. Click Save. The action is added to the Actions management page.

Create a PagerDuty notification action for cluster down

Follow these steps to create a PagerDuty webhook notification action that can be assigned to one or more triggers. This example creates a PagerDuty notification to create a critical incident for Control Center's cluster down.

Prerequisites:

  1. Click the Alerts bell icon in the top banner. The Alerts page opens to the History tab by default.

  2. Click the Actions tab -> + New action. The New action form appears.

  3. Enter a unique name for the Action Name. For example, Control Center Cluster down PagerDuty incident.

    Note

    Uniqueness is not enforced. As a recommended best practice, use unique and descriptive names to avoid confusion.

  4. (Optional) The action is enabled by default. Click the Enabled toggle to Disabled if you are not ready to activate the action. The action remains disabled regardless of the pause or resume all actions status.

  5. From the Triggers list, select one or more triggers to assign to the action.

  6. In the Actions pane, set fields as follows:

    • Select Send PagerDuty notification for the Action field.

    • Enter a brief subject for the notification. For example: Attention DevOps: Control Center Cluster Down Alert!

    • Enter a Max send rate value. For example, 1 for once.

    • Select the frequency option. For example, Per day.

    • Enter the Integration key generated from following the PagerDuty instructions.

    • Select the Payload severity for the incident that the notification will create automatically. Available options are: Critical, Error, Warning, or Info. For this example, select Critical.

      ../../_images/c3-down-pagerduty-action.png
  7. Click Save. The action is added to the Actions management page.