.. _examples_c3_alert_actions_triggers: **************************** Example triggers and actions **************************** Follow these examples to set up triggers and actions. .. _config_triggers_ex: Example triggers ================ This section provides trigger examples using the |c3-short| web UI for alerts. .. _alert_trigger_cluster_offline: Create a trigger for |c3-short| cluster down -------------------------------------------- Follow these steps to create an alert trigger for cluster down using the UI rather than the :ref:`properties file configurations `. .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Trigger** tab. #. Enter a unique name for the trigger. For example, ``Cluster down``. .. include:: ../includes/unique-trigger-action-name.rst #. In the **Components** pane, set fields as follows: - For **Component type**, select **Cluster**. - For **Cluster id**, select the cluster ID for |c3-short| from the list. #. In the **Criteria** pane, set fields as follows: - For **Metric**, select **Cluster down**. - For **Condition**, select **Yes**. .. figure:: ../../images/c3-cluster-down-alert.png :width: 600px Cluster down trigger example #. Click **Submit**. The trigger saved message displays and prompts you to create or select an action now or do it later. Create a corresponding :ref:`action ` or actions for the trigger as applicable to your environment. For next steps, see the cluster down action examples: - :ref:`Create an email action ` - :ref:`Create a Slack webhook action ` - :ref:`Create a PagerDuty notification and incident action ` .. _URP_cluster_trigger: Create a cluster trigger for under-replicated partitions -------------------------------------------------------- Follow these steps to create an alert trigger for any under-replicated topic partitions (URP) in a cluster. Alert on topic partitions when the number of in-sync replicas is less than replication factor. The number of in sync replicas should equal the total number of replicas. Investigate whenever the under replicated partitions value is greater than zero (0). This example creates a cluster trigger. The under-replicated topic partitions metric is also available for a :ref:`topic trigger `. **To create a cluster trigger for URP:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Triggers** tab -> **+ New trigger**. The :ref:`New trigger form ` appears. #. Enter a unique name for the trigger. For example, ``URP my_cluster``. .. include:: ../includes/unique-trigger-action-name.rst #. In the **Components** pane, set fields as follows: - For **Component type**, select ``Cluster``. - For **Cluster id**, select the cluster. .. include:: ../includes/warn-select1-cluster-trigger.rst #. In the **Criteria** pane, set fields as follows: - For **Metric**, select ``Under-replicated topic partitions``. - For **Condition**, select ``Greater than``. - For **Value**, type ``0`` (zero). .. figure:: ../../images/c3-trigger-cluster-urp.png :width: 600px Under-replicated topic partitions cluster trigger example #. Click **Submit**. The trigger saved message displays and prompts you to create or select an action now or do it later. Create a corresponding :ref:`action ` or actions for the trigger as applicable to your environment. .. _URP_topic_trigger: Create a topic trigger for under-replicated partitions ------------------------------------------------------ Follow these steps to create an alert trigger for under-replicated partitions (URP) in a specific topic. This example creates a topic trigger. The under-replicated topic partitions metric is also available for a :ref:`cluster trigger `. **To create a topic trigger for URP:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Triggers** tab -> **+ New trigger**. The :ref:`New trigger form ` appears. #. Enter a unique name for the trigger. For example, ``URP users topic``. .. include:: ../includes/unique-trigger-action-name.rst #. In the **Components** pane, set fields as follows: - For **Component type**, select ``Topic``. - For **Cluster id**, select the cluster. - For **Condition**, select ``Equals`` (to select one specific topic). - For **Topic name**, select the name of the topic to alert for under-replicated partitions. Enter some text of the topic name to filter the list if necessary. #. In the **Criteria** pane, set fields as follows: - For **Metric**, select ``Under-replicated topic partitions``. - For **Condition**, select ``Greater than``. - For **Value**, type ``0`` (zero). .. figure:: ../../images/c3-trigger-topic-urp.png :width: 600px Under-replicated topic partitions topic trigger example #. Click **Submit**. The trigger saved message displays and prompts you to create or select an action now or do it later. Create a corresponding :ref:`action ` or actions for the trigger as applicable to your environment. .. _AC_cluster_trigger: Create a cluster trigger for active controller count ---------------------------------------------------- Follow these steps to create an alert trigger for active controller count in a specific cluster. There should be only one active controller at any given time. If multiple brokers (or no brokers) report as being the controller, controller threads are not working properly. To remedy, investigate and restart the brokers in the cluster. The active controller count metric represents the number of brokers in the cluster reporting as the active controller in the last interval. The number of active controllers is displayed in: - The **Active controller** card of the :ref:`c3_brokers_overview_metrics` for a cluster. - The **Broker uptime** pane of the Brokers overview > :ref:`Metrics page ` for a cluster. - The deprecated :ref:`System health ` monitoring page. .. figure:: ../../images/c3-brokers-active-controller.png :scale: 50% Multiple active controllers issue **To create a cluster trigger for active controller count:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Triggers** tab -> **+ New trigger**. The :ref:`New trigger form ` appears. #. Enter a unique name for the trigger. For example, ``Active controller count``. .. include:: ../includes/unique-trigger-action-name.rst #. In the **Components** pane, set fields as follows: - For **Component type**, select ``Cluster``. - For **Cluster id**, select the cluster. #. In the **Criteria** pane, set fields as follows: - For **Metric**, select ``Active controller count``. - For **Condition**, select ``Not equal to``. - For **Value**, type ``1``. .. figure:: ../../images/c3-cluster-trigger-active-controller.png :width: 600px Active controller count cluster trigger example #. Click **Submit**. The trigger saved message displays and prompts you to create or select an action now or do it later. Create a corresponding :ref:`action ` or actions for the trigger as applicable to your environment. .. _consumer_lag_trigger: Create a consumer group trigger for consumer lag ------------------------------------------------ Follow these steps to create an alert trigger for consumer lag in a specific consumer group. Consumer lag measures how many messages a consumer group is behind. This example sets up the alert from the context of the Consumer groups page. Most of the fields are already pre-populated. Just enter a trigger name and the lag value you want to alert on as appropriate for your environment. This example triggers an alert when the consumer lag is greater than 100. **To create a consumer lag trigger:** #. From the navigation menu, click **Cluster > Consumers**. #. Click the link for the consumer group. The Consumer lag tab appears. .. figure:: ../../images/c3-consumer-lag-alert-button.png :width: 600px :alt: Consumer lag page set up an alert Consumer lag page set up an alert #. Click the **Set up an alert** button. The New trigger form appears with most fields pre-populated. #. Enter a unique name for the trigger. For example, ``Consumer group lag``. .. include:: ../includes/unique-trigger-action-name.rst #. In the **Criteria** pane, set fields as follows: - For **Condition**, select ``Greater than``. - For **Value**, type ``100``. .. figure:: ../../images/c3-cgroup-trigger-clag.png :width: 600px :alt: Consumer lag consumer group trigger example Consumer lag consumer group trigger example #. Click **Submit**. The trigger saved message displays and prompts you to create or select an action now or do it later. Create a corresponding :ref:`action ` or actions for the trigger as applicable to your environment. .. _config_actions_ex: Example actions =============== This section provides action examples using the web UI for alerts. .. _alert_email_action: Create an email action for cluster down --------------------------------------- Follow these steps to create an email action that can be assigned to one or more triggers. This example creates an action to send an email alert for cluster down once per hour. Prerequisites: - :ref:`Enable email alerts ` - :ref:`Configure SMTP settings ` - :ref:`alert_trigger_cluster_offline` **To create an email action:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Actions** tab -> **+ New action**. The New action form appears. #. Enter a unique name for the **Action Name**. For example: ``Send email to DevOps``. .. include:: ../includes/unique-trigger-action-name.rst #. (Optional) The action is enabled by default. Click the **Enabled** toggle to **Disabled** if you are not ready to activate the action. The action remains disabled regardless of the :ref:`pause or resume all actions ` status. #. From the **Triggers** list, select one or more triggers to assign to the Action. #. In the **Actions** pane, set fields as follows: - Select **Send email** for the **Action** field. - Enter one or more email addresses separated by a comma in the **Recipient email address** field. - Enter a subject for the email. For example: ``Attention DevOps: Control Center Cluster Down Alert!`` - Enter a **Max send rate value**. For example, ``1`` for once. - Select the frequency option. For example, ``Per hour``. .. figure:: ../../images/c3-send-alert-email-action.png :width: 600px #. Click **Save**. The action is added to the Actions management page. .. _alert_slack_webhook_action: Create a Slack notification action for cluster down --------------------------------------------------- Follow these steps to create a Slack webhook action that can be assigned to one or more triggers. This example creates a Slack notification to post to a designated Slack channel once per day until the issue is acknowledged and resolved. Prerequisites: - Configure and test the incoming webhook in `Slack `_. - Ensure the webhook |c3-short| configuration setting is enabled (it is by default). See :ref:`enable the webhook action `. - Create one or more :ref:`triggers ` to associate the action with. For this example, see :ref:`alert_trigger_cluster_offline`. **To create a Slack action:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Actions** tab -> **+ New action**. The :ref:`New action form ` appears. #. Enter a unique name for the **Action Name**. For example, ``Control Center Cluster down Slack webhook``. .. include:: ../includes/unique-trigger-action-name.rst #. (Optional) The action is enabled by default. Click the **Enabled** toggle to **Disabled** if you are not ready to activate the action. The action remains disabled regardless of the :ref:`pause or resume all actions ` status. #. From the **Triggers** list, select one or more triggers to assign to the action. #. In the **Actions** pane, set fields as follows: - Select **Send Slack notification** for the **Action** field. - Enter the Slack webhook URL in the **Webhook URL** field. - Enter a brief subject for the notification. For example: ``Attention DevOps: Control Center Cluster Down Alert!`` - Enter a value for **Max send rate**. For example, ``1`` for once. - Select the frequency option. For example, ``Per day``. - (Optional) To override the username originally configured in Slack, enter the Slack handle. For example, ``@otherslackusername``. - (Optional) To override the slack channel originally configured in Slack, enter the alternative channel. For example, ``#DevOpsOnCall``. .. figure:: ../../images/c3-down-slack-action.png :width: 600px #. Click **Save**. The action is added to the Actions management page. .. _alert_pagerduty_webhook_action: Create a PagerDuty notification action for cluster down ------------------------------------------------------- Follow these steps to create a PagerDuty webhook notification action that can be assigned to one or more triggers. This example creates a PagerDuty notification to create a critical incident for |c3-short|'s cluster down. Prerequisites: - Configure the PagerDuty Service integration and generate an Integration Key. Refer to the `PagerDuty `_ documentation for complete details. - Ensure the webhook |c3-short| configuration setting is enabled (it is by default). See :ref:`enable the webhook action `. - Create one or more :ref:`triggers ` to associate the action with. For this example, see :ref:`alert_trigger_cluster_offline`. **To create a PagerDuty action:** .. first nav step click bell .. include:: ../includes/alerts-bell-nav.rst #. Click the **Actions** tab -> **+ New action**. The :ref:`New action form ` appears. #. Enter a unique name for the **Action Name**. For example, ``Control Center Cluster down PagerDuty incident``. .. include:: ../includes/unique-trigger-action-name.rst #. (Optional) The action is enabled by default. Click the **Enabled** toggle to **Disabled** if you are not ready to activate the action. The action remains disabled regardless of the :ref:`pause or resume all actions ` status. #. From the **Triggers** list, select one or more triggers to assign to the action. #. In the **Actions** pane, set fields as follows: - Select ``Send PagerDuty notification`` for the **Action** field. - Enter a brief subject for the notification. For example: ``Attention DevOps: Control Center Cluster Down Alert!`` - Enter a **Max send rate value**. For example, ``1`` for once. - Select the frequency option. For example, ``Per day``. - Enter the **Integration key** generated from following the PagerDuty instructions. - Select the **Payload severity** for the incident that the notification will create automatically. Available options are: Critical, Error, Warning, or Info. For this example, select ``Critical``. .. figure:: ../../images/c3-down-pagerduty-action.png :width: 600px #. Click **Save**. The action is added to the Actions management page.