.. _controlcenter_userguide_alerts: Alerts ====== |c3-short| provides functionality for detecting anomalous events in your monitoring data and performing actions when they occur. .. _concepts: Concepts -------- Detection of anomalous events (*triggering*) is decoupled from the *actions* that should be taken when they occur. Each time interceptor data is received by |c3-short|, metric values (consumption difference and latency) of the corresponding time window(s) are updated to reflect the new data. All newly updated metric values are then checked against all configured triggers to determine whether the trigger should fire or not. .. note:: Interceptors can conceivably report data related to any time - alerting works across all time windows not just those near real time. Triggers can be associated with any number of actions. When a trigger fires, it will cause all associated actions to be executed for which the *max send rate* of the action has not been exceeded. If the max send rate of a particular action has been exceeded, the trigger event will be added to a list associated with the action and included in the action event the next time it is executed (actions report a set of triggers, not just one). Because of normal lag in the system, time windows close to real time will frequently have associated metric values that would be cause for concern if the time window was further behind real time. For this reason, triggers have an associated buffer value. A triggered event that is within *buffer* seconds of real time will not be registered against actions immediately - when the time window ultimately moves greater than *buffer* seconds behind real time, any associated metric value that would still cause a trigger to be fired will be registered against any appropriate actions then. User Interface -------------- Overview page ^^^^^^^^^^^^^ The **Overview** page has three main sections, accessible via the sub menus at the top of the page: - **HISTORY** - Historical alert log - **TRIGGERS** - Trigger management - **ACTIONS** - Action management .. figure:: images/c3alertssubmenus.png :scale: 50% :align: center Integration page ^^^^^^^^^^^^^^^^ The **Integration** page provides details of the alerts REST endpoint which can be used to programatically obtain historical alert information. .. figure:: images/c3alertsintegration.png :scale: 50% :align: center Trigger Management ------------------ Clicking on the triggers tab shows a summary of all configured triggers: .. figure:: images/c3alertstriggersoverview.png :scale: 50% :align: center You can edit/delete an existing trigger using the edit/delete links in this table or create a new trigger via the "+ new trigger" button on the top right of the screen. You can also initiate creation of a new trigger by clicking on a consumer group delivery or latency monitoring chart progress indicator, and clicking "setup alert" button. This will pre-populate the new trigger form with relevant information. .. figure:: images/c3alertsprepopulate.png :align: center .. _trigger: New/Edit Trigger Form ^^^^^^^^^^^^^^^^^^^^^ The form for creating and editing triggers is identical. When editing, the form is pre-populated with the current trigger values. There are three types of triggers that can be created: topics_, `consumer groups`_ or brokers_. .. _topics: Topic ^^^^^^^^^^^^^^^^^^ .. figure:: images/c3alertstopictrigger.png :align: center Trigger name A name used to identify the trigger (for example ' production requests'). Uniqueness is not enforced but you should use different names to avoid confusion for all triggers. Component type Should be selected as **Topic**. If this is not the case, see the other form trigger documentation. Cluster id The trigger for a topic will be limited to a specific cluster id. If you require a topic to be triggered by multiple clusters, create independant triggers for each cluster. Topic(s) name A select list of options for matching against the value field (below). The name of the topic can **Equal**, **Begin with**, **End with** or **Contains** a specified value. .. note:: For example, selecting **Contains** and then entering 'topic' into the value field will match 'my topic', 'topical', and 'topics with data'. If **Begins with** is selected, the trigger will only match 'topical' and 'topics with data', not 'my topic'. Topics(s) name value The name or part of a topic name to be triggered against. Works in conjunction with *Topic(s) Name* in order to match against one or many topics. .. note:: If multiple topics match against topic name, the trigger will be *per topic*, not aggregate. In the case where we have two topics that **Begin with** 'topic', and we set trigger **Bytes in** *greater than 100*, any topic will fire the trigger if they get > 100 **Bytes In**. Metric The value to check for the trigger alert. The possible values for this dropdown are: Bytes in Amount of bytes per second coming in to a topic. Bytes out Amount of bytes per second going out from a topic (does not account for internal replication traffic). .. note:: Prior to Kafka 0.11.0.0 the ``BytesOutPerSec`` accounted for traffic from the consumer as well as internal replication, this has been changed to only account for consumer traffic for this topic. Please adjust alerts accordingly. Out of sync replica count .. include:: fragments/topicInSyncReplica.rst Production request count Amount of production requests per second to a topic in a cluster. Under replicated topic partitions Amount of under replicated topic partitions. A use case for this metric would be wanting to know if a Kafka broker crashed while holding a specific topic partition. Condition The trigger will fire when *Condition* is true of the difference between the value of the metric being monitored and the value of the *Value* field. Possibly **Greater than**, **Less than**, **Equal to**, or **Not equal to**. Value The value to which the topic *Metric* is compared. .. _consumergroups: Consumer Groups ^^^^^^^^^^^^^^^ .. figure:: images/c3alertsedittrigger.png :align: center Trigger name A name used to identify the trigger (for example ' under consumption'). Uniqueness is not enforced but you should use different names to avoid confusion. Component type Should be selected as 'Consumer group'. If this is not the case, see the other form trigger documentation. Consumer group name The name of the consumer group to monitor for anomalies. Metric The metric to monitor. One of **maximum latency (ms)**, **average latency (ms)** or **consumption difference**. Condition The trigger will fire when *Condition* is true of the difference between the value of the metric being monitored and the value of the *Value* field. Possibly **Greater than**, **Less than**, **Equal to**, or **Not equal to**. Value The value to which the monitored consumer group *Metric* is compared. Buffer The delay behind real time to wait until a time window is considered for triggering (refer to :ref:`concepts` for more information). .. _brokers: Brokers ^^^^^^^ .. figure:: images/c3alertsbroker.png :align: center Broker clusters The cluster(s) to look at for a specific broker metric. Trigger name A name used to identify the trigger (for example 'Broker zookeeper down'). Uniqueness is not enforced but you should use different names to avoid confusion. Broker clusters One or many broker clusters to trigger based on conditions. .. note:: If multiple clusters are selected, the trigger will be *per cluster*, not aggregate, but not in all cases. See below for more details. Metric Some of the values in *Metric* are triggered on a per broker basis and some are only availbale cluster wide. The selection method in *Broker clusters* is by cluster, but there are specific metrics in the brokers of clusters that can cause triggers. .. important:: Any **broker** that meets the *Condition* below will trigger discretely Bytes in Number of bytes per second produced to this broker Bytes out Number of bytes per second fetched from this broker (does not account for internal replication traffic) .. note:: Prior to Kafka 0.11.0.0 the ``BytesOutPerSec`` accounted for traffic from the consumer as well as internal replication, this has been changed to only account for consumer traffic for this broker. Please adjust alerts accordingly. Production request latency .. include:: fragments/brokerProductionRequestLatency.rst Production request count Total number of produce requests to this broker (requests per minute) Fetch request latency .. include:: fragments/brokerFetchRequestLatency.rst .. important:: Any **cluster** that meets the *Condition* below will trigger Under replicated topic partitions .. include:: fragments/brokerClusterUnderReplicated.rst A trigger should be created for values ``> 0`` Offline topic partitions .. include:: fragments/brokerClusterOfflineTopicPartitions.rst A trigger should be created for values ``> 0`` ZooKeeper status Are brokers able to connect to ZooKeeper? 'Offline' / 'Online' are possible values. ZooKeeper expiration rate .. include:: fragments/brokerClusterZooKeeperExpires.rst Active controller count .. include:: fragments/brokerClusterActiveController.rst A trigger should be created for values ``!= 1`` Leader election rate .. include:: fragments/brokerClusterLeaderElection.rst Unclean election count .. include:: fragments/brokerClusterUncleanCount.rst A trigger should be created for values ``!= 0`` Condition The trigger will fire when *Condition* is true of the difference between the value of the metric being monitored and the value of the *Value* field. Possibly **Greater than**, **Less than**, **Equal to**, **Not equal to**, **Online**, or **Offline**, depending on the *Metric* selected. Value The value to which the broker *Metric* is compared. .. _actions: Actions Management ------------------ After creating a trigger_, you will be given the option to go to the action management page to associate it with one or more existing actions, or if none exist, create a new action. Before being able to send email actions you will need to enable emails, and properly :ref:`config |c3-short| to communicate with your SMTP server`. At the very least you will need to set .. sourcecode:: bash # this enables sending mail via c3 confluent.controlcenter.mail.enabled=true # this is the host name of your mail server confluent.controlcenter.mail.host.name=mymail.server # this is the port your mail server is running on confluent.controlcenter.mail.port=25 # we also recommend setting rest.listeners explicitly as well because # this will control the |c3-short| link that is embedded in the # body of any alert emails confluent.controlcenter.rest.listeners=control-center.server Clicking on the actions tab shows a summary of all configured actions: .. figure:: images/c3alertsactionsoverview.png :scale: 50% :align: center You can edit/delete an existing action using the edit/delete links in this table or create a new action via the "+ new action" button on the top right of the screen. New/Edit Action Form ^^^^^^^^^^^^^^^^^^^^ The form for creating and editing actions is identical. When editing, the form is pre-populated with the current action values. .. figure:: images/c3alertseditaction.png :scale: 50% :align: center A description of each field follows (all fields are required): Action name A name for the action (for example 'email ). Uniqueness is not enforced, but you should use different names to avoid confusion. Enabled/Disabled You may wish to temporarily disable actions. You can choose whether an action is currently enabled or disabled with this field. Triggers One or more triggers that will cause the action to be executed (refer to :ref:`concepts` for more information). Action The type of action to perform. Currently we only support sending emails, but in the future we may support other action types. Recipient email address The email address associated with this action. A message will be sent to this address each time the action is executed. Subject The subject line of the email associated with this action. Max send rate The maximum rate at which the action may be executed (refer to :ref:`concepts` for more information). Alert History ------------- Selecting the history tab shows a table summarizing every trigger that has fired that caused an action to be executed (note: it does not list every triggered event). .. figure:: images/c3alertshistory.png :scale: 50% :align: center You can see contextual information for some items by clicking the "view" link, or in the case of broker alerts, the click will navigate to the 'System Health' page for further diagnostics. .. figure:: images/c3alertsitem.png :scale: 50% :align: center