From 99ebfa43cc739dac07f559e23581c30c271e5f96 Mon Sep 17 00:00:00 2001 From: Joel Hans Date: Fri, 18 Dec 2020 10:34:51 -0700 Subject: Add centralized Cloud notifications to core docs (#10374) * Add centralized Cloud notifs to core docs * Cleanup and copyediting --- docs/monitor/enable-notifications.md | 154 +++++++++++++++++------------------ docs/monitor/view-active-alarms.md | 4 +- 2 files changed, 75 insertions(+), 83 deletions(-) (limited to 'docs') diff --git a/docs/monitor/enable-notifications.md b/docs/monitor/enable-notifications.md index 60dfddb539..68beba53e7 100644 --- a/docs/monitor/enable-notifications.md +++ b/docs/monitor/enable-notifications.md @@ -1,16 +1,61 @@ -# Enable notifications +# Enable alarm notifications -Netdata comes with a [notification system](/health/notifications/README.md) that supports more than a dozen services, -such as email, Slack, PagerDuty, Twilio, Amazon SNS, Discord, and much more. You can enable as many platforms as you -want, and configure them to match your organization's needs with features like role-based notifications. +Netdata offers two ways to receive alarm notifications on external platforms. These methods work independently _or_ in +parallel, which means you can enable both at the same time to send alarm notifications to any number of endpoints. -## Supported notification platforms +Both methods use a node's health alarms to generate the content of alarm notifications. Read the doc on [configuring +alarms](/docs/monitor/configure-alarms.md) to change the preconfigured thresholds or to create tailored alarms for your +infrastructure. + +Netdata Cloud offers [centralized alarm notifications](#netdata-cloud) via email, which leverages the health status +information already streamed to Netdata Cloud from claimed nodes to send notifications to those who have enabled them. + +The Netdata Agent has a [notification system](#netdata-agent) that supports more than a dozen services, such as email, +Slack, PagerDuty, Twilio, Amazon SNS, Discord, and much more. + +For example, use centralized alarm notifications in Netdata Cloud for immediate, zero-configuration alarm notifications +for your team, then configure individual nodes send notifications to a PagerDuty endpoint for an automated incident +response process. + +## Netdata Cloud + +Netdata Cloud's [centralized alarm notifications](https://learn.netdata.cloud/docs/cloud/monitoring/notifications/) is a +zero-configuration way to get notified when an anomaly or incident strikes any node or application in your +infrastructure. The advantage of using centralized alarm notifications from Netdata Cloud is that you don't have to +worry about configuring each node in your infrastructure. + +To enable centralized alarm notifications for a Space, click on **Manage Space** in the left-hand menu, then click on +the **Notifications** tab. Click the toggle switch next to **E-mail** to enable this notification method. + +Next, enable notifications on a user level by clicking on your profile icon, then **Profile** in the dropdown. The +**Notifications** tab reveals rich management settings, including the ability to enable/disable methods entirely or +choose what types of notifications to receive from each War Room. + +![Enabling and configuring alarm notifications in Netdata +Cloud](https://user-images.githubusercontent.com/1153921/101936280-93c50900-3b9d-11eb-9ba0-d6927fa872b7.gif) + +See the [centralized alarm notifications](https://learn.netdata.cloud/docs/cloud/monitoring/notifications/) reference +doc for further details about what information is conveyed in an email notification, flood protection, and more. + +## Netdata Agent + +The Netdata Agent's [notification system](/health/notifications/README.md) runs on every node and dispatches +notifications based on configured endpoints and roles. You can enable multiple endpoints on any one node _and_ use Agent +notifications in parallel with centralized alarm notifications in Netdata Cloud. + +> ❗ If you want to enable notifications from multiple nodes in your infrastructure, each running the Netdata Agent, you +> must configure each node individually. + +Below, we'll use [Slack notifications](#enable-slack-notifications) as an example of the process of enabling any +notification platform. + +### Supported notification endpoints - [**alerta.io**](/health/notifications/alerta/README.md) - [**Amazon SNS**](/health/notifications/awssns/README.md) @@ -38,78 +83,16 @@ want, and configure them to match your organization's needs with features like r - [**Telegram**](/health/notifications/telegram/README.md) - [**Twilio**](/health/notifications/twilio/README.md) -This doc covers enabling email and Slack notifications, but the same process applies to enabling any other notification -platform. - -## Enable email notifications - -To use [email notifications](/health/notifications/email/README.md), you need -[`sendmail`](http://www.postfix.org/sendmail.1.html) or an equivalent installed on your system. - -Edit the `health_alarm_notify.conf` file, which resides in your Netdata [config -directory](/docs/configure/nodes.md#netdata-config-directory). - -```bash -sudo ./edit-config health_alarm_notify.conf -``` - -Look for the following lines: - -```conf -# if a role recipient is not configured, an email will be sent to: -DEFAULT_RECIPIENT_EMAIL="root" -# to receive only critical alarms, set it to "root|critical" -``` - -Change the value of `DEFAULT_RECIPIENT_EMAIL` to the email address at which you'd like to receive notifications. +### Enable Slack notifications -```conf -# if a role recipient is not configured, an email will be sent to: -DEFAULT_RECIPIENT_EMAIL="me@example.com" -# to receive only critical alarms, set it to "root|critical" -``` - -Test email notifications system by first becoming the Netdata user and then asking Netdata to send a test alarm: - -```bash -sudo su -s /bin/bash netdata -/usr/libexec/netdata/plugins.d/alarm-notify.sh test -``` - -You should see output similar to this: - -```bash -# SENDING TEST WARNING ALARM TO ROLE: sysadmin -2019-10-17 18:23:38: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is WARNING to 'me@example.com' -# OK - -# SENDING TEST CRITICAL ALARM TO ROLE: sysadmin -2019-10-17 18:23:38: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is CRITICAL to 'me@example.com' -# OK - -# SENDING TEST CLEAR ALARM TO ROLE: sysadmin -2019-10-17 18:23:39: alarm-notify.sh: INFO: sent email notification for: hostname test.chart.test_alarm is CLEAR to 'me@example.com' -# OK -``` - -Check your email. You should receive three separate emails for each health status change: `WARNING`, `CRITICAL`, and -`CLEAR`. - -See the [email notifications](/health/notifications/email/README.md) doc for more options and information. - -## Enable Slack notifications - -If you're one of the many who spend their workday getting pinged with GIFs by your colleagues, why not add Netdata -notifications to the mix? It's a great way to immediately see, collaborate around, and respond to anomalies in your -infrastructure. - -To get Slack notifications working, you first need to add an [incoming -webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) to the channel of your choice. Click the green **Add to -Slack** button, choose the channel, and click the **Add Incoming WebHooks Integration** button. +First, [Add an incoming webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) in Slack for the channel where you +want to see alarm notifications from Netdata. Click the green **Add to Slack** button, choose the channel, and click the +**Add Incoming WebHooks Integration** button. On the following page, you'll receive a **Webhook URL**. That's what you'll need to configure Netdata, so keep it handy. -Time to dive back into your `health_alarm_notify.conf` file: +Navigate to your [Netdata config directory](/docs/configure/nodes.md#netdata-config-directory) and use `edit-config` to +open the `health_alarm_notify.conf` file: ```bash sudo ./edit-config health_alarm_notify.conf @@ -128,25 +111,34 @@ to send a notification to the channel you configured with the incoming webhook. DEFAULT_RECIPIENT_SLACK="#" ``` -Time to test the notifications again: +To test Slack notifications, switch to the Netdata user. ```bash sudo su -s /bin/bash netdata +``` + +Next, run the `alarm-notify` script using the `test` option. + +```bash /usr/libexec/netdata/plugins.d/alarm-notify.sh test ``` You should receive three notifications in your Slack channel for each health status change: `WARNING`, `CRITICAL`, and `CLEAR`. -See the [Slack notifications](/health/notifications/slack/README.md) doc for more options and information. +See the [Agent Slack notifications](/health/notifications/slack/README.md) doc for more options and information. ## What's next? -Learn more about Netdata's notifications system in the [notifications](/health/notifications/README.md) docs. +Now that you have health entities configured to your infrastructure's needs and notifications to inform you of anomalies +or incidents, your health monitoring setup is complete. -Now that you have health entities configured to your infrastructure's needs, and notifications to inform you of -anomalies, you have everything you need to monitor the health of your infrastructure. To make your dashboards most -useful during root cause analysis, you can use Netdata's [distributed data +To make your dashboards most useful during root cause analysis, use Netdata's [distributed data architecture](/docs/store/distributed-data-architecture.md) for the best-in-class performance and scalability. +### Related reference documentation + +- [Netdata Cloud · Alarm notifications](https://learn.netdata.cloud/docs/cloud/monitoring/notifications/) +- [Netdata Agent · Notifications](/health/notifications/README.md) + [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fmonitor%2Fenable-notifications&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) diff --git a/docs/monitor/view-active-alarms.md b/docs/monitor/view-active-alarms.md index 55ba139cd9..8837e48adb 100644 --- a/docs/monitor/view-active-alarms.md +++ b/docs/monitor/view-active-alarms.md @@ -69,7 +69,7 @@ With the information that appears on Netdata Cloud and the local dashboard about alarms](/docs/monitor/configure-alarms.md) to match your infrastructure's needs or your team's goals. If you're happy with the pre-configured alarms, skip ahead to [enable -notifications](/docs/monitor/enable-notifications.md) to instantly see alarms in email, Slack, PagerDuty, Twilio, and -many other platforms. +notifications](/docs/monitor/enable-notifications.md) to use Netdata Cloud's centralized alarm notifications and/or +per-node notifications to endpoints like Slack, PagerDuty, Twilio, and more. [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fmonitor%2Fview-active-alarms&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>) -- cgit v1.2.3