Improve support for slack recipients (#4765)

Fixes #3722 
Fixes #4755 

##### Summary

- Removed the default addition of '#' to the "channel", so it can support both channels and users
- Add the '#' if it's not already there in the recipient, to specify a channel (for backwards compatibility). 
- If the recipient is just a '#', netdata will not be sending the channel at all. This means that users only need to configure the channel or the user on the Slack webhook.

##### Component Name
health/notifications

##### Additional Information

3 files changed, 35 insertions, 14 deletions

diff --git a/health/notifications/alarm-notify.sh.in b/health/notifications/alarm-notify.sh.in
index ea82230971..ec2a1af69f 100755
--- a/health/notifications/alarm-notify.sh.in
+++ b/health/notifications/alarm-notify.sh.in
@@ -1476,9 +1476,21 @@ send_slack() {
 
     for channel in ${channels}
     do
+        # Default entry in the recipient is without a hash in front (backwards-compatible). Accept specification of channel or user.
+        if [ "${channel::1}" != "#" ] && [ "${channel::1}" != "@" ] ; then channel="#$channel"; fi
+        
+        # If channel is equal to "#" then do not send the channel attribute at all. Slack also defines channels and users in webhooks.
+        if [ "${channel}" = "#" ] ; then 
+            ch=""
+            chstr="without specifying a channel"
+        else
+            ch="\"channel\": \"${channel}\","
+            chstr="to '${channel}'"
+        fi
+        
         payload="$(cat <<EOF
         {
-            "channel": "#${channel}",
+            $ch
             "username": "netdata on ${host}",
             "icon_url": "${images_base_url}/images/banner-icon-144x144.png",
             "text": "${host} ${status_message}, \`${chart}\` (_${family}_), *${alarm}*",
@@ -1511,10 +1523,10 @@ EOF
         httpcode=$(docurl -X POST --data-urlencode "payload=${payload}" "${webhook}")
         if [ "${httpcode}" = "200" ]
         then
-            info "sent slack notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
+            info "sent slack notification for: ${host} ${chart}.${name} is ${status} ${chstr}"
             sent=$((sent + 1))
         else
-            error "failed to send slack notification for: ${host} ${chart}.${name} is ${status} to '${channel}', with HTTP error code ${httpcode}."
+            error "failed to send slack notification for: ${host} ${chart}.${name} is ${status} ${chstr}, with HTTP error code ${httpcode}."
         fi
     done
 
diff --git a/health/notifications/health_alarm_notify.conf b/health/notifications/health_alarm_notify.conf
index a997765a6a..27de95b840 100755
--- a/health/notifications/health_alarm_notify.conf
+++ b/health/notifications/health_alarm_notify.conf
@@ -324,7 +324,7 @@ DEFAULT_RECIPIENT_TELEGRAM=""
 # slack (slack.com) global notification options
 
 # multiple recipients can be given like this:
-#                  "CHANNEL1 CHANNEL2 ..."
+#                  "RECIPIENT1 RECIPIENT2 ..."
 
 # enable/disable sending slack notifications
 SEND_SLACK="YES"
@@ -335,9 +335,11 @@ SEND_SLACK="YES"
 # Get yours from: https://api.slack.com/incoming-webhooks
 SLACK_WEBHOOK_URL=""
 
-# if a role's recipients are not configured, a notification will be send to
-# this slack channel (empty = do not send a notification for unconfigured
-# roles):
+# if a role's recipients are not configured, a notification will be send to:
+# - A slack channel (syntax: '#channel' or 'channel')  
+# - A slack user (syntax: '@user')
+# - The channel or user defined in slack for the webhook (syntax: '#')
+# empty = do not send a notification for unconfigured roles 
 DEFAULT_RECIPIENT_SLACK=""
 
 #------------------------------------------------------------------------------
diff --git a/health/notifications/slack/README.md b/health/notifications/slack/README.md
index 7d44257ea4..836c273a94 100644
--- a/health/notifications/slack/README.md
+++ b/health/notifications/slack/README.md
@@ -17,7 +17,7 @@ Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system r
 # sending slack notifications
 
 # note: multiple recipients can be given like this:
-#                  "CHANNEL1 CHANNEL2 ..."
+#                  "RECIPIENT1 RECIPIENT2 ..."
 
 # enable/disable sending pushover notifications
 SEND_SLACK="YES"
@@ -27,19 +27,26 @@ SEND_SLACK="YES"
 # Without it, netdata cannot send slack notifications.
 SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXX/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
 
-# if a role recipient is not configured, a notification will be send to
-# this slack channel:
+# if a role's recipients are not configured, a notification will be send to:
+# - A slack channel (syntax: '#channel' or 'channel')  
+# - A slack user (syntax: '@user')
+# - The channel or user defined in slack for the webhook (syntax: '#')
+# empty = do not send a notification for unconfigured roles 
 DEFAULT_RECIPIENT_SLACK="alarms"
 
 ```
 
-You can define multiple channels like this: `alarms systems`.
-You can give different channels per **role** using these (at the same file):
+You can define multiple recipients like this: `# #alarms systems @myuser`. 
+This example will send the alarm to:
+- The recipient defined in slack for the webhook (not known to netdata)
+- The channel 'alarms'
+- The channel 'systems'
+- The user @myuser
+
+You can give different recipients per **role** using these (at the same file):
 
 ```
 role_recipients_slack[sysadmin]="systems"
 role_recipients_slack[dba]="databases systems"
 role_recipients_slack[webmaster]="marketing development"
 ```
-
-The keywords `systems`, `databases`, `marketing`, `development` are slack.com channels (they should already exist in slack).