Create a template for all dimensions (#6560)

* health_connection: Comments inside Health Config

To try to understand better what is necessary to change and where it is necessary
to change anything inside the health, I commented the functions inside this file"
"

* health_connection: Comments about Health in other files

This commit brings the rest of the comments that were missed for health"

* health_connection: Comments on health_log

I had to append more comments on health_log

* health_connection: Create a new variable

New variable is created to work with foreach

* health_connection:  Fix new option and doc

The first implementation of the 'foreach' had a problem, this fixes the error.
This commit also brings the updates for the documentation

* health_connection:  Understanding health

This commit is to save the place that I am working, it has the map to understand all the alam process

* health_connection:  Update map

I changed the position of the error message to identify the correct place to add new alarms

* health_connection: End of simple alarm

This commit finishes what is necessary to bring the same lookup for different dimensions  in one unique line

* health_connection:  Documentation and template steps

This commit brings the documentation missed for template and comments to help in the next
step of apply a template to create an alarm.

* health_connection: Restoring

After some tests, it was detected that the alarms were not working as expected

* health_connection: Fix bug and bring dimension to template

This commit brings a fix for an old Netdata bug, before this the Netdata always tried to create
a new entry in an index with the same id raising an error.
It also brings the possibility to use 'foreach' in  template

* health_connection: Fix cmake compilation

There was a problem with cmake compilation fixed by this commit

* health_connection: shell script

Finilize the shell script to test the PR

* health_connection: Remove debug message

During the development, I used some messages to understand the code
this commit removes the last message

* health_connection: Fix bugs

This commits fix bugs reported by tests

* health_connection: Alarm working

This commit brings the necessary change for the alarms work, but it is missing the unlink from the newest list

* health_connection: Template code written

This commit finishes the creation of alarm from template, but it was not tested yet.

* health_connection: Remove comments

I am removing the comments from this PR to bring back late

* health_connection: Remove lines

Another commit to restore the files before they to be commented

* health_connection: New alarm and remove messages

I am bringing a new alarm to test template with SP and removing comments used during the development

* health_connection: Functional test review

After to review the functional test script, it was necessary to small adjust to
test all the features available with the new version

* health_connection: Free structure

I am moving the free list for the correct place, the previous place was not safe

* health_connection: ShellCheck

This commit fixes the problems with shellcheck

* health_connection: FIx hash

This commit fix the hash calculation that was using wrong input

* health_connection: Fix message error

The system was showing a wronge message, because when we have foreach
the alarm created with templated is added in a second stage to the index

* health_connection: Fix documentation

In this commit I am fixing the grammar of the previous doc and bringing
two examples

* health_connection: Fix examples

This commit fix the last two examples that was brought in this PR

* health_connection: Fix example doc

When I brought the correct grammar in the last commit, I lost a mark

* health_connection: Grammar fix

Fixing grammar of the documentation

* health_connection: Memory leak

This commit fixes the memory leak that was present in the PR

* health_connection: Reload

This commit fix the problem that the alarms were not linked after
to receive a SIGUSR2

* health_connection: False Positive from codacy

Codacy was given a false positive, I changed the function to avoid it.

* health_connection: dead code

Remove dead code from the code.

* health_connection: Memory Leak

Remove memory leak when clean simple pattern

* health_connection: Script format

With this commit I am formatting the last message to return
for the default color on terminal

* health_connection: Script format 2

With this commit I am formatting the last message to return
for the default color on terminal

* health_connection: Script format 3

With this commit I am formatting the error message to return
for the default color on terminal

8 files changed, 175 insertions, 0 deletions

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 179b04864b..c549cbd473 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -8,6 +8,7 @@ CLEANFILES = \
 	acls/acl.sh \
 	urls/request.sh \
 	alarm_repetition/alarm.sh \
+	template_dimension/template_dim.sh \
 	$(NULL)
 
 include $(top_srcdir)/build/subst.inc
@@ -28,6 +29,7 @@ dist_noinst_DATA = \
     acls/acl.sh.in \
     urls/request.sh.in \
     alarm_repetition/alarm.sh.in \
+    template_dimension/template_dim.sh.in \
 	$(NULL)
 
 dist_plugins_SCRIPTS = \
@@ -35,6 +37,7 @@ dist_plugins_SCRIPTS = \
 	acls/acl.sh \
 	urls/request.sh \
 	alarm_repetition/alarm.sh \
+	template_dimension/template_dim.sh \
 	$(NULL)
 
 dist_noinst_SCRIPTS = \
diff --git a/tests/template_dimension/system_cpu.conf.alarm_foreach b/tests/template_dimension/system_cpu.conf.alarm_foreach
new file mode 100644
index 0000000000..21a8cbb723
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.alarm_foreach
@@ -0,0 +1,8 @@
+ alarm: dev_dim_template
+    on: system.cpu
+    os: linux
+lookup: sum -3s at 0 every 3 percentage foreach system,user,nice
+ units: %
+ every: 1s
+  warn: $this > 1
+  crit: $this > 4
diff --git a/tests/template_dimension/system_cpu.conf.alarm_foreach_sp b/tests/template_dimension/system_cpu.conf.alarm_foreach_sp
new file mode 100644
index 0000000000..fdd19e8134
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.alarm_foreach_sp
@@ -0,0 +1,8 @@
+ alarm: dev_dim_template
+    on: system.cpu
+    os: linux
+lookup: sum -3s at 0 every 3 percentage foreach *
+ units: %
+ every: 1s
+  warn: $this > 1
+  crit: $this > 4
diff --git a/tests/template_dimension/system_cpu.conf.template_alarm b/tests/template_dimension/system_cpu.conf.template_alarm
new file mode 100644
index 0000000000..2bd12a1012
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.template_alarm
@@ -0,0 +1,26 @@
+template: dev_dim_template_system
+      on: system.cpu
+      os: linux
+  lookup: sum -3s at 0 every 3 percentage of system
+   units: %
+   every: 1s
+    warn: $this > 1
+    crit: $this > 4
+
+template: dev_dim_template_user
+      on: system.cpu
+      os: linux
+  lookup: sum -3s at 0 every 3 percentage of user
+   units: %
+   every: 1s
+    warn: $this > 1
+    crit: $this > 4
+
+template: dev_dim_template_nice
+      on: system.cpu
+      os: linux
+  lookup: sum -3s at 0 every 3 percentage of nice
+   units: %
+   every: 1s
+    warn: $this > 1
+    crit: $this > 4
diff --git a/tests/template_dimension/system_cpu.conf.template_foreach b/tests/template_dimension/system_cpu.conf.template_foreach
new file mode 100644
index 0000000000..c75c15b2a1
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.template_foreach
@@ -0,0 +1,8 @@
+template: dev_dim_template
+      on: system.cpu
+      os: linux
+  lookup: sum -3s at 0 every 3 percentage foreach system,user,nice
+   units: %
+   every: 1s
+    warn: $this > 1
+    crit: $this > 4
diff --git a/tests/template_dimension/system_cpu.conf.template_foreach_sp b/tests/template_dimension/system_cpu.conf.template_foreach_sp
new file mode 100644
index 0000000000..f50a832c93
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.template_foreach_sp
@@ -0,0 +1,8 @@
+ template: dev_dim_template
+       on: system.cpu
+       os: linux
+   lookup: sum -3s at 0 every 3 percentage foreach *
+    units: %
+    every: 1s
+     warn: $this > 1
+     crit: $this > 4
diff --git a/tests/template_dimension/system_cpu.conf.unique_alarm b/tests/template_dimension/system_cpu.conf.unique_alarm
new file mode 100644
index 0000000000..0f38b6e12d
--- /dev/null
+++ b/tests/template_dimension/system_cpu.conf.unique_alarm
@@ -0,0 +1,26 @@
+ alarm: dev_dim_template_system
+    on: system.cpu
+    os: linux
+lookup: sum -3s at 0 every 3 percentage of system
+ units: %
+ every: 1s
+  warn: $this > 1
+  crit: $this > 4
+
+ alarm: dev_dim_template_user
+    on: system.cpu
+    os: linux
+lookup: sum -3s at 0 every 3 percentage of user
+ units: %
+ every: 1s
+  warn: $this > 1
+  crit: $this > 4
+
+ alarm: dev_dim_template_nice
+    on: system.cpu
+    os: linux
+lookup: sum -3s at 0 every 3 percentage of nice
+ units: %
+ every: 1s
+  warn: $this > 1
+  crit: $this > 4
diff --git a/tests/template_dimension/template_dim.sh.in b/tests/template_dimension/template_dim.sh.in
new file mode 100644
index 0000000000..88978fd849
--- /dev/null
+++ b/tests/template_dimension/template_dim.sh.in
@@ -0,0 +1,88 @@
+#!/bin/bash
+
+#The health directory to put the alarms
+HEALTHDIR="@configdir_POST@/health.d/"
+
+#the current time
+OUTDIR="alarms"
+QUERY="/api/v1/alarms?all"
+MURL="http://localhost:19999$QUERY"
+
+#error messages
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+NOCOLOR='\033[0m'
+
+ALARMTEST="dev_dim_template"
+
+change_alarm_file() {
+  if [ -f "$1" ]; then
+    rm "$1"
+  fi
+
+  #copy keeping the permissions
+  cp -a "$2" "$3"
+}
+
+netdata_test_download() {
+   grep "HTTP/1.1 200 OK" "$1" 2>/dev/null 1>/dev/null
+   TEST="$?"
+   if [ "$TEST" -ne "0" ]; then
+      echo -e "${RED} Error to get the alarm log. ${NOCOLOR}"
+      exit 1
+   fi
+
+   TOTALARM=$(grep "$ALARMTEST" "$2" | grep name | cut -d: -f2 | grep -c "$ALARMTEST")
+
+  if [ "$TOTALARM" -ne "$3" ]; then
+     echo -e "${RED} The number of actives alarms with the name $SYSTEMALARM is wrong ${NOCOLOR}"
+     exit 1
+  fi
+}
+
+get_the_logs() {
+  curl -v -k --create-dirs -o "$OUTDIR/$1.out" "$MURL" 2> "$OUTDIR/$1.err"
+  netdata_test_download "$OUTDIR/$1.err" "$OUTDIR/$1.out" "$2"
+}
+
+process_data() {
+  netdata -D &
+  NETDATAPID=$!
+  echo -e "${NOCOLOR}Sleeping during 15 seconds to create alarms"
+  sleep 15
+  kill $NETDATAPID
+  get_the_logs "$1" "$2"
+}
+
+mkdir "$OUTDIR"
+CREATEDIR="$?"
+if [ "$CREATEDIR" -ne "0" ]; then
+  echo  -e "${RED}Cannot create the output directory, it already exists. The test will overwrite previous results. ${NOCOLOR}"
+fi
+
+if [ -n "$1" ]; then
+  MURL="$1$QUERY"
+fi
+
+change_alarm_file "./0" "system_cpu.conf.unique_alarm" "$HEALTHDIR/dim_double_without_template.conf"
+process_data "double_without_template" 3 "$HEALTHDIR/dim_double_without_template.conf"
+
+change_alarm_file "$HEALTHDIR/dim_double_without_template.conf" "system_cpu.conf.alarm_foreach" "$HEALTHDIR/dim_foreach_without_template.conf"
+process_data "foreach_without_template" 3 "$HEALTHDIR/dim_foreach_without_template.conf"
+
+change_alarm_file "$HEALTHDIR/dim_foreach_without_template.conf" "system_cpu.conf.alarm_foreach_sp" "$HEALTHDIR/dim_foreach_without_template_sp.conf"
+process_data "foreach_without_template" 10 "$HEALTHDIR/dim_foreach_without_template_sp.conf"
+
+change_alarm_file "$HEALTHDIR/dim_foreach_without_template_sp.conf" "system_cpu.conf.template_alarm" "$HEALTHDIR/dim_double_with_template.conf"
+process_data "double_with_template" 3 "$HEALTHDIR/dim_double_with_template.conf"
+
+change_alarm_file "$HEALTHDIR/dim_double_with_template.conf" "system_cpu.conf.template_foreach" "$HEALTHDIR/dim_foreach_with_template.conf"
+process_data "foreach_with_template" 3 "$HEALTHDIR/dim_foreach_with_template.conf"
+
+change_alarm_file "$HEALTHDIR/dim_foreach_with_template.conf" "system_cpu.conf.template_foreach_sp" "$HEALTHDIR/dim_foreach_with_template_sp.conf"
+process_data "foreach_with_template" 10 "$HEALTHDIR/dim_foreach_with_template_sp.conf"
+
+rm "$HEALTHDIR/dim_foreach_with_template_sp.conf"
+rm -rf "$OUTDIR"
+
+echo  -e "${GREEN} all the tests were sucessful ${NOCOLOR}"