Add alarms for abnormally high load averages. (#4175)

* Add alarms for abnormally high load averages.

This adds reasonably conservative alarms to send alarts on abnormally
high load averages.  Such a situation may be indicative of a DoS attack,
runaway processes, or simply use of underpowered hardware.

This intentionally does not compute averages, as doing so would be
redundant (we are dealing with load _averages_ after all), which makes
the lookup lines look a bit odd in comparison to most other alarms.

The actual alarm calculation is as-follows:

* Compute the baseline trigger threshold.  This is either 2 or the
  maximum number of CPU's that were present in the system over the last
  minute, whichever is higher.  This special-cases single-CPU systems to
  be a bit less aggressive,a s they are more often over-committed than
  systems with multiple cores.
* For the 15 minute load average, if the maximum value over the last
  minute is greater than twice the trigger threshold, issue a warning.
* For the 5 minute load average, if the maximum value over the last
  minute is greater than four times the ttrigger value, issue a warning.
* For the 1 minute load average, if the maximum value over the last
  minute is greater than eight times the trigger value, issue a warning.
* For all the load averages, if the value is greater than twice the
  warning requirement, issue a critical alert.
* Down-hysteriesis is provided so that each alarm only resets wheen the
  value goes below 7/8 of the value for that alarm status.
* Each alarm is evaluated once per minute.

This behavior should be suitable for most server type systems and many
workstations, but may be a bit overaggressive for certain types of system
(build systems for example).

* Fixed calculations of the base trigger value.

Credit goes to @ktsaou for pointing out how the original implementation
was incorrect.

* Update alarms with correct OS information.

2 files changed, 57 insertions, 0 deletions

diff --git a/conf.d/Makefile.am b/conf.d/Makefile.am
index a28b78ddf8..e1bc551122 100644
--- a/conf.d/Makefile.am
+++ b/conf.d/Makefile.am
@@ -112,6 +112,7 @@ dist_healthconfig_DATA = \
     health.d/ipmi.conf \
     health.d/isc_dhcpd.conf \
     health.d/lighttpd.conf \
+    health.d/load.conf \
     health.d/mdstat.conf \
     health.d/megacli.conf \
     health.d/memcached.conf \
diff --git a/conf.d/health.d/load.conf b/conf.d/health.d/load.conf
new file mode 100644
index 0000000000..11a89fa878
--- /dev/null
+++ b/conf.d/health.d/load.conf
@@ -0,0 +1,56 @@
+
+# you can disable an alarm notification by setting the 'to' line to: silent
+
+# Calculate the base trigger point for the load average alarms.
+# This is the maximum number of CPU's in the system over the past 1
+# minute, with a special case for a single CPU of setting the trigger at 2.
+template: load_trigger
+      on: system.cpu
+      os: linux
+   hosts: *
+    calc: ($processors <= 2) ? ( 2 ) : ( $processors )
+   units: cpus
+   every: 1m
+    info: trigger point for load average alarms
+
+# Send alarms if the load average is unusually high.
+# These intentionally _do not_ calculate the average over the sampled
+# time period because the values being checked already are averages.
+template: load_average_15
+      on: system.load
+      os: linux
+   hosts: *
+  lookup: max -1m unaligned of load15
+   units: load
+   every: 1m
+    warn: $this > (($status >= $WARNING)  ? (1.75 * $load_trigger) : (2 * $load_trigger))
+    crit: $this > (($status == $CRITICAL) ? (3.5 * $load_trigger) : (4 * $load_trigger))
+   delay: down 15m multiplier 1.5 max 1h
+    info: fifteen-minute load average
+      to: sysadmin
+
+template: load_average_5
+      on: system.load
+      os: linux
+   hosts: *
+  lookup: max -1m unaligned of load5
+   units: load
+   every: 1m
+    warn: $this > (($status >= $WARNING)  ? (3.5 * $load_trigger) : (4 * $load_trigger))
+    crit: $this > (($status == $CRITICAL) ? (7 * $load_trigger) : (8 * $load_trigger))
+   delay: down 15m multiplier 1.5 max 1h
+    info: five-minute load average
+      to: sysadmin
+
+template: load_average_1
+      on: system.load
+      os: linux
+   hosts: *
+  lookup: max -1m unaligned of load1
+   units: load
+   every: 1m
+    warn: $this > (($status >= $WARNING)  ? (7 * $load_trigger) : (8 * $load_trigger))
+    crit: $this > (($status == $CRITICAL) ? (14 * $load_trigger) : (16 * $load_trigger))
+   delay: down 15m multiplier 1.5 max 1h
+    info: one-minute load average
+      to: sysadmin