From 3161e32dc24d42e8da38f5b59d3d4481e5e9565d Mon Sep 17 00:00:00 2001 From: "Austin S. Hemmelgarn" Date: Wed, 12 Sep 2018 15:20:39 -0400 Subject: Add alarms for abnormally high load averages. (#4175) * Add alarms for abnormally high load averages. This adds reasonably conservative alarms to send alarts on abnormally high load averages. Such a situation may be indicative of a DoS attack, runaway processes, or simply use of underpowered hardware. This intentionally does not compute averages, as doing so would be redundant (we are dealing with load _averages_ after all), which makes the lookup lines look a bit odd in comparison to most other alarms. The actual alarm calculation is as-follows: * Compute the baseline trigger threshold. This is either 2 or the maximum number of CPU's that were present in the system over the last minute, whichever is higher. This special-cases single-CPU systems to be a bit less aggressive,a s they are more often over-committed than systems with multiple cores. * For the 15 minute load average, if the maximum value over the last minute is greater than twice the trigger threshold, issue a warning. * For the 5 minute load average, if the maximum value over the last minute is greater than four times the ttrigger value, issue a warning. * For the 1 minute load average, if the maximum value over the last minute is greater than eight times the trigger value, issue a warning. * For all the load averages, if the value is greater than twice the warning requirement, issue a critical alert. * Down-hysteriesis is provided so that each alarm only resets wheen the value goes below 7/8 of the value for that alarm status. * Each alarm is evaluated once per minute. This behavior should be suitable for most server type systems and many workstations, but may be a bit overaggressive for certain types of system (build systems for example). * Fixed calculations of the base trigger value. Credit goes to @ktsaou for pointing out how the original implementation was incorrect. * Update alarms with correct OS information. --- conf.d/Makefile.am | 1 + conf.d/health.d/load.conf | 56 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 conf.d/health.d/load.conf diff --git a/conf.d/Makefile.am b/conf.d/Makefile.am index a28b78ddf8..e1bc551122 100644 --- a/conf.d/Makefile.am +++ b/conf.d/Makefile.am @@ -112,6 +112,7 @@ dist_healthconfig_DATA = \ health.d/ipmi.conf \ health.d/isc_dhcpd.conf \ health.d/lighttpd.conf \ + health.d/load.conf \ health.d/mdstat.conf \ health.d/megacli.conf \ health.d/memcached.conf \ diff --git a/conf.d/health.d/load.conf b/conf.d/health.d/load.conf new file mode 100644 index 0000000000..11a89fa878 --- /dev/null +++ b/conf.d/health.d/load.conf @@ -0,0 +1,56 @@ + +# you can disable an alarm notification by setting the 'to' line to: silent + +# Calculate the base trigger point for the load average alarms. +# This is the maximum number of CPU's in the system over the past 1 +# minute, with a special case for a single CPU of setting the trigger at 2. +template: load_trigger + on: system.cpu + os: linux + hosts: * + calc: ($processors <= 2) ? ( 2 ) : ( $processors ) + units: cpus + every: 1m + info: trigger point for load average alarms + +# Send alarms if the load average is unusually high. +# These intentionally _do not_ calculate the average over the sampled +# time period because the values being checked already are averages. +template: load_average_15 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load15 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (1.75 * $load_trigger) : (2 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (3.5 * $load_trigger) : (4 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: fifteen-minute load average + to: sysadmin + +template: load_average_5 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load5 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (3.5 * $load_trigger) : (4 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (7 * $load_trigger) : (8 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: five-minute load average + to: sysadmin + +template: load_average_1 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load1 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (7 * $load_trigger) : (8 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (14 * $load_trigger) : (16 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: one-minute load average + to: sysadmin -- cgit v1.2.3