diff options
author | Austin S. Hemmelgarn <ahferroin7@gmail.com> | 2018-09-12 15:20:39 -0400 |
---|---|---|
committer | Costa Tsaousis <costa@tsaousis.gr> | 2018-09-12 22:20:39 +0300 |
commit | 3161e32dc24d42e8da38f5b59d3d4481e5e9565d (patch) | |
tree | 40427e2da81523a922599cc4623be4532504319a /conf.d | |
parent | 760c5ee4e6ffd3c1503edd5ce5fda79dd0a19cad (diff) |
Add alarms for abnormally high load averages. (#4175)
* Add alarms for abnormally high load averages.
This adds reasonably conservative alarms to send alarts on abnormally
high load averages. Such a situation may be indicative of a DoS attack,
runaway processes, or simply use of underpowered hardware.
This intentionally does not compute averages, as doing so would be
redundant (we are dealing with load _averages_ after all), which makes
the lookup lines look a bit odd in comparison to most other alarms.
The actual alarm calculation is as-follows:
* Compute the baseline trigger threshold. This is either 2 or the
maximum number of CPU's that were present in the system over the last
minute, whichever is higher. This special-cases single-CPU systems to
be a bit less aggressive,a s they are more often over-committed than
systems with multiple cores.
* For the 15 minute load average, if the maximum value over the last
minute is greater than twice the trigger threshold, issue a warning.
* For the 5 minute load average, if the maximum value over the last
minute is greater than four times the ttrigger value, issue a warning.
* For the 1 minute load average, if the maximum value over the last
minute is greater than eight times the trigger value, issue a warning.
* For all the load averages, if the value is greater than twice the
warning requirement, issue a critical alert.
* Down-hysteriesis is provided so that each alarm only resets wheen the
value goes below 7/8 of the value for that alarm status.
* Each alarm is evaluated once per minute.
This behavior should be suitable for most server type systems and many
workstations, but may be a bit overaggressive for certain types of system
(build systems for example).
* Fixed calculations of the base trigger value.
Credit goes to @ktsaou for pointing out how the original implementation
was incorrect.
* Update alarms with correct OS information.
Diffstat (limited to 'conf.d')
-rw-r--r-- | conf.d/Makefile.am | 1 | ||||
-rw-r--r-- | conf.d/health.d/load.conf | 56 |
2 files changed, 57 insertions, 0 deletions
diff --git a/conf.d/Makefile.am b/conf.d/Makefile.am index a28b78ddf8..e1bc551122 100644 --- a/conf.d/Makefile.am +++ b/conf.d/Makefile.am @@ -112,6 +112,7 @@ dist_healthconfig_DATA = \ health.d/ipmi.conf \ health.d/isc_dhcpd.conf \ health.d/lighttpd.conf \ + health.d/load.conf \ health.d/mdstat.conf \ health.d/megacli.conf \ health.d/memcached.conf \ diff --git a/conf.d/health.d/load.conf b/conf.d/health.d/load.conf new file mode 100644 index 0000000000..11a89fa878 --- /dev/null +++ b/conf.d/health.d/load.conf @@ -0,0 +1,56 @@ + +# you can disable an alarm notification by setting the 'to' line to: silent + +# Calculate the base trigger point for the load average alarms. +# This is the maximum number of CPU's in the system over the past 1 +# minute, with a special case for a single CPU of setting the trigger at 2. +template: load_trigger + on: system.cpu + os: linux + hosts: * + calc: ($processors <= 2) ? ( 2 ) : ( $processors ) + units: cpus + every: 1m + info: trigger point for load average alarms + +# Send alarms if the load average is unusually high. +# These intentionally _do not_ calculate the average over the sampled +# time period because the values being checked already are averages. +template: load_average_15 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load15 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (1.75 * $load_trigger) : (2 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (3.5 * $load_trigger) : (4 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: fifteen-minute load average + to: sysadmin + +template: load_average_5 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load5 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (3.5 * $load_trigger) : (4 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (7 * $load_trigger) : (8 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: five-minute load average + to: sysadmin + +template: load_average_1 + on: system.load + os: linux + hosts: * + lookup: max -1m unaligned of load1 + units: load + every: 1m + warn: $this > (($status >= $WARNING) ? (7 * $load_trigger) : (8 * $load_trigger)) + crit: $this > (($status == $CRITICAL) ? (14 * $load_trigger) : (16 * $load_trigger)) + delay: down 15m multiplier 1.5 max 1h + info: one-minute load average + to: sysadmin |