Workers utilization charts (#12807)

* initial version of worker utilization

* working example

* without mutexes

* monitoring DBENGINE, ACLKSYNC, WEB workers

* added charts to monitor worker usage

* fixed charts units

* updated contexts

* updated priorities

* added documentation

* converted threads to stacked chart

* One query per query thread

* Revert "One query per query thread"

This reverts commit 6aeb391f5987c3c6ba2864b559fd7f0cd64b14d3.

* fixed priority for web charts

* read worker cpu utilization from proc

* read workers cpu utilization via /proc/self/task/PID/stat, so that we have cpu utilization even when the jobs are too long to finish within our update_every frequency

* disabled web server cpu utilization monitoring - it is now monitored by worker utilization

* tight integration of worker utilization to web server

* monitoring statsd worker threads

* code cleanup and renaming of variables

* contrained worker and statistics conflict to just one variable

* support for rendering jobs per type

* better priorities and removed the total jobs chart

* added busy time in ms per job type

* added proc.plugin monitoring, switch clock to MONOTONIC_RAW if available, global statistics now cleans up old worker threads

* isolated worker thread families

* added cgroups.plugin workers

* remove unneeded dimensions when then expected worker is just one

* plugins.d and streaming monitoring

* rebased; support worker_is_busy() to be called one after another

* added diskspace plugin monitoring

* added tc.plugin monitoring

* added ML threads monitoring

* dont create dimensions and charts that are not needed

* fix crash when job types are added on the fly

* added timex and idlejitter plugins; collected heartbeat statistics; reworked heartbeat according to the POSIX

* the right name is heartbeat for this chart

* monitor streaming senders

* added streaming senders to global stats

* prevent division by zero

* added clock_init() to external C plugins

* added freebsd and macos plugins

* added freebsd and macos to global statistics

* dont use new as a variable; address compiler warnings on FreeBSD and MacOS

* refactored contexts to be unique; added health threads monitoring

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>

2 files changed, 9 insertions, 0 deletions

diff --git a/parser/parser.c b/parser/parser.c
index 5137cb61cf..0707f71727 100644
--- a/parser/parser.c
+++ b/parser/parser.c
@@ -133,10 +133,13 @@ int parser_add_keyword(PARSER *parser, char *keyword, keyword_function func)
 
     tmp_keyword = callocz(1, sizeof(*tmp_keyword));
 
+    tmp_keyword->worker_job_id = parser->worker_job_ids++;
     tmp_keyword->keyword = strdupz(keyword);
     tmp_keyword->keyword_hash = keyword_hash;
     tmp_keyword->func[tmp_keyword->func_no++] = (void *) func;
 
+    worker_register_job_name(tmp_keyword->worker_job_id, tmp_keyword->keyword);
+
     tmp_keyword->next = parser->keyword;
     parser->keyword = tmp_keyword;
     return tmp_keyword->func_no;
@@ -265,10 +268,12 @@ inline int parser_action(PARSER *parser, char *input)
 
     uint32_t command_hash = simple_hash(command);
 
+    size_t worker_job_id;
     while(tmp_keyword) {
         if (command_hash == tmp_keyword->keyword_hash &&
                 (!strcmp(command, tmp_keyword->keyword))) {
                     action_function_list = &tmp_keyword->func[0];
+                    worker_job_id = tmp_keyword->worker_job_id;
                     break;
         }
         tmp_keyword = tmp_keyword->next;
@@ -284,12 +289,14 @@ inline int parser_action(PARSER *parser, char *input)
 #endif
     }
     else {
+        worker_is_busy(worker_job_id);
         while ((action_function = *action_function_list) != NULL) {
                 rc = action_function(words, parser->user, parser->plugins_action);
                 if (unlikely(rc == PARSER_RC_ERROR || rc == PARSER_RC_STOP))
                     break;                
                 action_function_list++;
         }
+        worker_is_idle();
     }
 
     if (likely(input == parser->buffer))
diff --git a/parser/parser.h b/parser/parser.h
index 8d11a90074..65a4e1ab30 100644
--- a/parser/parser.h
+++ b/parser/parser.h
@@ -54,6 +54,7 @@ typedef enum parser_input_type {
 typedef PARSER_RC (*keyword_function)(char **, void *, PLUGINSD_ACTION  *plugins_action);
 
 typedef struct parser_keyword {
+    size_t      worker_job_id;
     char        *keyword;
     uint32_t    keyword_hash;
     int         func_no;
@@ -67,6 +68,7 @@ typedef struct parser_data {
 } PARSER_DATA;
 
 typedef struct parser {
+    size_t worker_job_ids;
     uint8_t version;                // Parser version
     RRDHOST *host;
     void *input;                    // Input source e.g. stream