diff options
author | Costa Tsaousis <costa@netdata.cloud> | 2022-10-05 14:13:46 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-10-05 14:13:46 +0300 |
commit | 8fc3b351a2e7fc96eced8f924de2e9cec9842128 (patch) | |
tree | bde41c66573ccaf8876c280e00742cc6096b587c /libnetdata | |
parent | 6850878e697d66dc90b9af1e750b22238c63c292 (diff) |
Allow netdata plugins to expose functions for querying more information about specific charts (#13720)
* function renames and code cleanup in popen.c; no actual code changes
* netdata popen() now opens both child process stdin and stdout and returns FILE * for both
* pass both input and output to parser structures
* updated rrdset to call custom functions
* RRDSET FUNCTION leading calls for both sync and async operation
* put RRDSET functions to a separate file
* added format and timeout at function definition
* support for synchronous (internal plugins) and asynchronous (external plugins and children) functions
* /api/v1/function endpoint
* functions are now attached to the host and there is a dictionary view per chart
* functions implemented at plugins.d
* remove the defer until keyword hook from plugins.d when it is done
* stream sender implementation of functions
* sanitization of all functions so that certain characters are only allowed
* strictier sanitization
* common max size
* 1st working plugins.d example
* always init inflight dictionary
* properly destroy dictionaries to avoid parallel insertion of items
* add more debugging on disconnection reasons
* add more debugging on disconnection reasons again
* streaming receiver respects newlines
* dont use the same fp for both streaming receive and send
* dont free dbengine memory with internal checks
* make sender proceed in the buffer
* added timing info and garbage collection at plugins.d
* added info about routing nodes
* added info about routing nodes with delay
* added more info about delays
* added more info about delays again
* signal sending thread to wake up
* streaming version labeling and commented code to support capabilities
* added functions to /api/v1/data, /api/v1/charts, /api/v1/chart, /api/v1/info
* redirect top output to stdout
* address coverity findings
* fix resource leaks of popen
* log attempts to connect to individual destinations
* better messages
* properly parse destinations
* try to find a function from the most matching to the least matching
* log added streaming destinations
* rotate destinations bypassing a node in the middle that does not accept our connection
* break the loops properly
* use typedef to define callbacks
* capabilities negotiation during streaming
* functions exposed upstream based on capabilities; compression disabled per node persisting reconnects; always try to connect with all capabilities
* restore functionality to lookup functions
* better logging of capabilities
* remove old versions from capabilities when a newer version is there
* fix formatting
* optimization for plugins.d rrdlabels to avoid creating and destructing dictionaries all the time
* delayed health initialization for rrddim and rrdset
* cleanup health initialization
* fix for popen() not returning the right value
* add health worker jobs for initializing rrdset and rrddim
* added content type support for functions; apps.plugin permanent function to display all the processes
* fixes for functions parameters parsing in apps.plugin
* fix for process matching in apps.plugiin
* first working function for apps.plugin
* Dashboard ACL is disabled for functions; Function errors are all in JSON format
* apps.plugin function processes returns json table
* use json_escape_string() to escape message
* fix formatting
* apps.plugin exposes all its metrics to function processes
* fix json formatting when filtering out some rows
* reopen the internal pipe of rrdpush in case of errors
* misplaced statement
* do not use buffer->len
* support for GLOBAL functions (functions that are not linked to a chart
* added /api/v1/functions endpoint; removed format from the FUNCTIONS api;
* swagger documentation about the new api end points
* added plugins.d documentation about functions
* never re-close a file
* remove uncessesary ifdef
* fixed issues identified by codacy
* fix for null label value
* make edit-config copy-and-paste friendly
* Revert "make edit-config copy-and-paste friendly"
This reverts commit 54500c0e0a97f65a0c66c4d34e966f6a9056698e.
* reworked sender handshake to fix coverity findings
* timeout is zero, for both send_timeout() and recv_timeout()
* properly detect that parent closed the socket
* support caching of function responses; limit function response to 10MB; added protection from malformed function responses
* disabled excessive logging
* added units to apps.plugin function processes and normalized all values to be human readable
* shorter field names
* fixed issues reported
* fixed apps.plugin error response; tested that pluginsd can properly handle faulty responses
* use double linked list macros for double linked list management
* faster apps.plugin function printing by minimizing file operations
* added memory percentage
* fix compatibility issues with older compilers and FreeBSD
* rrdpush sender code cleanup; rrhost structure cleanup from sender flags and variables;
* fix letftover variable in ifdef
* apps.plugin: do not call detach from the thread; exit immediately when input is broken
* exclude AR charts from health
* flush cleaner; prefer sender output
* clarity
* do not fill the cbuffer if not connected
* fix
* dont enabled host->sender if streaming is not enabled; send host label updates to parent;
* functions are only available through ACLK
* Prepared statement reports only in dev mode
* fix AR chart detection
* fix for streaming not being enabling itself
* more cleanup of sender and receiver structures
* moved read-only flags and configuration options to rrdhost->options
* fixed merge with master
* fix for incomplete rename
* prevent service thread from working on charts that are being collected
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Diffstat (limited to 'libnetdata')
-rw-r--r-- | libnetdata/buffer/buffer.c | 48 | ||||
-rw-r--r-- | libnetdata/buffer/buffer.h | 1 | ||||
-rw-r--r-- | libnetdata/circular_buffer/circular_buffer.c | 5 | ||||
-rw-r--r-- | libnetdata/circular_buffer/circular_buffer.h | 1 | ||||
-rw-r--r-- | libnetdata/config/appconfig.c | 8 | ||||
-rw-r--r-- | libnetdata/dictionary/dictionary.c | 27 | ||||
-rw-r--r-- | libnetdata/dictionary/dictionary.h | 6 | ||||
-rw-r--r-- | libnetdata/inlined.h | 41 | ||||
-rw-r--r-- | libnetdata/libnetdata.c | 133 | ||||
-rw-r--r-- | libnetdata/libnetdata.h | 9 | ||||
-rw-r--r-- | libnetdata/popen/popen.c | 437 | ||||
-rw-r--r-- | libnetdata/popen/popen.h | 26 | ||||
-rw-r--r-- | libnetdata/required_dummies.h | 1 | ||||
-rw-r--r-- | libnetdata/socket/security.c | 45 | ||||
-rw-r--r-- | libnetdata/socket/security.h | 14 | ||||
-rw-r--r-- | libnetdata/socket/socket.c | 101 | ||||
-rw-r--r-- | libnetdata/socket/socket.h | 28 | ||||
-rw-r--r-- | libnetdata/threads/threads.c | 2 |
18 files changed, 633 insertions, 300 deletions
diff --git a/libnetdata/buffer/buffer.c b/libnetdata/buffer/buffer.c index 3d29c30255..d0940588fc 100644 --- a/libnetdata/buffer/buffer.c +++ b/libnetdata/buffer/buffer.c @@ -148,6 +148,54 @@ void buffer_print_ll(BUFFER *wb, long long value) buffer_print_llu(wb, value); } +static unsigned char bits03_to_hex[16] = { + [0] = '0', + [1] = '1', + [2] = '2', + [3] = '3', + [4] = '4', + [5] = '5', + [6] = '6', + [7] = '7', + [8] = '8', + [9] = '9', + [10] = 'A', + [11] = 'B', + [12] = 'C', + [13] = 'D', + [14] = 'E', + [15] = 'F' +}; + +void buffer_print_llu_hex(BUFFER *wb, unsigned long long value) +{ + unsigned char buffer[sizeof(unsigned long long) * 2 + 2 + 1]; // 8 bytes * 2 + '0x' + '\0' + unsigned char *e = &buffer[sizeof(unsigned long long) * 2 + 2]; + unsigned char *p = e; + + *p-- = '\0'; + *p-- = bits03_to_hex[value & 0xF]; + value >>= 4; + if(value) { + *p-- = bits03_to_hex[value & 0xF]; + value >>= 4; + + while(value) { + *p-- = bits03_to_hex[value & 0xF]; + value >>= 4; + + if(value) { + *p-- = bits03_to_hex[value & 0xF]; + value >>= 4; + } + } + } + *p-- = 'x'; + *p = '0'; + + buffer_fast_strcat(wb, (char *)p, e - p); +} + void buffer_fast_strcat(BUFFER *wb, const char *txt, size_t len) { if(unlikely(!txt || !*txt)) return; diff --git a/libnetdata/buffer/buffer.h b/libnetdata/buffer/buffer.h index c134cc9aca..fe45b1c6d6 100644 --- a/libnetdata/buffer/buffer.h +++ b/libnetdata/buffer/buffer.h @@ -79,6 +79,7 @@ extern char *print_number_llu_r_smart(char *str, unsigned long long uvalue); extern void buffer_print_llu(BUFFER *wb, unsigned long long uvalue); extern void buffer_print_ll(BUFFER *wb, long long value); +extern void buffer_print_llu_hex(BUFFER *wb, unsigned long long value); static inline void buffer_need_bytes(BUFFER *buffer, size_t needed_free_size) { if(unlikely(buffer->size - buffer->len < needed_free_size)) diff --git a/libnetdata/circular_buffer/circular_buffer.c b/libnetdata/circular_buffer/circular_buffer.c index f074996d9d..c791b420b0 100644 --- a/libnetdata/circular_buffer/circular_buffer.c +++ b/libnetdata/circular_buffer/circular_buffer.c @@ -89,3 +89,8 @@ size_t cbuffer_next_unsafe(struct circular_buffer *buf, char **start) { } return buf->size - buf->read; } + +void cbuffer_flush(struct circular_buffer*buf) { + buf->write = 0; + buf->read = 0; +}
\ No newline at end of file diff --git a/libnetdata/circular_buffer/circular_buffer.h b/libnetdata/circular_buffer/circular_buffer.h index e5addc50f4..ba3ddd9cd9 100644 --- a/libnetdata/circular_buffer/circular_buffer.h +++ b/libnetdata/circular_buffer/circular_buffer.h @@ -14,5 +14,6 @@ extern int cbuffer_add_unsafe(struct circular_buffer *buf, const char *d, size_t extern void cbuffer_remove_unsafe(struct circular_buffer *buf, size_t num); extern size_t cbuffer_next_unsafe(struct circular_buffer *buf, char **start); extern size_t cbuffer_available_size_unsafe(struct circular_buffer *buf); +extern void cbuffer_flush(struct circular_buffer*buf); #endif diff --git a/libnetdata/config/appconfig.c b/libnetdata/config/appconfig.c index 1288366da8..ba1f059d57 100644 --- a/libnetdata/config/appconfig.c +++ b/libnetdata/config/appconfig.c @@ -32,8 +32,8 @@ _CONNECTOR_INSTANCE *add_connector_instance(struct section *connector, struct se local_ci = callocz(1, sizeof(struct _connector_instance)); local_ci->instance = instance; local_ci->connector = connector; - strncpy(local_ci->instance_name, instance->name, CONFIG_MAX_NAME); - strncpy(local_ci->connector_name, connector->name, CONFIG_MAX_NAME); + strncpyz(local_ci->instance_name, instance->name, CONFIG_MAX_NAME); + strncpyz(local_ci->connector_name, connector->name, CONFIG_MAX_NAME); local_ci->next = global_connector_instance; global_connector_instance = local_ci; @@ -686,14 +686,14 @@ int appconfig_load(struct config *root, char *filename, int overwrite_used, cons int rc; rc = is_valid_connector(s, 0); if (likely(rc)) { - strncpy(working_connector, s, CONFIG_MAX_NAME); + strncpyz(working_connector, s, CONFIG_MAX_NAME); s = s + rc + 1; if (unlikely(!(*s))) { _connectors++; sprintf(buffer, "instance_%d", _connectors); s = buffer; } - strncpy(working_instance, s, CONFIG_MAX_NAME); + strncpyz(working_instance, s, CONFIG_MAX_NAME); working_connector_section = NULL; if (unlikely(appconfig_section_find(root, working_instance))) { error("Instance (%s) already exists", working_instance); diff --git a/libnetdata/dictionary/dictionary.c b/libnetdata/dictionary/dictionary.c index 88de7df3af..9d19d5965d 100644 --- a/libnetdata/dictionary/dictionary.c +++ b/libnetdata/dictionary/dictionary.c @@ -249,6 +249,9 @@ void dictionary_register_conflict_callback(DICTIONARY *dict, bool (*conflict_cal if(unlikely(is_view_dictionary(dict))) fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ ); + internal_error(!(dict->options & DICT_OPTION_DONT_OVERWRITE_VALUE), "DICTIONARY: registering conflict callback without DICT_OPTION_DONT_OVERWRITE_VALUE"); + dict->options |= DICT_OPTION_DONT_OVERWRITE_VALUE; + dictionary_hooks_allocate(dict); dict->hooks->conflict_callback = conflict_callback; dict->hooks->conflict_callback_data = data; @@ -457,6 +460,10 @@ static inline REFCOUNT DICTIONARY_ITEM_REFCOUNT_GET(DICTIONARY *dict, DICTIONARY return (REFCOUNT)__atomic_load_n(&item->refcount, __ATOMIC_SEQ_CST); } +static inline REFCOUNT DICTIONARY_ITEM_REFCOUNT_GET_SOLE(DICTIONARY_ITEM *item) { + return (REFCOUNT)__atomic_load_n(&item->refcount, __ATOMIC_SEQ_CST); +} + // ---------------------------------------------------------------------------- // callbacks execution @@ -617,6 +624,12 @@ static void ll_recursive_unlock(DICTIONARY *dict, char rw) { } } +void dictionary_write_lock(DICTIONARY *dict) { + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE); +} +void dictionary_write_unlock(DICTIONARY *dict) { + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE); +} static inline void dictionary_index_lock_rdlock(DICTIONARY *dict) { if(unlikely(is_dictionary_single_threaded(dict))) @@ -1494,7 +1507,9 @@ static bool dictionary_free_all_resources(DICTIONARY *dict, size_t *mem, bool fo #endif // destroy the index + dictionary_index_lock_wrlock(dict); index_size += hashtable_destroy_unsafe(dict); + dictionary_index_lock_unlock(dict); ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE); DICTIONARY_ITEM *item = dict->items.list; @@ -1797,6 +1812,7 @@ size_t dictionary_destroy(DICTIONARY *dict) { if(!dict) return 0; + dict_flag_set(dict, DICT_FLAG_DESTROYED); DICTIONARY_STATS_DICT_DESTRUCTIONS_PLUS1(dict); size_t referenced_items = dictionary_referenced_items(dict); @@ -1936,20 +1952,23 @@ void dictionary_acquired_item_release(DICTIONARY *dict, DICT_ITEM_CONST DICTIONA // get the name/value of an item const char *dictionary_acquired_item_name(DICT_ITEM_CONST DICTIONARY_ITEM *item) { - api_internal_check(NULL, item, true, false); return item_get_name(item); } void *dictionary_acquired_item_value(DICT_ITEM_CONST DICTIONARY_ITEM *item) { - // we allow the item to be NULL here - api_internal_check(NULL, item, true, true); - if(likely(item)) return item->shared->value; return NULL; } +size_t dictionary_acquired_item_references(DICT_ITEM_CONST DICTIONARY_ITEM *item) { + if(likely(item)) + return DICTIONARY_ITEM_REFCOUNT_GET_SOLE(item); + + return 0; +} + // ---------------------------------------------------------------------------- // DEL an item diff --git a/libnetdata/dictionary/dictionary.h b/libnetdata/dictionary/dictionary.h index 3cbaa87f07..c51746dbff 100644 --- a/libnetdata/dictionary/dictionary.h +++ b/libnetdata/dictionary/dictionary.h @@ -180,7 +180,7 @@ extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_set_and_acquire_item_advanced #define dictionary_view_set_and_acquire_item(dict, name, master_item) dictionary_view_set_and_acquire_item_advanced(dict, name, -1, master_item) extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_view_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item); #define dictionary_view_set(dict, name, master_item) dictionary_view_set_advanced(dict, name, -1, master_item) -extern void *dictionary_view_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item); +extern void *dictionary_view_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICT_ITEM_CONST DICTIONARY_ITEM *master_item); // ---------------------------------------------------------------------------- // Get an item from the dictionary @@ -211,6 +211,7 @@ extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY extern const char *dictionary_acquired_item_name(DICT_ITEM_CONST DICTIONARY_ITEM *item); extern void *dictionary_acquired_item_value(DICT_ITEM_CONST DICTIONARY_ITEM *item); +extern size_t dictionary_acquired_item_references(DICT_ITEM_CONST DICTIONARY_ITEM *item); // ---------------------------------------------------------------------------- // Traverse (walk through) the items of the dictionary. @@ -252,6 +253,9 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)( #define DICTIONARY_LOCK_WRITE 'w' #define DICTIONARY_LOCK_REENTRANT 'z' +extern void dictionary_write_lock(DICTIONARY *dict); +extern void dictionary_write_unlock(DICTIONARY *dict); + typedef DICTFE_CONST struct dictionary_foreach { DICTIONARY *dict; // the dictionary upon we work diff --git a/libnetdata/inlined.h b/libnetdata/inlined.h index 5c265fc015..788a0312b6 100644 --- a/libnetdata/inlined.h +++ b/libnetdata/inlined.h @@ -42,24 +42,12 @@ static inline uint32_t simple_uhash(const char *name) { return hval; } -static inline int simple_hash_strcmp(const char *name, const char *b, uint32_t *hash) { - unsigned char *s = (unsigned char *) name; - uint32_t hval = 0x811c9dc5; - int ret = 0; - while (*s) { - if(!ret) ret = *s - *b++; - hval *= 16777619; - hval ^= (uint32_t) *s++; - } - *hash = hval; - return ret; -} - static inline int str2i(const char *s) { int n = 0; - char c, negative = (*s == '-'); + char c, negative = (char)(*s == '-'); + const char *e = &s[30]; // max number of character to iterate - for(c = (negative)?*(++s):*s; c >= '0' && c <= '9' ; c = *(++s)) { + for(c = (char)((negative)?*(++s):*s); c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -73,8 +61,9 @@ static inline int str2i(const char *s) { static inline long str2l(const char *s) { long n = 0; char c, negative = (*s == '-'); + const char *e = &s[30]; // max number of character to iterate - for(c = (negative)?*(++s):*s; c >= '0' && c <= '9' ; c = *(++s)) { + for(c = (negative)?*(++s):*s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -88,7 +77,9 @@ static inline long str2l(const char *s) { static inline uint32_t str2uint32_t(const char *s) { uint32_t n = 0; char c; - for(c = *s; c >= '0' && c <= '9' ; c = *(++s)) { + const char *e = &s[30]; // max number of character to iterate + + for(c = *s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -98,7 +89,9 @@ static inline uint32_t str2uint32_t(const char *s) { static inline uint64_t str2uint64_t(const char *s) { uint64_t n = 0; char c; - for(c = *s; c >= '0' && c <= '9' ; c = *(++s)) { + const char *e = &s[30]; // max number of character to iterate + + for(c = *s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -108,7 +101,9 @@ static inline uint64_t str2uint64_t(const char *s) { static inline unsigned long str2ul(const char *s) { unsigned long n = 0; char c; - for(c = *s; c >= '0' && c <= '9' ; c = *(++s)) { + const char *e = &s[30]; // max number of character to iterate + + for(c = *s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -118,7 +113,9 @@ static inline unsigned long str2ul(const char *s) { static inline unsigned long long str2ull(const char *s) { unsigned long long n = 0; char c; - for(c = *s; c >= '0' && c <= '9' ; c = *(++s)) { + const char *e = &s[30]; // max number of character to iterate + + for(c = *s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } @@ -137,7 +134,9 @@ static inline long long str2ll(const char *s, char **endptr) { long long n = 0; char c; - for(c = *s; c >= '0' && c <= '9' ; c = *(++s)) { + const char *e = &s[30]; // max number of character to iterate + + for(c = *s; c >= '0' && c <= '9' && s < e ; c = *(++s)) { n *= 10; n += c - '0'; } diff --git a/libnetdata/libnetdata.c b/libnetdata/libnetdata.c index 5b6c541edf..ee5e4851da 100644 --- a/libnetdata/libnetdata.c +++ b/libnetdata/libnetdata.c @@ -1535,6 +1535,121 @@ char *find_and_replace(const char *src, const char *find, const char *replace, c return value; } +inline int pluginsd_space(char c) { + switch(c) { + case ' ': + case '\t': + case '\r': + case '\n': + case '=': + return 1; + + default: + return 0; + } +} + +inline int config_isspace(char c) +{ + switch (c) { + case ' ': + case '\t': + case '\r': + case '\n': + case ',': + return 1; + + default: + return 0; + } +} + +// split a text into words, respecting quotes +inline int quoted_strings_splitter(char *str, char **words, int max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover) +{ + char *s = str, quote = 0; + int i = 0, rec = 0; + char *recover = recover_input; + + // skip all white space + while (unlikely(custom_isspace(*s))) + s++; + + // check for quote + if (unlikely(*s == '\'' || *s == '"')) { + quote = *s; // remember the quote + s++; // skip the quote + } + + // store the first word + words[i++] = s; + + // while we have something + while (likely(*s)) { + // if it is escape + if (unlikely(*s == '\\' && s[1])) { + s += 2; + continue; + } + + // if it is quote + else if (unlikely(*s == quote)) { + quote = 0; + if (recover && rec < max_recover) { + recover_location[rec++] = s; + *recover++ = *s; + } + *s = ' '; + continue; + } + + // if it is a space + else if (unlikely(quote == 0 && custom_isspace(*s))) { + // terminate the word + if (recover && rec < max_recover) { + if (!rec || recover_location[rec-1] != s) { + recover_location[rec++] = s; + *recover++ = *s; + } + } + *s++ = '\0'; + + // skip all white space + while (likely(custom_isspace(*s))) + s++; + + // check for quote + if (unlikely(*s == '\'' || *s == '"')) { + quote = *s; // remember the quote + s++; // skip the quote + } + + // if we reached the end, stop + if (unlikely(!*s)) + break; + + // store the next word + if (likely(i < max_words)) + words[i++] = s; + else + break; + } + + // anything else + else + s++; + } + + // terminate the words + memset(&words[i], 0, (max_words - i) * sizeof (char *)); + + return i; +} + +inline int pluginsd_split_words(char *str, char **words, int max_words, char *recover_input, char **recover_location, int max_recover) +{ + return quoted_strings_splitter(str, words, max_words, pluginsd_space, recover_input, recover_location, max_recover); +} bool bitmap256_get_bit(BITMAP256 *ptr, uint8_t idx) { if (unlikely(!ptr)) @@ -1550,3 +1665,21 @@ void bitmap256_set_bit(BITMAP256 *ptr, uint8_t idx, bool value) { else ptr->data[idx / 64] &= ~(1ULL << (idx % 64)); } + +bool run_command_and_copy_output_to_stdout(const char *command, int max_line_length) { + pid_t pid; + FILE *fp = netdata_popen(command, &pid, NULL); + + if(fp) { + char buffer[max_line_length + 1]; + while (fgets(buffer, max_line_length, fp)) + fprintf(stdout, "%s", buffer); + } + else { + error("Failed to execute command '%s'.", command); + return false; + } + + netdata_pclose(NULL, fp, pid); + return true; +} diff --git a/libnetdata/libnetdata.h b/libnetdata/libnetdata.h index 7c84359d82..2b60b5366b 100644 --- a/libnetdata/libnetdata.h +++ b/libnetdata/libnetdata.h @@ -388,6 +388,15 @@ typedef struct bitmap256 { extern bool bitmap256_get_bit(BITMAP256 *ptr, uint8_t idx); extern void bitmap256_set_bit(BITMAP256 *ptr, uint8_t idx, bool value); +#define COMPRESSION_MAX_MSG_SIZE 0x4000 +#define PLUGINSD_LINE_MAX (COMPRESSION_MAX_MSG_SIZE - 1024) +extern int config_isspace(char c); +extern int pluginsd_space(char c); +int quoted_strings_splitter(char *str, char **words, int max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover); +extern int pluginsd_split_words(char *str, char **words, int max_words, char *recover_string, char **recover_location, int max_recover); + +extern bool run_command_and_copy_output_to_stdout(const char *command, int max_line_length); + extern void netdata_cleanup_and_exit(int ret) NORETURN; extern void send_statistics(const char *action, const char *action_result, const char *action_data); extern char *netdata_configured_host_prefix; diff --git a/libnetdata/popen/popen.c b/libnetdata/popen/popen.c index eaeffd32dd..b67d6b096b 100644 --- a/libnetdata/popen/popen.c +++ b/libnetdata/popen/popen.c @@ -2,81 +2,129 @@ #include "../libnetdata.h" -static pthread_mutex_t myp_lock; -static int myp_tracking = 0; +// ---------------------------------------------------------------------------- +// popen with tracking -struct mypopen { +static pthread_mutex_t netdata_popen_tracking_mutex; +static bool netdata_popen_tracking_enabled = false; + +struct netdata_popen { pid_t pid; - struct mypopen *next; - struct mypopen *prev; + struct netdata_popen *next; + struct netdata_popen *prev; }; -static struct mypopen *mypopen_root = NULL; +static struct netdata_popen *netdata_popen_root = NULL; // myp_add_lock takes the lock if we're tracking. -static void myp_add_lock(void) { - if (myp_tracking == 0) +static void netdata_popen_tracking_lock(void) { + if(!netdata_popen_tracking_enabled) return; - netdata_mutex_lock(&myp_lock); + netdata_mutex_lock(&netdata_popen_tracking_mutex); } // myp_add_unlock release the lock if we're tracking. -static void myp_add_unlock(void) { - if (myp_tracking == 0) +static void netdata_popen_tracking_unlock(void) { + if(!netdata_popen_tracking_enabled) return; - netdata_mutex_unlock(&myp_lock); + netdata_mutex_unlock(&netdata_popen_tracking_mutex); } // myp_add_locked adds pid if we're tracking. // myp_add_lock must have been called previously. -static void myp_add_locked(pid_t pid) { - struct mypopen *mp; - - if (myp_tracking == 0) +static void netdata_popen_tracking_add_pid_unsafe(pid_t pid) { + if(!netdata_popen_tracking_enabled) return; - mp = mallocz(sizeof(struct mypopen)); + struct netdata_popen *mp; + + mp = mallocz(sizeof(struct netdata_popen)); mp->pid = pid; - mp->next = mypopen_root; - mp->prev = NULL; - if (mypopen_root != NULL) - mypopen_root->prev = mp; - mypopen_root = mp; - netdata_mutex_unlock(&myp_lock); + DOUBLE_LINKED_LIST_PREPEND_UNSAFE(netdata_popen_root, mp, prev, next); } // myp_del deletes pid if we're tracking. -static void myp_del(pid_t pid) { - struct mypopen *mp; - - if (myp_tracking == 0) +static void netdata_popen_tracking_del_pid(pid_t pid) { + if(!netdata_popen_tracking_enabled) return; - netdata_mutex_lock(&myp_lock); - for (mp = mypopen_root; mp != NULL; mp = mp->next) { - if (mp->pid == pid) { - if (mp->next != NULL) - mp->next->prev = mp->prev; - if (mp->prev != NULL) - mp->prev->next = mp->next; - if (mypopen_root == mp) - mypopen_root = mp->next; - freez(mp); + struct netdata_popen *mp; + + netdata_mutex_lock(&netdata_popen_tracking_mutex); + + DOUBLE_LINKED_LIST_FOREACH_FORWARD(netdata_popen_root, mp, prev, next) { + if(unlikely(mp->pid == pid)) break; - } } - if (mp == NULL) + if(mp) { + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(netdata_popen_root, mp, prev, next); + freez(mp); + } + else error("Cannot find pid %d.", pid); - netdata_mutex_unlock(&myp_lock); + netdata_mutex_unlock(&netdata_popen_tracking_mutex); } -#define PIPE_READ 0 -#define PIPE_WRITE 1 +// netdata_popen_tracking_init() should be called by apps which act as init +// (pid 1) so that processes created by mypopen and mypopene +// are tracked. This enables the reaper to ignore processes +// which will be handled internally, by calling myp_reap, to +// avoid issues with already reaped processes during wait calls. +// +// Callers should call myp_free() to clean up resources. +void netdata_popen_tracking_init(void) { + info("process tracking enabled."); + netdata_popen_tracking_enabled = true; + + if (netdata_mutex_init(&netdata_popen_tracking_mutex) != 0) + fatal("netdata_popen_tracking_init() mutex init failed."); +} + +// myp_free cleans up any resources allocated for process +// tracking. +void netdata_popen_tracking_cleanup(void) { + if(!netdata_popen_tracking_enabled) + return; + + netdata_mutex_lock(&netdata_popen_tracking_mutex); + netdata_popen_tracking_enabled = false; + + while(netdata_popen_root) { + struct netdata_popen *mp = netdata_popen_root; + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(netdata_popen_root, mp, prev, next); + freez(mp); + } + + netdata_mutex_unlock(&netdata_popen_tracking_mutex); +} + +// myp_reap returns 1 if pid should be reaped, 0 otherwise. +int netdata_popen_tracking_pid_shoud_be_reaped(pid_t pid) { + if(!netdata_popen_tracking_enabled) + return 0; + + netdata_mutex_lock(&netdata_popen_tracking_mutex); + + int ret = 1; + struct netdata_popen *mp; + DOUBLE_LINKED_LIST_FOREACH_FORWARD(netdata_popen_root, mp, prev, next) { + if(unlikely(mp->pid == pid)) { + ret = 0; + break; + } + } + + netdata_mutex_unlock(&netdata_popen_tracking_mutex); + return ret; +} + +// ---------------------------------------------------------------------------- +// helpers static inline void convert_argv_to_string(char *dst, size_t size, const char *spawn_argv[]) { int i; @@ -89,118 +137,185 @@ static inline void convert_argv_to_string(char *dst, size_t size, const char *sp } } +// ---------------------------------------------------------------------------- +// the core of netdata popen + /* * Returns -1 on failure, 0 on success. When POPEN_FLAG_CREATE_PIPE is set, on success set the FILE *fp pointer. */ -static int custom_popene(volatile pid_t *pidptr, char **env, uint8_t flags, FILE **fpp, const char *command, const char *spawn_argv[]) { +#d |