| Age | Commit message (Collapse) | Author |
|---|
|
Improve the metadata detection for containers. The system_info structure has been updated to hold separate copies of OS_NAME, OS_ID, OS_ID_LIKE, OS_VERSION, OS_VERSION_ID and OS_DETECTION for both the container environment and the host. This new information is communicated through the /api/v1/info endpoint. For the streaming interface a partial copy of the info is carried until the stream protocol is upgraded. The anonymous_statistics script has been updated to carry the new data to Google Analytics. Some minor improvements have been made to OS-X / FreeBSD detection, and the detection of virtualization. The docs have been updated to explain how to pass the host environment to the docker container running Netdata.
|
|
* add swagger docu for `fixed_width_lbl` and `fxed_width_val`
|
|
|
|
|
|
* Add labels to the JSON exporting connector
* Add labels to the Graphite exporting connector
* Add labels to the OpenTSDB telnet exporting connector
* Add labels to the OpenTSDB HTTP exporting connector
* Replace control characters in JSON strings
* Add unit tests
|
|
* Remove host labels from the Swagger specification
* Remove host labels from the api responses
|
|
Initial work on host labels from the dedicated branch. Includes work for issues #7096, #7400, #7411, #7369, #7410, #7458, #7459, #7412 and #7408 by @vlvkobal, @thiagoftsm, @cakrit and @amoss.
|
|
* Start of testing partial requests.
Need to stash this to checkout a PR to test.
* Disambiguated error messages during header validation.
The mocking has blown up in the linker, need to wipe out repo local changes and restart from a known good state.
* Test failures.
CMocka is really not designed for parametric tests which is making it difficult to test the http validation properly.
We have some problems in the web_client.c code that are causing early failures in the testing sequence, and it is
causing CMocka to abort the sequence. Need to try a different approach to building the tests...
* Pedantic style pass.
* Test generation.
There must be another value hidden in the system that CMocka uses. This sets up 3278 tests but the results from
cmocka_run_group_tests_name show 0 tests were run.
* The problem was the "helper"-macro.
Calling CMocka directly, moved the setup/teardown into explicit fixtures. Successfully runs the family of tests over
the same (empty) state.
* Parameterised family of tests runs.
The api_next() acts as a counter, the least significant digit is the prefix_len using the web_buffer in the test_family
struct as a template to walk throufh. The most significant digit is the number of headers to use in the request.
Checked that this walk executes correctly and all the tests run before putting the test payloads back in. We trigger a
failure about 3-4 tests in that takes down the process. Currently investigating which parts are not mocked correctly.
* Pedantic style pass
* Adding a mocking for fatal.
That weird thing with the linker has happened again, need to clean repo and rebuild fresh.
* Full test sequence executes.
The test parameter counter jammed after a failure - we cannot rely on anything in the main test body being executed
after we call the functionailty under test. A failure will skip the rest of the execution.
Moved the counter stepping to the top of the function (i.e. it is now a ++i instead of a i++). Adjusted the initial
state to compensate.
This now steps through all of the test-sequence, but it raises an ugly issue - the post-test cleanup will not be
executed on a failure.
TODO:
* Move the test-state into the test_family.
* Do the clean-up of the previous test (if necesarry) in the step function.
* Fix the assertion on the web_client state.
* Pedantic style pass
* Test state is now in the test_family.
This addresses the issue with leaking on failure and not performing clean-up - we don't really care about memory leaks
during unit-testing, but we do care about reseting the system-under-test back to a known state to guarantee
independence across the tests. The clean-up is now triggered in api_next().
* Flip the wait flag assertions.
Partial requests should leave the web_client waiting to receive more data.
* Fixing ACL flags in test-driver.
This makes some tests pass - but far too many. Probably need a proper debugging function to show the request / response
in a readable format.
* Result from the api mocking.
Setting a successful return code in the api mocking makes the non-partial tests pass.
Zero'ing out the web_client before use has not fixed the initialization errors, there is still some history on the
parse_tries that needs to be tracked down.
Some of the other errors are spurious - they result from stream multiplexing in the testdriver - be careful with less.
* Fix warnings.
Switched the build configuration to CFLAGS="-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all
-DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1".
The memset introduced last night to zero out the initial web_client state had transposed parameters. Now that the
state is initially zero before hitting the http request processing most issues have disappeared. There are 3000+
passing tests and 48 boundary cases to track down.
* Pushing log entries from each test into a buffer.
This will allow suppression of logs from tests that pass.
* Switched to a unique test definition structure per test.
This cleans up the code as it means that a list of tests can be constructed during the first walk through the parameter
space. There is no need to walk the space twice and keep both walks aligned. Removed the cmocka_unit_test macro and
build the CMUnitTest structures directly -> this allows a real name per test instead of the procedure name.
The walking/step function api_info_next has been folded back into the test procedure as it is simpler to walk the list
in the shared test state.
Current TODO:
* There is a bug, the check on the wait state in the buffer is not being handled properly, investigate why
everything fails.
* The results don't match the old code, are we handing the correct web_buffer to each tested piece of code?
* Capture the test success state -> dump the log buffer on failures.
* State is properly passed through the tests.
Spent a long time chasing a horrible bug that seems to be inside CMocka? The state parameter being passed to each
unit test is different on each call, i.e. it looks like a unique void** where the void * (*state) has been overwritten
with the original value on each iteration through the testing loop. This behaviour does not match the CMocka source
code, which does thread the given valud through the unit test calls. It could be a side-effect of the memory
check-pointing, but the net-effect is that we cannot change the shared state between tests. It can be set in the
setup-fixture and used in each test, but not altered for the subsequent test.
This took a long time to diagnose - the fix is simple, we just share the state in a global pointer. This shared state
is used to walk through the list of test_def structures so that each unit-test knows where it is in the parameter-
space.
* With the correct state the bug in triggering the correct assertions is gone.
* Dump out the buffered logs on test failure.
* The only failing case (relative to these assertions) are the ulta-short partial-requests.
* Check the web_client->mode is set properly.
* Style pass
* Checking values passed to the API despatch point.
* Disabled the parametric tests to do some low-level testings.
Later on both sets of tests will be active. While the low-level url encoding tests are being developed the dynamically
generated set is disabled to make the output easier to read.
Working through the W3C URL spec, against RFC3986 and comparing the cases in available url-parsing test-suites to build
our test-suite.
* Start of the URL test-suite.
The percent-decoding in the current implementation is in the wrong place - it happens too early and causes
non-delimitor characters in the URL to be treated as delimitors. Current unit-tests seem to cover the range of
checks that we need CMocka to make. The handling of output is a little awkward - need something like the dynamic
cases that can output the log on a failure or skip it on a pass.
* Raw material for low-level testing.
* Adding more families in here is getting too messy.
About to switch over to multiple testdrivers.
* Need to clean repo to work around wrapping failures in CMocka.
* CMocka is not compatible with LTO.
The weird wrapping issues that come and go are as a result of LTO. My typical netdata-installer command-line that I use
to reboot the project state disables LTO, while my normal autoreconf / configure command-line does not causing this
problem to reappear seemingly-randomly. To build a single test-driver target this works:
autoreconf -ivf && CFLAGS='-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1' ./configure --disable-lto && make web/api/tests/web_api_valid_urls
The actual change in this commit is just a bug-fix.
* Ripping out the parameterized test generator.
Each of the URL cases is slightly and subtly different. This can't be done using the parameterization and will need a
healthy dose of cut and paste.
CMocka does not recognise the mocking for mysendfile, which is necessary to capture the exit route from the URL
parsing.
* Weird bug in CMocka?
For some reason CMocka will not mock out the mysendfile() procedure. We need to mock this to capture the behaviour of
the URL parsing as it is one of the exit paths. The wrapping is setup the same way as for the procedures so I cannot
see any reason that the library would not overwrite the calls. The only difference that I can find is that mysendfile
is in the unit being tested and the other mocked procedures are in different translation units.
This should not make a difference, but we have to disable LTO to get CMocka to work and the symbol patchs is some kind
of linker hack so there could be an issue if LTO is not running and the patch target is inside the same translation
unit.
Hiding it for now with a #ifndef UNIT_TESTING, which then compiles find and control flow hits the mock...
* Converting the ascii comments into unit_tests.
* More nasty cases for unit testing.
The commented out case will trigger a buffer overflow in the netdata agent and crash it.
* Last of the individual unit tests planned before the demo.
* Removing warnings.
* Switching on the rest of the parametric set - the other case with CRs.
* Fix Travis build failure under docker.
* Change the name of a define so it does not collide with existing testing in Travis.
* Add CMocka unit tests to CMake
* Linting pass
* Adding RFC comment to test.
* Buffer overflow checks on the captured logs.
This fixes the seg-fault seen by @vlvkobal and @thiagoftsm during testing.
* Chasing down other valgrind reports.
This gets rid of all of the uninitialised variable warnings. We stil have a memory leak, the headers that are set
during the unit testing switch on compression. This causes the web_client code to call deflatInit2 and allocate
structures for the compressor. We do not have a matching call to deflateEnd anywhere in the code so the memory leaks.
* Cleaning up a comment.
* Fixing review comments from @vlvkobal.
Also noticed that the buffer overflow fix this morning was killing the logfile output, fixed this as well.
* Addressing @thiagoftsm's concerns about the changing number of failures.
Switched the log dump for failing cases to repr().
Found a bug in the test case generator (not storing the flag for `\r`.
Verified that the 58 failing cases are the correct set of failures for the tested code.
|
|
* Use 4 spaces for indentation of non-recipe lines in Makefile.am files
* Be more consistent in the use of space before = in Makefile.am files
|
|
* Start of unit-test for http request processing.
A CMocka test-driver has been added to replay the live-capture headers against the http request processing code. Enough stubs / mocking code have been written to check that we can inject a chosen raw request into the web_client code and check how it was processed. A parameterised buffer building function can setup requests for testing in `web_buffer` structures. Each endpoint in the API will be checked against a variety of http requests. This PoC demonstrates enough functionality to show building the request on demand for each test-case and cleaning up afterwards.
|
|
This fixes the "structural error" messages that were introduced in commit 7b3f17d32425...
|
|
New testing tool for the web API.
We are calling this a "fuzzer" until a better name is suggested. This tool reads the swagger definitions of the API and parses the format of the requests and responses. The tool can generate randomized requests, which are sent to a netdata host, and then validate the json responses against the schema defined in the swagger.
A traditional fuzzer only produces a single bit of information about each test (did the target system crash). This tool verifies that the call into the API produced a valid response structure, which produces more information about the correct functioning of the host.
This current version performs a small sweep through the API calls as that is sufficient to find some incorrect response codes, and for testing the URL parser in the next issue (#7229) . A future update (in the next sprint) will add options to perform a deeper scan that brute-forces the parameter-space of the API, and combine it with our standard approach to stress-testing.
|
|
Documenting the structure of the /web/api/data responses (Fixes #7005)
Swagger2.0 semi-supports this - they understand the sub-typing but do not show
the sub-types in the model of the response directly. There are comments in
each of the cases to guide the user towards the correct part of the model. For each
kind of format request we have a data model of the response.
Miscellaneous style improvements and typo fixes.
|
|
* clang_warnings: Fix unecessary comparison
Netdata was verifying whether a pointer that will never be NULL could be NULL.
This commit removes this
* clang_warnings: Fix unecessary comparison
Netdata was doing another unecessary comparison in other file
* clang_warnings: Unecessary parenthesis
This commit removes the excess of parenthesis in a file
* clang_warnings: Remove unecessary initialization
Remove from json file a initial set that is overwritten few lines late
* clang_warnings: Comments
Fix comments on top of the function
* clang_warnings: Missing Cast
Volatile variable generates warnings with Clang sometimes, so it
was necessary to cast variables
* clang_warnings: Return from previous
Considering the possible problems given by the solution,
I am returning for the previous stage
|
|
* issue #3925 implement optional fixed size badges
* add docu related to #3925
* explicit cast (thiagoftsm) an split to multiline
* implement clipping of text when fixed_width_label enabled
* Update web/api/badges/README.md
improved docu based on suggestion from @joelhans
Co-Authored-By: Joel Hans <joel.g.hans@gmail.com>
* update docu per req. of @thiagoftsm
|
|
|
|
* Restore original alignment behaviour of RRDR
|
|
warnings (#6880)
* remove dollar sign from bash code
* remove dollar sign from sh
* clean up uninstall doc
* remove dollar sign from shell code
* fix remark after removing dollar sign
* fix HAProxy dollar sign and clean up remark
|
|
* remove "Netdata is using # MB of memory on HOSTNAME for # hour, # minutes, and # seconds of real-time history."
* Added "memory_mode" key to the get "charts" API call
* don't show db engine tip when user has it already installed
* add back hostname information
* add oxford comma (only for db-engine users)
* update main.js hash
* <b> --> <strong> (but only in sidebar info, main.js)
|
|
* health_connection: http_error_pattern
This commit brings an unique pattern for the Netdata webserver errors,
now Netdata uses define for all web error
* http_error_pattern: API v1
This PR also brings the pattern for the web_api_v1.c
|
|
* Variable Granularity support for data collection in the dbengine.
* Variable Granularity support for data collection in the daemon.
* Added tests to validate the data being queried after having been collected by changing data collection interval
* Fix memory corruption
* Updated database engine documentation about data collection frequency behaviour
|
|
##### Summary
This is implementation of a prerequisite for the requested feature #6536 (Generate an overall status badge/chart for the health of category)
##### Component Name
web/api/
health/
##### Details
Provide a new, `alarm_count` API call that returns the total number of alarms for given contexts and alarm states. Default is the total number of raised alarms, for all contexts.
|
|
##### Summary
Implements feature #6054
Now requests like
http://localhost:19999/api/v1/chart?chart=example.random
http://localhost:19999/api/v1/data?chart=example.random&options=jsonwrap&options=showcustomvars
- return chart variables in their responses. Chart variables include only those with options set to RRDVAR_OPTION_CUSTOM_CHART_VAR
- for /api/v1/data requests chart variables are returned when parameter options=jsonwrap and options=showcustomvars
##### Component Name
[/database](https://github.com/netdata/netdata/tree/master/database/)
[/web/api/formatters](https://github.com/netdata/netdata/tree/master/web/api/formatters)
|
|
* make remark access all directories
* detailed fix after autofix by remark lint
* cross check autofix for this set of files
* crosscheck more files
* crosschecking and small fixes
* crosscheck autofixed md files
|
|
* First pass of changing netdata to Netdata
* Second pass of netdata -> Netdata
* Starting work on netdata with no whitespace after
* Pass for netdata with no whitespace at the end
* Pass for netdata with no whitespace at the front
|
|
* URL_parser_review comments 1
* URL_parser_review restoring web_client.c
* URL_parser_review restoring url.h
* URL_parser_review restoring web_client.h
* URL_parser_review restoring inlined.h
* URL_parser_review restoring various
* URL_parser_review commenting!
* URL_parser_review last checks!
* URL_parser_review registry!
* URL_parser_review codacy errors!
* URL_parser_review codacy errors 2!
* URL_parser_review end of request!
* URL_parser_review
* URL_parser_review format fix
* URL_parser_review restoring
* URL_parser_review stopped at 5!
* URL_parser_review formatting!
* URL_parser_review:
Started the map of the query string when it is necessary
* URL_parser_review:
With these adjusts in the URL library we are now able to parser all the escape characters!
* URL_parser_review: code review
Fixes problems and format asked by coworkers!
* URL_parser_review: adjust script
The script was not 100% according the shellcheck specifications, no less important
it was a direct script instead a .in file
* sslstream: Rebase 2
It was necessary to change a function due the UTF-8
* sslstream: Fixing 6426
We had a cast error introduced by other PR, so I am fixing here
* URL_parser_review
Change .gitignore to avoid considering a script file.
|
|
#### Summary
Fixes #3117
Additionally it adds support for UTF-8 in URL parser (as it should).
Label sizes now are updated by browser with JavaScript (although guess is still calculated by verdana11_widths with minor improvements)
#### Component Name
API/Badges, LibNetData/URL
#### Additional Information
It was found that not only verdana11_widths need to be updated but the url parser replaces international characters with spaces (one space per each byte of multibyte character).
Therefore I update both to support international chars.
|
|
* Update docs health monitoring and health management api
* Update docs health monitoring and health management api
|
|
Fix the documention about Health, we were missing ```
|
|
This PR was created to fix #3414, here I am completing the job initiated by Christopher, among the newest features that we are bring we have
JSON inside the core - We are bringing to the core the capacity to work with JSON files, this is available either using the JSON-C library case it is present in the system or using JSMN library that was incorporated to our core. The preference is to have JSON-C, because it is a more complete library, but case the user does not have the library installed we are keeping the JSMN for we do not lose the feature.
Health LIST - We are bringing more one command to the Health API, now with the LIST it is possible to get in JSON format the alarms active with Netdata.
Health reorganized - Previously we had duplicated code in different files, this PR is fixing this (Thanks @cakrit !), the Health is now better organized.
Removing memory leak - The first implementation of the json.c was creating SILENCERS without to link it in anywhere. Now it has been linked properly.
Script updated - We are bringing some changes to the script that tests the Health.
This PR also fixes the race condition created by the previous new position of the SILENCERS creation, I had to move it to daemon/main.c, because after various tests, it was confirmed that the error could happen in different parts of the code, case it was not initialized before the threads starts.
Component Name
health directory
health-cmd
Additional Information
Fixes #6356 and #3414
|
|
* Alarm_repeat mergin the original!
* Alarm_repeat binary tree!
* Alarm_repeat binary tree finished!
* Alarm_repeat move function and format string
* Alarms bringing a new Binary tree
* Alarms fixing the last two
* Alarm_repeat useless var!
* Alarm fix format and repeat alarm!
* Alarm_backend steps!
* Alarm_repeat stopping to test cloud!
* Alarm_repeat stopping to test cloud 2!
* Alarm_repeat fixing when restart!
|
|
(#6274)"
This reverts commit 60a73e90de2aa1c2eaae2ebbc45dd1fb96034df2.
Emergency rollback of potential culprit as per issue #6356
Will be re-merging the change after investigation
|
|
* Alarms begin!
* Alarms web interface comments!
* Alarms web interface comments 2!
* Alarms bringing Christopher work!
* Alarms bringing Christopher work!
* Alarms commenting code that will be rewritten!
* Alarms json-c begin!
* Alarms json-c end!
* Alarms missed script!
* Alarms fix json-c parser and change script to test LIST!
* Alarms fix test script!
* Alarms documentation!
* Alarms script step 1!
* Alarms fix script!
* Alarms fix testing script and code!
* Alarms missing arguments to pkg_check_modules
* SSL_backend indentation!
* Alarms, description in Makefile
* Alarms missing extern!
* Alarms compilation!
* Alarms libnetdata/health!
* Alarms fill library!
* Alarms fill CMakeList!
* Alarm fix version!
* Alarm remove readme!
* Alarm fix readme version!
|
|
This reverts commit 58b7d95a7ec9c576f8a06bbab07f755846b5349a.
---
As agreed with @thiago and @cakrit we revert URL parser changes,
to buy the time on a more detailed investigation
---
|
|
This reverts commit 083378203c405665092f358e5f356f2afbb2ca15.
---
As agreed with @thiago and @cakrit we revert URL parser changes,
to buy the time on a more detailed investigation
---
|
|
This reverts commit 1c60c6eb8749c80acf5262253963d9d08f0d468e.
--
As agreed with @thiago and @cakrit we revert URL parser changes,
to buy the time on a more detailed investigation
--
|
|
|
|
* URL_parser fixing allmetrics!
* URL_parser fixing allmetrics 2!
|
|
* URL_parser 3
* URL_parser rebase 2!
* URL_parameter parsing 3
* URL_parameter parsing 4
* URL_parameter parsing 5
* URL_parser alarms
* URL_parser finish the basic structure
* URL_parser codacity fixes!
* URL_parser scripts!
* URL_parser codacy!
* URL_parser rebase 3!
* URL_parser host fixes!
* URL_parser host fixes 2!
* URL_parser fix spaces!
* URL_parser error message!
* URL_parser Christopher requests!
* URL_parser alarms fixed!
* URL_parser health fixed!
* URL_parser rebase 4!
* URL_parser C fix write format!
* URL_parser fix bugs due cache!
|
|
|
|
* Database engine prototype version 0
* Database engine initial integration with netdata POC
* Scalable database engine with file and memory management.
* Database engine integration with netdata
* Added MIN MAX definitions to fix alpine build of travis CI
* Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks
* DB engine disk protocol correction
* Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine
* Fix configure to require openSSL for DB engine
* Fix netdata daemon health not holding read lock when iterating chart dimensions
* Optimized query API for new DB engine and old netdata DB fallback code-path
* netdata database internal query API improvements and cleanup
* Bugfix for DB engine queries returning empty values
* Added netdata internal check for data queries for old and new DB
* Added statistics to DB engine and fixed memory corruption bug
* Added preliminary charts for DB engine statistics
* Changed DB engine ratio statistics to incremental
* Added netdata statistics charts for DB engine internal statistics
* Fix for netdata not compiling successfully when missing dbengine dependencies
* Added DB engine functional test to netdata unittest command parameter
* Implemented DB engine dataset generator based on example.random chart
* Fix build error in CI
* Support older versions of libuv1
* Fixes segmentation fault when using multiple DB engine instances concurrently
* Fix memory corruption bug
* Fixed createdataset advanced option not exiting
* Fix for DB engine not working on FreeBSD
* Support FreeBSD library paths of new dependencies
* Workaround for unsupported O_DIRECT in OS X
* Fix unittest crashing during cleanup
* Disable DB engine FS caching in Apple OS X since O_DIRECT is not available
* Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host
* Modified DB engine dataset generator to create multiple files
* Toned down overzealous page cache prefetcher
* Reduce internal memory fragmentation for page-cache data pages
* Added documentation describing the DB engine
* Documentation bugfixes
* Fixed unit tests compilation errors since last rebase
* Added note to back-up the DB engine files in documentation
* Added codacy fix.
* Support old gcc versions for atomic counters in DB engine
|
|
* Fix segmentation fault
* Make system info printing safe
* Fix quotes for OS name
|
|
* Add system info streaming
* Fix segmentation fault in unit testing
|
|
|
|
Return 503 instead of 400 when netdata hasnt started yet, move struct definitions in .c, swagger update (#5891)
|
|
* Add array of collector plugins-modules to api/v1/info
* Add system info to api/v1/info, collect data from separate script, use environment vars in anonymous statistics script
|
|
|
|
* Support older prometheus metric unit naming and allow removal of units from metrics
* Update swagger
* Added bats tests, improved efficiency of checking units
|
|
* Correct version check in UI. Support stable and nightly release channel.
* Use github releases instead of latest versions, get nightlies from GCS
* Prevent cross-origin errors by using the google API
|
|
* Add IPC message queue charts
* Add obsolete flag for dimensions
* Delete obsolete dimensions from memory
* Remove files for obsolete dimensions, filter requests
* Make empty charts obsolete
* Minimize obsolete dimension checks
* Limit the number of dimensions in memory
* Remove obsolete dimensions on netdata exit
* Update documentation
* Move flag to the end
* Fix typo
* Fix typo
|
