summaryrefslogtreecommitdiffstats
path: root/Makefile.am
AgeCommit message (Collapse)Author
2019-12-12Implement the main flow for the Exporting Engine (#7149)Vladimir Kobal
* Add top level tests * Add a skeleton for preparing buffers * Initialize graphite instance * Prepare buffers for all instances * Add Grafite collected value formatter * Add support for exporting.conf read and parsing * - Use new exporting_config instead of netdata_config * Implement Grafite worker * Disable exporting engine compilation if libuv is not available * Add mutex locks - Configure connectors as connector_<type> in sections of exporting.conf - Change exporting_select_type to check for connector_ fields * - Override exporting_config structure if there no exporting.conf so that look ups don't fail and we maintain backwards compatibility * Separate fixtures in unit tests * Test exporting_discard_responce * Test response receiving * Test buffer sending * Test simple connector worker - Instance section has the format connector:instance_name e.g graphite:my_graphite_instance - Connectors with : in their name e.g graphite:plaintext are reserved So graphite:plaintext is not accepted because it would activate an instance with name "plaintext" It should be graphite:plaintext:instance_name * - Enable the add_connector_instance to cleanup the internal structure by passing NULL,not NULL arguments * Implement configurable update interval - Add additional check to verify instance uniqueness across connectors * Add host and chart filters * Add the value calculation over a database series * Add the calculated over stored data graphite connector * Add tests for graphite connector * Add JSON connector * Add tests for JSON formatting functions * Add OpenTSDB connector * Add tests for the OpenTSDB connector * Add temporaty notes to the documentation
2019-12-04Implement netdata command server and cli tool (#7325)Markos Fountoulakis
* Checkpoint commit (POC) * Implemented command server in the daemon * Add netdatacli implementation * Added prints in command server setup functions * Make libuv version 1 a hard dependency for the agent * Additional documentation * Improved accuracy of names and documentation * Fixed documentation * Fixed buffer overflow * Added support for exit status in cli. Added prefixes for exit code, stdout and stderr. Fixed parsers. * Fix compilation errors * Fix compile errors * Fix compile errors * Fix compile error * Fix linker error for muslc
2019-12-02proc.plugin: add pressure stall information (#7209)Haochen Tong
* proc.plugin: add pressure stall information * dashboard_info: add "Pressure" section * proc.plugin: mention PSI collector in doc * dashboard_info: fix grammar in PSI section * proc_pressure: fix wrong line name for "full" metrics * proc_pressure: fix copypasta * proc_pressure: refactor to prepare for cgroup changes * cgroups.plugin: add pressure monitoring * add proc_pressure.h to targets * Makefile.am: fix indentation * cgroups.plugin: remove a useless comment * cgroups.plugin: fix pressure config name * proc.plugin: arrange pressure charts under corresponding sections * dashboard_info: rearrange pressure chart descriptions * dashboard_info: reword PSI descriptions
2019-11-21CMocka tests for Issue 7274 (#7308)Andrew Moss
* Start of testing partial requests. Need to stash this to checkout a PR to test. * Disambiguated error messages during header validation. The mocking has blown up in the linker, need to wipe out repo local changes and restart from a known good state. * Test failures. CMocka is really not designed for parametric tests which is making it difficult to test the http validation properly. We have some problems in the web_client.c code that are causing early failures in the testing sequence, and it is causing CMocka to abort the sequence. Need to try a different approach to building the tests... * Pedantic style pass. * Test generation. There must be another value hidden in the system that CMocka uses. This sets up 3278 tests but the results from cmocka_run_group_tests_name show 0 tests were run. * The problem was the "helper"-macro. Calling CMocka directly, moved the setup/teardown into explicit fixtures. Successfully runs the family of tests over the same (empty) state. * Parameterised family of tests runs. The api_next() acts as a counter, the least significant digit is the prefix_len using the web_buffer in the test_family struct as a template to walk throufh. The most significant digit is the number of headers to use in the request. Checked that this walk executes correctly and all the tests run before putting the test payloads back in. We trigger a failure about 3-4 tests in that takes down the process. Currently investigating which parts are not mocked correctly. * Pedantic style pass * Adding a mocking for fatal. That weird thing with the linker has happened again, need to clean repo and rebuild fresh. * Full test sequence executes. The test parameter counter jammed after a failure - we cannot rely on anything in the main test body being executed after we call the functionailty under test. A failure will skip the rest of the execution. Moved the counter stepping to the top of the function (i.e. it is now a ++i instead of a i++). Adjusted the initial state to compensate. This now steps through all of the test-sequence, but it raises an ugly issue - the post-test cleanup will not be executed on a failure. TODO: * Move the test-state into the test_family. * Do the clean-up of the previous test (if necesarry) in the step function. * Fix the assertion on the web_client state. * Pedantic style pass * Test state is now in the test_family. This addresses the issue with leaking on failure and not performing clean-up - we don't really care about memory leaks during unit-testing, but we do care about reseting the system-under-test back to a known state to guarantee independence across the tests. The clean-up is now triggered in api_next(). * Flip the wait flag assertions. Partial requests should leave the web_client waiting to receive more data. * Fixing ACL flags in test-driver. This makes some tests pass - but far too many. Probably need a proper debugging function to show the request / response in a readable format. * Result from the api mocking. Setting a successful return code in the api mocking makes the non-partial tests pass. Zero'ing out the web_client before use has not fixed the initialization errors, there is still some history on the parse_tries that needs to be tracked down. Some of the other errors are spurious - they result from stream multiplexing in the testdriver - be careful with less. * Fix warnings. Switched the build configuration to CFLAGS="-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1". The memset introduced last night to zero out the initial web_client state had transposed parameters. Now that the state is initially zero before hitting the http request processing most issues have disappeared. There are 3000+ passing tests and 48 boundary cases to track down. * Pushing log entries from each test into a buffer. This will allow suppression of logs from tests that pass. * Switched to a unique test definition structure per test. This cleans up the code as it means that a list of tests can be constructed during the first walk through the parameter space. There is no need to walk the space twice and keep both walks aligned. Removed the cmocka_unit_test macro and build the CMUnitTest structures directly -> this allows a real name per test instead of the procedure name. The walking/step function api_info_next has been folded back into the test procedure as it is simpler to walk the list in the shared test state. Current TODO: * There is a bug, the check on the wait state in the buffer is not being handled properly, investigate why everything fails. * The results don't match the old code, are we handing the correct web_buffer to each tested piece of code? * Capture the test success state -> dump the log buffer on failures. * State is properly passed through the tests. Spent a long time chasing a horrible bug that seems to be inside CMocka? The state parameter being passed to each unit test is different on each call, i.e. it looks like a unique void** where the void * (*state) has been overwritten with the original value on each iteration through the testing loop. This behaviour does not match the CMocka source code, which does thread the given valud through the unit test calls. It could be a side-effect of the memory check-pointing, but the net-effect is that we cannot change the shared state between tests. It can be set in the setup-fixture and used in each test, but not altered for the subsequent test. This took a long time to diagnose - the fix is simple, we just share the state in a global pointer. This shared state is used to walk through the list of test_def structures so that each unit-test knows where it is in the parameter- space. * With the correct state the bug in triggering the correct assertions is gone. * Dump out the buffered logs on test failure. * The only failing case (relative to these assertions) are the ulta-short partial-requests. * Check the web_client->mode is set properly. * Style pass * Checking values passed to the API despatch point. * Disabled the parametric tests to do some low-level testings. Later on both sets of tests will be active. While the low-level url encoding tests are being developed the dynamically generated set is disabled to make the output easier to read. Working through the W3C URL spec, against RFC3986 and comparing the cases in available url-parsing test-suites to build our test-suite. * Start of the URL test-suite. The percent-decoding in the current implementation is in the wrong place - it happens too early and causes non-delimitor characters in the URL to be treated as delimitors. Current unit-tests seem to cover the range of checks that we need CMocka to make. The handling of output is a little awkward - need something like the dynamic cases that can output the log on a failure or skip it on a pass. * Raw material for low-level testing. * Adding more families in here is getting too messy. About to switch over to multiple testdrivers. * Need to clean repo to work around wrapping failures in CMocka. * CMocka is not compatible with LTO. The weird wrapping issues that come and go are as a result of LTO. My typical netdata-installer command-line that I use to reboot the project state disables LTO, while my normal autoreconf / configure command-line does not causing this problem to reappear seemingly-randomly. To build a single test-driver target this works: autoreconf -ivf && CFLAGS='-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1' ./configure --disable-lto && make web/api/tests/web_api_valid_urls The actual change in this commit is just a bug-fix. * Ripping out the parameterized test generator. Each of the URL cases is slightly and subtly different. This can't be done using the parameterization and will need a healthy dose of cut and paste. CMocka does not recognise the mocking for mysendfile, which is necessary to capture the exit route from the URL parsing. * Weird bug in CMocka? For some reason CMocka will not mock out the mysendfile() procedure. We need to mock this to capture the behaviour of the URL parsing as it is one of the exit paths. The wrapping is setup the same way as for the procedures so I cannot see any reason that the library would not overwrite the calls. The only difference that I can find is that mysendfile is in the unit being tested and the other mocked procedures are in different translation units. This should not make a difference, but we have to disable LTO to get CMocka to work and the symbol patchs is some kind of linker hack so there could be an issue if LTO is not running and the patch target is inside the same translation unit. Hiding it for now with a #ifndef UNIT_TESTING, which then compiles find and control flow hits the mock... * Converting the ascii comments into unit_tests. * More nasty cases for unit testing. The commented out case will trigger a buffer overflow in the netdata agent and crash it. * Last of the individual unit tests planned before the demo. * Removing warnings. * Switching on the rest of the parametric set - the other case with CRs. * Fix Travis build failure under docker. * Change the name of a define so it does not collide with existing testing in Travis. * Add CMocka unit tests to CMake * Linting pass * Adding RFC comment to test. * Buffer overflow checks on the captured logs. This fixes the seg-fault seen by @vlvkobal and @thiagoftsm during testing. * Chasing down other valgrind reports. This gets rid of all of the uninitialised variable warnings. We stil have a memory leak, the headers that are set during the unit testing switch on compression. This causes the web_client code to call deflatInit2 and allocate structures for the compressor. We do not have a matching call to deflateEnd anywhere in the code so the memory leaks. * Cleaning up a comment. * Fixing review comments from @vlvkobal. Also noticed that the buffer overflow fix this morning was killing the logfile output, fixed this as well. * Addressing @thiagoftsm's concerns about the changing number of failures. Switched the log dump for failing cases to repr(). Found a bug in the test case generator (not storing the flag for `\r`. Verified that the 58 failing cases are the correct set of failures for the tested code.
2019-11-18installer: add missing trailing backslash (#7326)Mansour Behabadi
Fixes #7324
2019-11-11Makefile.am files indentation (#7252)Konstantinos Natsakis
* Use 4 spaces for indentation of non-recipe lines in Makefile.am files * Be more consistent in the use of space before = in Makefile.am files
2019-11-07Initial CMocka testing against web_client.c (issue #7229). (#7264)Andrew Moss
* Start of unit-test for http request processing. A CMocka test-driver has been added to replay the live-capture headers against the http request processing code. Enough stubs / mocking code have been written to check that we can inject a chosen raw request into the web_client code and check how it was processed. A parameterised buffer building function can setup requests for testing in `web_buffer` structures. Each endpoint in the API will be checked against a variety of http requests. This PoC demonstrates enough functionality to show building the request on demand for each test-case and cleaning up afterwards.
2019-10-21[collector/proc.plugin] Add /proc/pagetypeinfo parser (#6843)Adrien Mahieux
* [proc.plugin/proc_pagetypeinfo] Initial commit * [Fix] Generate graphs for pagetypeinfo * [Fix] Create node/zone/type graphs * [Fix] Use directly size and order * [Add] Configuration handling * [Imp] Changed SetId to identify NodeNumber * [Fix] Standard name for chart priority and value * [Fix] use dynamic pagesize * [Enh] allow prefix for containerized netdata * [Fix] global system graph always on, but for explicit no * [Fix] Add more checks for pageorders_cnt and really use it * [Enh] Special config value of netdata_zero_metrics_enabled * [Fix] Check we parsed at least a valid line
2019-10-18Fix build when CMocka isn't installed (#7129)Vladimir Kobal
2019-10-15Add CMocka unit tests (#6985)Vladimir Kobal
* Add str2ld test * Build test with Autotools * Add storage_number test * Configure tests in CMake
2019-09-18Collector slabinfo (#6800)Adrien Mahieux
### Summary Provide new collector parsing `/proc/slabinfo` to provide details on kernel slab structures. Asked by issue #13 (very happy for the oldest issue in backlog) ##### Component Name collectors/slabinfo.plugin ##### Additional Information This slabinfo details allows to have clues on actions done on your system. In the following screenshot, you can clearly see a `find` done on a ext4 filesystem (the number of `ext4_inode_cache` & `dentry` are rising fast), and a few seconds later, an admin issued a `echo 3 > /proc/sys/vm/drop_cached` as their count dropped.
2019-09-12netdata/ci: second batch of fixes for coverity scan script and others (#6804)Paul Emm. Katsoulakis
* netdata/ci: fine grain coverity scan toolkit 1) Deprecate coverity-install.sh 2) add set -e to raise errors more aggressively when something is wrong 3) refactor some variable definitions for temp and install paths, use mktemp and clean up temp dir when all ok * netdata/ci: reduce the scope * netdata/ci: require curl * netdata/ci: go soft on functions loading * netdata/ci: allow override of build version without touching the script * netdata/ci: handle shellcheck errors, some were silenced because we dont handle them * netdata/ci: coverity scan, parse any extra options other than --with-install and pass them to configure 1) add default configure arguments if no extras are given 2) parse all possible arguments individually, handle --with-install and pass the rest to scanit * netdata/ci: focus, argument parsing in one place a bit more consistently * netdata/ci: prepare for using * netdata/ci: coverity - add more flags, adjust deps list 1) Add with math and with zlib flags on the script 2) add xenstat dependencies 3) run in coverity with custom flags, so that we add xenstat (no need to be by default for the users of the script) * netdata/packaging: add xen-utils also * test the flow * netdata/ci: dont forget to go back to the original directory * netdata/ci: please, travis, stop it * netdata/ci: how about this then * netdata/ci: silence that, we cant call it otherwise * netdata/packaging: update documentation around xenstat on centos * netdata/packaging: dont enable xenstat at this stage * netdata/ci: add += instead of variable reusage
2019-08-14Add MongoDB backend (#6524)Vladimir Kobal
* Add mongodb backend skeleton * Send data to the backend * Send metrics as separate JSON documents * Add a configuration file * Send all metrics in a batch * Update the documentation * Free configuration strings on exit * Make socket timeout configurable
2019-08-12(re-open) ZRAM info collector module (proc.plugin) (#6424)Vilkov Adel
* ZRAM collector module ZRAM: Implemented zram device id detection ZRAM: Implemented zram device enumeration WIP ZRAM: Memory usage graph (needs other graphs) ZRAM: Added ratio and efficiency graph ZRAM: Added chart description and context names, code formatting * ZRAM: Proper handling of zram device removal * ZRAM: Added additional checks, removed redundant logging
2019-08-12netdata/packaging: put go.d version in one place (#6557)Paul Emm. Katsoulakis
* netdata/packaging: [ci skip] Add go.d version to a single place * netdata/packaging: [ci skip] Things are a bit different in debian build, we need to define the path in a different way so pass it from the upper layer * netdata/packaging: [ci skip] Missed to add on the makefile * netdata/packaging: bump go.d version
2019-07-24Utf8 Badge Fix And URL Parser International Support (initial) (#6426)Timo
#### Summary Fixes #3117 Additionally it adds support for UTF-8 in URL parser (as it should). Label sizes now are updated by browser with JavaScript (although guess is still calculated by verdana11_widths with minor improvements) #### Component Name API/Badges, LibNetData/URL #### Additional Information It was found that not only verdana11_widths need to be updated but the url parser replaces international characters with spaces (one space per each byte of multibyte character). Therefore I update both to support international chars.
2019-07-09Revert "Add ZRAM collector module to the proc plugin"Pavlos Emm. Katsoulakis
This reverts commit c7ab028f787f1c3f1325f6195ea0cb2afc95ab95. **Removed as it was seen to cause crashes. Change will be revised and re-published at a later stage**
2019-07-09Add ZRAM collector module to the proc pluginVilkov Adel
The module gets the ZRAM device list by reading /proc/devices, obtaining an device ID from it, then enumerating the devices in /dev filtering them by corresponding major device number it got from previous step. It takes the data from /sys/block/{name}/mm_stat.
2019-07-01Easily disable alarms, by persisting the silencers configuration (#6360)thiagoftsm
This PR was created to fix #3414, here I am completing the job initiated by Christopher, among the newest features that we are bring we have JSON inside the core - We are bringing to the core the capacity to work with JSON files, this is available either using the JSON-C library case it is present in the system or using JSMN library that was incorporated to our core. The preference is to have JSON-C, because it is a more complete library, but case the user does not have the library installed we are keeping the JSMN for we do not lose the feature. Health LIST - We are bringing more one command to the Health API, now with the LIST it is possible to get in JSON format the alarms active with Netdata. Health reorganized - Previously we had duplicated code in different files, this PR is fixing this (Thanks @cakrit !), the Health is now better organized. Removing memory leak - The first implementation of the json.c was creating SILENCERS without to link it in anywhere. Now it has been linked properly. Script updated - We are bringing some changes to the script that tests the Health. This PR also fixes the race condition created by the previous new position of the SILENCERS creation, I had to move it to daemon/main.c, because after various tests, it was confirmed that the error could happen in different parts of the code, case it was not initialized before the threads starts. Component Name health directory health-cmd Additional Information Fixes #6356 and #3414
2019-06-28Revert "Easily disable alarms, by persisting the silencers configuration ↵Pavlos Emm. Katsoulakis
(#6274)" This reverts commit 60a73e90de2aa1c2eaae2ebbc45dd1fb96034df2. Emergency rollback of potential culprit as per issue #6356 Will be re-merging the change after investigation
2019-06-27Easily disable alarms, by persisting the silencers configuration (#6274)thiagoftsm
* Alarms begin! * Alarms web interface comments! * Alarms web interface comments 2! * Alarms bringing Christopher work! * Alarms bringing Christopher work! * Alarms commenting code that will be rewritten! * Alarms json-c begin! * Alarms json-c end! * Alarms missed script! * Alarms fix json-c parser and change script to test LIST! * Alarms fix test script! * Alarms documentation! * Alarms script step 1! * Alarms fix script! * Alarms fix testing script and code! * Alarms missing arguments to pkg_check_modules * SSL_backend indentation! * Alarms, description in Makefile * Alarms missing extern! * Alarms compilation! * Alarms libnetdata/health! * Alarms fill library! * Alarms fill CMakeList! * Alarm fix version! * Alarm remove readme! * Alarm fix readme version!
2019-06-20Perf plugin (#6225)Vladimir Kobal
* Add perf plugin skeleton * Initialize events * Collect data * Configure default counters * Add charts for hardware and software counters * Add charts for cache counters * Don't show zeroes for non-existent metrics * Reinit events when stalled * Do not reinit disabled events * Update the documentation * Scale values when multiplexing is happening
2019-06-09netdata/packaging: Introducing automatic binary packages generation and ↵Paul Emm. Katsoulakis
delivery for RPM types (Phase 1) (#6223) * netdata/packaging: Introducing automatic binary packages generation and delivery for RPM types (Phase 1) With this commit we introduce our packaging toolkits and workflows to automate the delivery for RPM packages to packagecloud.io, our packaging content delivery infrastructure. At this stage we have only prepared the required flows to publish our latest stable release. Missing items for now: 1) Packaging versioning - we won't be providing increasing version numbers for the same upstream netdata version 2) Nightlies - our nightlies won't be provided by this channel, yet With this changeset, we also introduce the basic artifacts for DEB packaging generation, with the same misses mentioned above. Debian is likely not to be functional within this sprint run though. Here's a more detailed list of the changes, as they were generated from the original branch: 89baa00b netdata/packging: Prior to the PR, place the master bracnh as the place to execute packaging acfd2804 export variables at before_script b95b112b remove that README.md, we will cover details from the distributions document in packaging folder e52ab82e Update README.md 8df1f8e9 [ci skip] There is no need for package all option. There is absolutely no uniformity on available images or distro per arch to do our job for all at the same time, so remove it for now 6fc1600c remove a few of the unavailable architectures from distros 7e2741de missed arch, we actually have i686 64c72b4e Add arm64 architecture as a build option (to be refined on follow up commit for the distros not having arm64 or amd64 not to trigger build) ead672b4 silence shellchecking, also fix a shameful recommendation from shellcheck so that array parsing can work 986d0e00 you disappoint me shellcheck, shame on you a5c11a75 remove debug ls command 4b5fade6 nit - too obvious to spot 7edb9215 nits and fixes 06cf10b9 fix the obvious miss - you need to the full path appended before the container name, otherwise no folder.. d777f8ae Give me a break travis, whats wrong with you -- define packages directory at the yaml config level f9756642 nits ee253493 missed some old parts 1506105c we dont need container root variable here, we just need to go through the base LXC directory 03dc231c restructure packaging preparation process 529cdedb simplify, unify logic around build arch handling between creation and building 17624c14 pull packages directory variable in the code. We calculate base on arch c47ea326 remove redudant package type var, also hard exit when not found the right stuff 8890041d Pull more arch dependent variables out of yaml config and into the code 832fbf71 remove unused variable b54d6ccb Move check lower 0b1a3a7e handle build arch requirement, plus some shellcheck fixes 07bb01fe missed this, should pass container name instead 1ab8a3a4 we dont use build arch within this scope, ditch the check 6d3df1b3 Add i386 packaging keywords 1b403680 Remove arch from the descriptions and the environment variables We wont be building sections per arch in travis, to minimize config file size. for each distro we will always build both ARCHS unless the commit message instructs us to build specific architecture 8f38caa5 Drop support of 42.3 8e394906 its word counting here, focus d3f6009c fix yanking command 5b4d7b9a dont call gpg flag on all distros, focus 7e0f4d64 better listing when yanking packages bdb2d3aa skipping gpg checks, for now 75841a67 use newer distro instead a439a9f9 downgrade base dist eba0fba4 Merge branch 'master' into package-cloud-deployer 241c0e10 nits and fixeS aad8d8bf fix naming, add update process in debian preparation 872b15af Separately list rpm and srpm contents for yanking b3ef3a6f el/8 and fedora/31 not ready yet d468cad7 Null commit 22135a2f fix stages for packaging 1464cd6b Null commit 5d44d4f9 implement yanking - when same version is uploaded, remove to re-add 0a4e2e46 sync the repo first of all 9f71583c Generalize repo tool definition on yaml config, so that we adjust per distro as needed 3988eb74 adjustments on the distro list, also remove coreutils -- not important 51571968 check if its just permission issue 38fa8b2b adjust syntax e08ebd2b Due to the peculiarity of the way we do things, script out the packages preparation so that we can sudo them 0a3ce2c1 Now that we solved all other issues, make sure you add rpm building dependencies to LXC aa0b370e a couple of nits and fixes 1) we need sudo within the container 2) just use wget everywhere, dont mix up too many packages for the same thing 3) do not try to add packages again, our dependency script should take care that. Single point of responsibility a3f42e8d You need to request home directory to be created (need to review under the different distros for the right syntax though) d7a3d0d0 dont go and add manually the required packages, call our dependency scriptlet 5c0888b7 skip GPG checking for now 9aa64bbe dont create from the python lib 19f8cc6b Add more keywords for better grouping 0d5b89d7 nits e260a57d python libs not in good shape, try running create from command line instead 7fa58921 netdata/packaging/ci: Bingo - add container name on the RPM jobs b99ff89e some nits and fixes, lets see 636057b2 bring back sudo, move to /var/lib/lxc, pass -E to sudo to preserve vars 1e487517 i wonder.. 97583506 Update SuSE list 2e92649b fix doc - wrong distro details 6f53a4c2 Initial distro document -- WIP d2ef5e4f Add some README stuff, also introduce yanking step in pre-deploy (empty script at this point) 465f293f Change the token 36431ab5 Add a package cloud wrapper (required for yanking RPMs) 2142a024 Auto-detect username for the beta deployment, so that others can use it. Secret key will remain as is for now, will do revisit this also, so that others can beta test the implementation too. Also include provide for the production deployment, when running on master ab5a85ad Add more special conditions, also fix spacing and style a little bit more: ce893eae build only on special conditions (branch name for now) 82340536 Attempt to find how to ref template in stages 5375cd93 one more thing - use ubuntu instead of download template, lets see 4ff26df4 Attempt to fetch lxc-templates (not installed by default 52715ba2 Bring back original implementation - tests gave some light to the problem 4607e7b8 lets see, try different template 4a453425 add some more debug stuff - will remove them once i figure it out 5ee2e03d Try to use the binaries for the creation, see what changes 95a22ef7 another re-arrangement 82855f91 Attempt to install a different version of libraries, as per another issue fbda0d3a Attempt to debug weird container failure d828de8c debugging 07bad55d reduce the noise, when not possible to create/modify the required directories dont do anything 6a2bd46b rename bba22329 netdata/health: shellchecking - SC2236 103a7df9 Adjust naming ea70d482 Run on Xenial c2b3ad77 You should sudo create the folder a2f00c26 Do not stop the build if one of the RPM fails, we want all of them to attempt to build c367ab6c Cleanup, bring also RPM template in and lets see what happens ba7b441b more nits -- add the right replacements 2dfbb3f5 move templated parts outside 3211bc11 another approach, missed something 2a47dd17 A more educated attempt based on other resources 670c35b7 first time in anchors, most likely is wrong but got to see what travis will do 6c00628e Binary release flows and other side fixes 1d89e519 Just in case, dont clean up if the env is messed up 06b8de6f start building up deb structure and flow, do some more re-arrangements also 05164ba6 longshot - bring the distro vars on the top to attempt to build a matrix. Also a fix: create the package folder prior populating it c9334c24 Move RPM code to separate folder, make room for the deb implementations 8474c823 do not copy over everything, only the RPM directories 13d8ac0a netdata/packages: attempt to copy over the folder to a different location 3cb0edf6 remove temporarily to check deploy process 79d3b69c Permissions 05c3d4bd Add listing of container fs contents to see whats the result eeee2249 fix container path, missed to adjust this after changing container name format 90a70b9c Add dependencies 34c3e1fa Dont forget to create the rpmbuild structure required 9bfb6026 reverse the order, so that we first add all required packages 663af0e5 Attempt to add all as user rather than w00t 0fa68530 more fixes 1) install wget within the container, so that we can then fetch our source 2) factor out the command execution, for cleaner code 23cc4517 better logging da58043b skip formatting, do the traditional way for now 818508c7 more fixes 1) escape underscore too. Actually previously the complain was about underscore, i wasnt paying attention 2) Parse all command results and make sure you break the script if something goes wrong, otherwise travis wont know the failures 78b9be3a Add more debug messages and also fix the non-escaped character % in string 2037f2c5 fixes and next steps 1) Use sudo, attempt to make privileged containers this way in case there is something messed up with homedirs in travis (long shot) 2) Implement the build process for the next step (not tested, we 'll see how it works) 3) Change the container name, to something more specific to the build we are preparing to help us identify the container more accurately (and avoid possible conflicts) 9dac37bf revert attempt, obviously it didnt provide any value 2d975a7d Attempt to catch any stacktrace, if any is raised 93826afe Reinstate skip_cleanup variable, as its needed for deploy anyway 54fd126a That is weird, but seems like lxc is there on subsequent runs, make sure you clear before you start f876e6a8 adjustments 1) add version string statically for now, for testing 2) bring more vars in 3) create the container from python all the way 67df8903 Flesh out the workflow for building up the builder environment on one distro 0906b41e change distro, dont forget 17e78ee1 fix distro keyword 26ca17d7 Create an experimental stage and bring onboard an encrypted token 4b0d8ce3 Introduce an experimental package cloud deployment stage * netdata/packaging: Remove hardcoded tmp director as per Codacy feedback * netdata/packaging: Update distributions document Update DISTRIBUTIONS.md based on Codacy warnings (round 1) * netdata/packaging: lets try to attend a few errors * Removing DISTRIBUTIONS.md from this PR As this is part of a separate task, it will be added on a separate branch and linked to the respective task, to unblock RPM generation
2019-06-07Prometheus remote write backend (#6062)Vladimir Kobal
* Add Prometheus remote write backend prototype * Fix autotools issues * Send HTTP POST request * Add parameters to HTTP header * Discard HTTP responce 200 * Update CMake build configuration * Fix Codacy issue * Check for C++ binary * Fix compilation without remote write backend * Add options to the installer script * Fix configure script warning * Fix make dist * Downgrade to ByteSize for better compatibility * Integrate remote write more tightly into the existing backends code * Cleanup * Fix build error * Parse host tags * Fix Codacy issue * Fix counters for buffered data * Rename preprocessor symbol * Better error handling * Cleanup * Update the documentation
2019-05-31SSL implementation for Netdata (#5956)thiagoftsm
* SSL implementation for Netdata * Upload of fixes asked by @paulkatsoulakis and @cakrit * Fix local computer * Adding openssl to webserver * fixing.. * HTTPS almost there * Codacity * HTTPS day 3 * HTTPS without Bio step 1 * HTTPS without Bio step 2 * HTTPS without Bio step 3 * HTTPS without Bio step 4 * HTTPS without Bio step 5 * HTTPS without Bio step 6 * HTTPS without Bio step 7 * HTTPS without Bio step 8 * HTTPS without Bio step 9 * HTTPS without Bio step 10 * SSL on streaming 1 * Daily pull * HTTPS without Bio step 11 * HTTPS without Bio step 12 * HTTPS without Bio step 13 * HTTPS without Bio step 14 * SSL_Interception change documentation * HTTPS without Bio step 15 * HTTPS without Bio step 16 * SSL_Interception fix codacity * SSL_Interception fix doc * SSL_Interception comments * SSL_Interception fixing problems! * SSL_Interception killing bugs * SSL_Interception changing parameter * SSL_Implementation documentation and script * SSL_Implementation multiple fixes * SSL_Implementation installer and cipher * SSL_Implementation Redirect 301 * SSL_Implementation webserver doc and install-or-update.sh * SSL_Implementation error 00000001:lib(0):func(0):reason(1) * SSL_Implementation web server doc * SSL_Implementation SEGFAULT on Fedora * SSL_Implementation fix ^SSL=force|optional * SSL_Implementation Redirect and Ciphers * SSL_Implementation race condition 1 * SSL_Implementation Fix Location * SSL_Implementation Fix Location 2 * SSL_Implementation Fix stream * SSL_Implementation Fix stream 2 * SSL_Implementation Fix stream 3 * SSL_Implementation last problems! * SSL_Implementation adjusts to commit! * SSL_Implementation documentation permission! * SSL_Implementation documentation permission 2! * SSL_Implementation documentation permission 3!
2019-05-30DB engine optimize RAM usage (#6134)Markos Fountoulakis
* Optimize memory footprint of DB engine * Update documentation with the new memory requirements of dbengine * Fixed code style * Fix code style * Fix compile error
2019-05-22Fix build errors (#6081)Markos Fountoulakis
* Fix build errors * Remove obsolete flags
2019-05-19netdata/daemon/backends: Fix AWS Kinesis link error when building on Mac OS ↵Paul Emm. Katsoulakis
(#6047) Agreed with @cakrit to proceed without waiting another approval
2019-05-15Database engine (#5282)Markos Fountoulakis
* Database engine prototype version 0 * Database engine initial integration with netdata POC * Scalable database engine with file and memory management. * Database engine integration with netdata * Added MIN MAX definitions to fix alpine build of travis CI * Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks * DB engine disk protocol correction * Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine * Fix configure to require openSSL for DB engine * Fix netdata daemon health not holding read lock when iterating chart dimensions * Optimized query API for new DB engine and old netdata DB fallback code-path * netdata database internal query API improvements and cleanup * Bugfix for DB engine queries returning empty values * Added netdata internal check for data queries for old and new DB * Added statistics to DB engine and fixed memory corruption bug * Added preliminary charts for DB engine statistics * Changed DB engine ratio statistics to incremental * Added netdata statistics charts for DB engine internal statistics * Fix for netdata not compiling successfully when missing dbengine dependencies * Added DB engine functional test to netdata unittest command parameter * Implemented DB engine dataset generator based on example.random chart * Fix build error in CI * Support older versions of libuv1 * Fixes segmentation fault when using multiple DB engine instances concurrently * Fix memory corruption bug * Fixed createdataset advanced option not exiting * Fix for DB engine not working on FreeBSD * Support FreeBSD library paths of new dependencies * Workaround for unsupported O_DIRECT in OS X * Fix unittest crashing during cleanup * Disable DB engine FS caching in Apple OS X since O_DIRECT is not available * Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host * Modified DB engine dataset generator to create multiple files * Toned down overzealous page cache prefetcher * Reduce internal memory fragmentation for page-cache data pages * Added documentation describing the DB engine * Documentation bugfixes * Fixed unit tests compilation errors since last rebase * Added note to back-up the DB engine files in documentation * Added codacy fix. * Support old gcc versions for atomic counters in DB engine
2019-05-13Add AWS Kinesis backend (#5914)Vladimir Kobal
* Add Kinesis backend * Separate config file * Send data in chunks * Fix minor issues * Add error handling * Use existing JSON functions * Do not retry on send failure * Implement building with autotools * Implement building with CMake * Fix CMake variables * Fix build when C++ compiler is not available * Add checks for C++11 * Don't reinitialize API * Don't reinitialize client * Minor cleanup * Fix Codacy warning * Separate sending records and receiving results * Add documentation * Make connection timeout configurable * Fix operation metrics * Fix typo * Change parameter names for credentials * Allow using the default SDK credentials configuration
2019-04-02netdata/packaging: Remove .tar.gz on distclean (#5763)Paul Emm. Katsoulakis
* netdata/packaging: Remove .tar.gz on distclean * netdata/packaging: Do not use pattern, there is dist variable to point to the specific file to delete upon clean
2019-03-27Add xenstat plugin (#5660)Vladimir Kobal
* Add xenstat plugin * Add basic domain charts * Initialize xl context * Use domain UUID instead of name * Make charts obsolete * Add tmem charts * Change algorithm for tmem puts and gets * Add VCPU charts * Minor formatting for sending charts functions * Add VBD charts * Add network charts * Assemble VCPU metrics in one chart * Fix chart names * Make write/sent dimensions negative * Minor formatting * Change id and context for domain charts * Add dashboard info * Get rid of global variables * Free libxenstat and libxl resourses * Free domain_metrics on VM shutdown * Add domain state chart * Add debug messages * Add branch prediction hints * Minor fix * Fix chart obsoleting * Make names more general * Fix CMake build of nfacct.plugin
2019-03-21bring uninstaller back (#5675)Paweł Krupa
2019-03-19netdata/build: A small house keeping around coverity scan (no functional or ↵Paul Katsoulakis
process change) (#5656) * netdata/build: a small house keeping around coverity scan (no functional or process change) 1) Split coverity install on seperate scriptlet, call it through travis when needed 2) Remove shellcheck exclussions, they dont seem to be raised any more (i might be using wrong version of shellcheck, will see) 3) Fix authors,copyrights etc 4) Make sure all echo goes to stderr 5) When installing, use mktemp for the temporary directory and put everything there. Remove afterwards * netdata/build: missed - remove temp folder once done * Rebumping with empty commit to trigger build * Temp commit to force execution of modified code for coverity * [ci skip] Revert "Temp commit to force execution of modified code for coverity" This reverts commit 70e770cbbcb2b9cb4c8040d6249688c810d823f3. * Temp commit to force execution of modified code for coverity * Temp commit to force execution of modified code for coverity * packaging/netdata/coverity-scan: Missed PATH export during scan phase * netdata/packaging/coverity-scan: the source folder not so temporary after all Use a more permanent path for the workdir and we will deep dive on the details of this if necessary at a later stage * netdata/packaging: This is a more general change, push REPOSITORY variable up to travis layer to help testing against personal accounts * netdata/packaging/coverity-scan: Missed folder creation (no longer using mktemp, duh), also make sure you do a couple of validations so that you know install is failed or not, otherwise we exit with success but nothing has happened * netdata/coverity-scan: a few more changes towards abstraction 1) move for submit email to travis environment variables 2) Use REPOSITORY in project name 3) Add some new lines on the last log message, to separate it from the curl result * netdata/coverity: missed parameter from echo * Reinstate the flag that controls coverity scan execution (was removed for testing purposes, to run coverity in a controlled environment) * netdata/coverity-scan: Control coverity scan results submission and report back the potential error without blocking the pipeline by failing the step completely
2019-02-13Split nfacct plugin into separate process (#5361)Vladimir Kobal
* Prepare build configuration * Prepare plugin for separating * Add command line options * Add debug messages * Use text API * Minor fixes * Update the documentation * Minor documentation formatting * Fix LGTM alerts * Fix building with CMake * Add nfacct and cups plugins to apps.plugin groups
2019-02-04install go.d.plugin (#5199)Paweł Krupa
[minor] release
2019-02-04CUPS plugin (#5188)Simon Nagl
* Implement a CUPS collector plugin * Compile cups.plugin only if enabled * Fix build with CFLAGS -O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1 * cups.plugin check for all used functions * Use cups-config to configure cups compiler flags * Do not quit if cups-config is not installed * Fix compiler warning collectors/cups.plugin/cups_plugin.c:359:27: warning: format '%d' expects argument of type 'int', but argument 6 has type 'unsigned int' [-Wformat=] * Add cups.plugin to toc and overview documentation * cups.plugin fix plugin doc layout * cups.plugin: Add prerequisites doc * cups.plugin: Fix error if cups is not installed
2019-01-18fix path to updater file (#5209)Paweł Krupa
* fix path to updater file * pack updater into dist package
2019-01-15