CMocka tests for Issue 7274 (#7308)

* Start of testing partial requests.

Need to stash this to checkout a PR to test.

* Disambiguated error messages during header validation.

The mocking has blown up in the linker, need to wipe out repo local changes and restart from a known good state.

* Test failures.

CMocka is really not designed for parametric tests which is making it difficult to test the http validation properly.
We have some problems in the web_client.c code that are causing early failures in the testing sequence, and it is
causing CMocka to abort the sequence. Need to try a different approach to building the tests...

* Pedantic style pass.

* Test generation.

There must be another value hidden in the system that CMocka uses. This sets up 3278 tests but the results from
cmocka_run_group_tests_name show 0 tests were run.

* The problem was the "helper"-macro.

Calling CMocka directly, moved the setup/teardown into explicit fixtures. Successfully runs the family of tests over
the same (empty) state.

* Parameterised family of tests runs.

The api_next() acts as a counter, the least significant digit is the prefix_len using the web_buffer in the test_family
struct as a template to walk throufh. The most significant digit is the number of headers to use in the request.

Checked that this walk executes correctly and all the tests run before putting the test payloads back in. We trigger a
failure about 3-4 tests in that takes down the process. Currently investigating which parts are not mocked correctly.

* Pedantic style pass

* Adding a mocking for fatal.

That weird thing with the linker has happened again, need to clean repo and rebuild fresh.

* Full test sequence executes.

The test parameter counter jammed after a failure - we cannot rely on anything in the main test body being executed
after we call the functionailty under test. A failure will skip the rest of the execution.

Moved the counter stepping to the top of the function (i.e. it is now a ++i instead of a i++). Adjusted the initial
state to compensate.

This now steps through all of the test-sequence, but it raises an ugly issue - the post-test cleanup will not be
executed on a failure.

TODO:
* Move the test-state into the test_family.
* Do the clean-up of the previous test (if necesarry) in the step function.
* Fix the assertion on the web_client state.

* Pedantic style pass

* Test state is now in the test_family.

This addresses the issue with leaking on failure and not performing clean-up - we don't really care about memory leaks
during unit-testing, but we do care about reseting the system-under-test back to a known state to guarantee
independence across the tests. The clean-up is now triggered in api_next().

* Flip the wait flag assertions.

Partial requests should leave the web_client waiting to receive more data.

* Fixing ACL flags in test-driver.

This makes some tests pass - but far too many. Probably need a proper debugging function to show the request / response
in a readable format.

* Result from the api mocking.

Setting a successful return code in the api mocking makes the non-partial tests pass.
Zero'ing out the web_client before use has not fixed the initialization errors, there is still some history on the
parse_tries that needs to be tracked down.
Some of the other errors are spurious - they result from stream multiplexing in the testdriver - be careful with less.

* Fix warnings.

Switched the build configuration to CFLAGS="-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all
-DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1".

The memset introduced last night to zero out the initial web_client state had transposed parameters. Now that the
state is initially zero before hitting the http request processing most issues have disappeared. There are 3000+
passing tests and 48 boundary cases to track down.

* Pushing log entries from each test into a buffer.

This will allow suppression of logs from tests that pass.

* Switched to a unique test definition structure per test.

This cleans up the code as it means that a list of tests can be constructed during the first walk through the parameter
space. There is no need to walk the space twice and keep both walks aligned. Removed the cmocka_unit_test macro and
build the CMUnitTest structures directly -> this allows a real name per test instead of the procedure name.

The walking/step function api_info_next has been folded back into the test procedure as it is simpler to  walk the list
in the shared test state.

Current TODO:
* There is a bug, the check on the wait state in the buffer is not being handled properly, investigate why
  everything fails.
* The results don't match the old code, are we handing the correct web_buffer to each tested piece of code?
* Capture the test success state -> dump the log buffer on failures.

* State is properly passed through the tests.

Spent a long time chasing a horrible bug that seems to be inside CMocka? The state parameter being passed to each
unit test is different on each call, i.e. it looks like a unique void** where the void * (*state) has been overwritten
with the original value on each iteration through the testing loop. This behaviour does not match the CMocka source
code, which does thread the given valud through the unit test calls. It could be a side-effect of the memory
check-pointing, but the net-effect is that we cannot change the shared state between tests. It can be set in the
setup-fixture and used in each test, but not altered for the subsequent test.

This took a long time to diagnose - the fix is simple, we just share the state in a global pointer. This shared state
is used to walk through the list of test_def structures so that each unit-test knows where it is in the parameter-
space.

* With the correct state the bug in triggering the correct assertions is gone.
* Dump out the buffered logs on test failure.
* The only failing case (relative to these assertions) are the ulta-short partial-requests.

* Check the web_client->mode is set properly.

* Style pass

* Checking values passed to the API despatch point.

* Disabled the parametric tests to do some low-level testings.

Later on both sets of tests will be active. While the low-level url encoding tests are being developed the dynamically
generated set is disabled to make the output easier to read.

Working through the W3C URL spec, against RFC3986 and comparing the cases in available url-parsing test-suites to build
our test-suite.

* Start of the URL test-suite.

The percent-decoding in the current implementation is in the wrong place - it happens too early and causes
non-delimitor characters in the URL to be treated as delimitors. Current unit-tests seem to cover the range of
checks that we need CMocka to make. The handling of output is a little awkward - need something like the dynamic
cases that can output the log on a failure or skip it on a pass.

* Raw material for low-level testing.

* Adding more families in here is getting too messy.

About to switch over to multiple testdrivers.

* Need to clean repo to work around wrapping failures in CMocka.

* CMocka is not compatible with LTO.

The weird wrapping issues that come and go are as a result of LTO. My typical netdata-installer command-line that I use
to reboot the project state disables LTO, while my normal autoreconf / configure command-line does not causing this
problem to reappear seemingly-randomly. To build a single test-driver target this works:

autoreconf -ivf && CFLAGS='-O1 -ggdb -Wall -Wextra -Wformat-signedness -fstack-protector-all -DNETDATA_INTERNAL_CHECKS=1 -D_FORTIFY_SOURCE=2 -DNETDATA_VERIFY_LOCKS=1' ./configure --disable-lto && make web/api/tests/web_api_valid_urls

The actual change in this commit is just a bug-fix.

* Ripping out the parameterized test generator.

Each of the URL cases is slightly and subtly different. This can't be done using the parameterization and will need a
healthy dose of cut and paste.

CMocka does not recognise the mocking for mysendfile, which is necessary to capture the exit route from the URL
parsing.

* Weird bug in CMocka?

For some reason CMocka will not mock out the mysendfile() procedure. We need to mock this to capture the behaviour of
the URL parsing as it is one of the exit paths. The wrapping is setup the same way as for the procedures so I cannot
see any reason that the library would not overwrite the calls. The only difference that I can find is that mysendfile
is in the unit being tested and the other mocked procedures are in different translation units.

This should not make a difference, but we have to disable LTO to get CMocka to work and the symbol patchs is some kind
of linker hack so there could be an issue if LTO is not running and the patch target is inside the same translation
unit.

Hiding it for now with a #ifndef UNIT_TESTING, which then compiles find and control flow hits the mock...

* Converting the ascii comments into unit_tests.

* More nasty cases for unit testing.

The commented out case will trigger a buffer overflow in the netdata agent and crash it.

* Last of the individual unit tests planned before the demo.

* Removing warnings.

* Switching on the rest of the parametric set - the other case with CRs.

* Fix Travis build failure under docker.

* Change the name of a define so it does not collide with existing testing in Travis.

* Add CMocka unit tests to CMake

* Linting pass

* Adding RFC comment to test.

* Buffer overflow checks on the captured logs.

This fixes the seg-fault seen by @vlvkobal and @thiagoftsm during testing.

* Chasing down other valgrind reports.

This gets rid of all of the uninitialised variable warnings. We stil have a memory leak, the headers that are set
during the unit testing switch on compression. This causes the web_client code to call deflatInit2 and allocate
structures for the compressor. We do not have a matching call to deflateEnd anywhere in the code so the memory leaks.

* Cleaning up a comment.

* Fixing review comments from @vlvkobal.

Also noticed that the buffer overflow fix this morning was killing the logfile output, fixed this as well.

* Addressing @thiagoftsm's concerns about the changing number of failures.

Switched the log dump for failing cases to repr().
Found a bug in the test case generator (not storing the flag for `\r`.
Verified that the 58 failing cases are the correct set of failures for the tested code.

1 files changed, 59 insertions, 1 deletions

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5158cdeba2..5e35dff6f3 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -897,6 +897,64 @@ if(BUILD_TESTING)
     target_link_libraries(storage_number_testdriver libnetdata ${NETDATA_COMMON_LIBRARIES} ${CMOCKA_LIBRARIES})
     add_test(NAME test_storage_number COMMAND storage_number_testdriver)
 
-    set_target_properties(str2ld_testdriver storage_number_testdriver PROPERTIES RUNTIME_OUTPUT_DIRECTORY tests)
+    set(WEB_API_TEST_FILES
+        web/api/tests/web_api.c
+        web/server/web_client.c
+        )
+    add_executable(web_api_testdriver ${WEB_API_TEST_FILES})
+    target_link_options(
+        web_api_testdriver
+        PRIVATE
+        -Wl,--wrap=rrdhost_find_by_hostname
+        -Wl,--wrap=finished_web_request_statistics
+        -Wl,--wrap=config_get
+        -Wl,--wrap=web_client_api_request_v1
+        -Wl,--wrap=rrdhost_find_by_guid
+        -Wl,--wrap=rrdset_find_byname
+        -Wl,--wrap=rrdset_find
+        -Wl,--wrap=rrdpush_receiver_thread_spawn
+        -Wl,--wrap=debug_int
+        -Wl,--wrap=error_int
+        -Wl,--wrap=info_int
+        -Wl,--wrap=fatal_int
+    )
+    target_link_libraries(web_api_testdriver libnetdata ${NETDATA_COMMON_LIBRARIES} ${CMOCKA_LIBRARIES})
+    add_test(NAME test_web_api COMMAND web_api_testdriver)
+
+    set(VALID_URLS_TEST_FILES
+        web/api/tests/valid_urls.c
+        web/server/web_client.c
+        )
+    add_executable(valid_urls_testdriver ${VALID_URLS_TEST_FILES})
+    target_link_options(
+        valid_urls_testdriver
+        PRIVATE
+        -Wl,--wrap=rrdhost_find_by_hostname
+        -Wl,--wrap=finished_web_request_statistics
+        -Wl,--wrap=config_get
+        -Wl,--wrap=web_client_api_request_v1
+        -Wl,--wrap=rrdhost_find_by_guid
+        -Wl,--wrap=rrdset_find_byname
+        -Wl,--wrap=rrdset_find
+        -Wl,--wrap=rrdpush_receiver_thread_spawn
+        -Wl,--wrap=debug_int
+        -Wl,--wrap=error_int
+        -Wl,--wrap=info_int
+        -Wl,--wrap=fatal_int
+        -Wl,--wrap=mysendfile
+        -DREMOVE_MYSENDFILE
+    )
+    target_link_libraries(valid_urls_testdriver libnetdata ${NETDATA_COMMON_LIBRARIES} ${CMOCKA_LIBRARIES})
+    add_test(NAME test_valid_urls COMMAND valid_urls_testdriver)
+
+    set_target_properties(
+        str2ld_testdriver
+        storage_number_testdriver
+        web_api_testdriver
+        valid_urls_testdriver
+        PROPERTIES RUNTIME_OUTPUT_DIRECTORY tests
+    )
+
+
 endif()
 endif()