summaryrefslogtreecommitdiffstats
path: root/test/testutil/driver.c
AgeCommit message (Collapse)Author
2023-09-28Copyright year updatesMatt Caswell
Reviewed-by: Richard Levitte <levitte@openssl.org> Release: yes
2023-09-20Enable the ability to seed the test RNG without randomising test orderingMatt Caswell
Numerous tests use the test_random() function to get a random number. If a test fails then the seed that was used for the test RNG is displayed. Setting the seed to the same value in a future run is supposed to cause the same random numbers to be generated again. The way to set the RNG seed again is to use the `OPENSSL_TEST_RAND_ORDER` environment variable. However setting this environment variable *also* randomises the test ordering as well as seeding the RNG. This in itself calls test_random() so, in fact, when the test finally runs it gets different random numbers to when it originally run (defeating the repeatability objective). This means that only way repeatability can be obtained is if the test was originally run with `OPENSSL_TEST_RAND_ORDER` set to 0. If that wasn't done then the seed printed when the test failed is not useful. We introduce a new environment variable `OPENSSL_TEST_RAND_SEED` which can be used to independently seed the test RNG without randomising the test ordering. This can be used to get repeatability in cases where test ordering randomisation was not done in the first place. Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from https://github.com/openssl/openssl/pull/22118)
2022-10-27tests: clear error queue before executing a testcaseTomas Mraz
There can be errors in the queue from previous tests and we look at it to verify we do not add spurious errors in some testcases. Fixes #19477 Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from https://github.com/openssl/openssl/pull/19483) (cherry picked from commit 4bae06d47ae26b37a948d31f11884e1813f6d669)
2022-10-21test: driver: fix -Wunused-but-set-variableSam James
The value of 'num_failed_inner' isn't ever used. Fixes this error with Clang 15: ``` test/testutil/driver.c:341:17: error: variable 'num_failed_inner' set but not used [-Werror,-Wunused-but-set-variable] int num_failed_inner = 0; ^ 1 error generated. ``` Signed-off-by: Sam James <sam@gentoo.org> Reviewed-by: Paul Dale <pauli@openssl.org> Reviewed-by: Hugo Landau <hlandau@openssl.org> Reviewed-by: Tomas Mraz <tomas@openssl.org> (Merged from https://github.com/openssl/openssl/pull/19450)
2021-06-29coverity #1486532: fix potential NULL dereference in test_mk_file_path()Tomas Mraz
Reviewed-by: Ben Kaduk <kaduk@mit.edu> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from https://github.com/openssl/openssl/pull/15942)
2021-06-26testutil: teach test_mk_file_path() how to merge VMS file specsRichard Levitte
This isn't a full solution, it only handles current use cases. Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from https://github.com/openssl/openssl/pull/15823)
2021-04-19Avoid more MSVC-specific C runtime library functionsTanzinul Islam
Reviewed-by: Richard Levitte <levitte@openssl.org> Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com> (Merged from https://github.com/openssl/openssl/pull/13540)
2021-04-18TEST: Modify testutil's run_tests to display NOSUBTEST cases individuallyRichard Levitte
When test cases were added with ADD_ALL_TESTS_NOSUBTEST(), all those iteration verdicts were summarized as if it was one single case. This modification gets each iteration verdict displayed separately instead. Reviewed-by: Tomas Mraz <tomas@openssl.org> (Merged from https://github.com/openssl/openssl/pull/14498)
2021-03-11Update copyright yearMatt Caswell
Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/14512)
2021-02-26tests: Always print errors before test verdictTomas Mraz
Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from https://github.com/openssl/openssl/pull/14316)
2021-02-23test: add framework for generic fake random number generatorPauli
Reviewed-by: Tim Hudson <tjh@openssl.org> (Merged from https://github.com/openssl/openssl/pull/13652)
2020-12-15test: print OPENSSL_TEST_RAND_ORDER=x when a randomised test fails.Pauli
The previous message "random seed x" is a lot less descriptive. Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org> (Merged from https://github.com/openssl/openssl/pull/13672)
2020-12-02Print random seed on test failure.Pauli
Tests randomisation wasn't reliably printing the seed used on failure in the CIs. Fixes: #13572 Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org> (Merged from https://github.com/openssl/openssl/pull/13573)
2020-06-25Update copyright yearMatt Caswell
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com> (Merged from https://github.com/openssl/openssl/pull/12273)
2020-06-06TESTUTIL: Adjust the rest of testutilRichard Levitte
Fixes #12054 Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/12057)
2019-12-14Deprecate most of debug-memoryRich Salz
Fixes #8322 The leak-checking (and backtrace option, on some platforms) provided by crypto-mdebug and crypto-mdebug-backtrace have been mostly neutered; only the "make malloc fail" capability remains. OpenSSL recommends using the compiler's leak-detection instead. The OPENSSL_DEBUG_MEMORY environment variable is no longer used. CRYPTO_mem_ctrl(), CRYPTO_set_mem_debug(), CRYPTO_mem_leaks(), CRYPTO_mem_leaks_fp() and CRYPTO_mem_leaks_cb() return a failure code. CRYPTO_mem_debug_{malloc,realloc,free}() have been removed. All of the above are now deprecated. Merge (now really small) mem_dbg.c into mem.c Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/10572)
2019-08-09Extend tests of SSL_check_chain()Matt Caswell
Actually supply a chain and then test: 1) A successful check of both the ee and chain certs 2) A failure to check the ee cert 3) A failure to check a chain cert Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org> (Merged from https://github.com/openssl/openssl/pull/9442)
2019-07-29Add weak platform independent PRNG to test framework.Pauli
Implement the GNU C library's random(3) pseudorandom number generator. The algorithm is described: https://www.mscs.dal.ca/~selinger/random/ The rationale is to make the tests repeatable across differing platforms with different underlying implementations of the random(3) library call. More specifically: when executing tests with random ordering. [extended tests] Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de> (Merged from https://github.com/openssl/openssl/pull/9463)
2019-07-16remove end of line spacesPauli
Reviewed-by: Paul Yang <yang.yang@baishancloud.com> (Merged from https://github.com/openssl/openssl/pull/9397)
2019-07-16Remove tab characters from C source files.Pauli
Some have been creeping into the source code. Reviewed-by: Paul Yang <yang.yang@baishancloud.com> (Merged from https://github.com/openssl/openssl/pull/9397)
2019-06-03Really fail if we have a test failureMatt Caswell
Commit c5f7a99645a broke the test framework such that some tests might fail, but the test framework still gives a PASS result overall. Reviewed-by: Paul Dale <paul.dale@oracle.com> (Merged from https://github.com/openssl/openssl/pull/9056)
2019-05-09Test skip option.Pauli
Provide C test cases with the option to skip tests and subtests. Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/8695)
2019-03-05testutil: ensure good treatment of argv on non-Unix platformsRichard Levitte
From a Unix point of view, some other platform families have certain quirks. Windows command prompt doesn't expand globs into actual file names, so we must do this. VMS has some oddity with argv pointer size that can cause crashes if you're not careful (by copying it to a less surprising pointer size array). The fixups already exist and are used in the apps/ code. However, the testutil code started using the opt routines from apps/ without including the non-Unix fixups. This change fixes that. For VMS' sake, libtestutil gets an app_malloc() shim, to avoid sucking in all of apps/apps.c. Reviewed-by: Paul Dale <paul.dale@oracle.com> (Merged from https://github.com/openssl/openssl/pull/8381)
2019-02-11Updated test command line parsing to support commmon commandsShane Lontis
Reviewed-by: Paul Dale <paul.dale@oracle.com> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/6975)
2018-12-09add missing "void" in prototype.Christos Zoulas
Reviewed-by: Richard Levitte <levitte@openssl.org> Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
2018-12-06Following the license change, modify the boilerplates in test/Richard Levitte
Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/7767)
2018-09-11Update copyright yearMatt Caswell
Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/7176)
2018-06-22testutil/driver.c: Fix function prototype warning [-Wstrict-prototypes]Dr. Matthias St. Pierre
(introduced by commit 91860165820d, which added -Wstrict-prototypes) Reviewed-by: Richard Levitte <levitte@openssl.org> Reviewed-by: Kurt Roeckx <kurt@roeckx.be> (Merged from https://github.com/openssl/openssl/pull/6555)
2018-06-21Add -Wstrict-prototypes option to --strict-warningsBernd Edlinger
[extended tests] Reviewed-by: Rich Salz <rsalz@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/6542)
2017-11-13Factorise duplicated code.FdaSilvaYY
Extract and factorise duplicated string glue code. Cache strlen result to avoid duplicate calls. [extended tests] Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4719)
2017-11-13Resolve warnings in VC-WIN32 build, which allows to add /WX.Andy Polyakov
It's argued that /WX allows to keep better focus on new code, which motivates its comeback... Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4721)
2017-08-22Use "" not <> for internal/ includesRich Salz
Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4217)
2017-08-18Remove tests dependence on e_os.hPauli
Apart from ssltest_old.c, the test suite relied on e_os.h for the OSSL_NELEM macro and nothing else. The ssltest_old.c also requires EXIT and some socket macros. Create a new header to define the OSSL_NELEM macro and use that instead. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4186)
2017-08-10Fix minor type warnings and risk of memory leak in testutil/driver.cDavid von Oheimb
Discussion is in https://github.com/openssl/openssl/issues/4127 Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4131)
2017-07-27Update the test framework so that the need for test_main is removed. EverythingPauli
that needed test_main now works using the same infrastructure as tests that used register_tests. This meant: * renaming register_tests to setup_tests and giving it a success/failure return. * renaming the init_test function to setup_test_framework. * renaming the finish_test function to pulldown_test_framework. * adding a user provided global_init function that runs before the test frame work is initialised. It returns a failure indication that stops the stest. * adding helper functions that permit tests to access their command line args. * spliting the BIO initialisation and finalisation out from the test setup and teardown. * hiding some of the now test internal functions. * fix the comments in testutil.h Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3953)
2017-06-16Add output routines to allow consistent formatting of memory, stringsPauli
and bignums. These have been refactored into their own file, along with their error displays. The formatting follows the output format used on error, except that bignums of sixty four bits or less are displayed in a more compact one line form. Added a TEST_note function for producing output without file and line information. Update the three tests that call BN_print so they use the new test infrastructure instead. Reviewed-by: Richard Levitte <levitte@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3655)
2017-05-15Randomise the ordering of the C unit tests.Pauli
Reviewed-by: Rich Salz <rsalz@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3393)
2017-05-12Add "Title" directive to evp_testRich Salz
Reviewed-by: Stephen Henson <steve@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3454)
2017-05-11testutil: Fix non-standard subtest outputRichard Levitte
In some cases, testutil outputs subtests like this: 1..6 # Subtest: progname The standard set by Test::More (because there really is no actual standard yet) gives this display: # Subtest: progname 1..6 Until the standard is actually agreed upon, let's do it like Test::More. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3446)
2017-05-11testutil: add the possibility to set the current test titleRichard Levitte
Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3445)
2017-04-28testutil: Add OpenSSL error stack printing wrapper TEST_openssl_errorsRichard Levitte
Also added a internal error printing callback to be used both with ERR_print_errors_cb() and with CRYPTO_mem_leaks_cb Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3345)
2017-04-28testutil: Add commodity printing functions test_printf_std{out,err}Richard Levitte
Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3345)
2017-04-28testutil: make subtest_level() internalRichard Levitte
Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3345)
2017-04-28testutil: Move printing function declarations to "internal" headerRichard Levitte
These functions aren't meant to be used directly by the test programs, reflect that by making the declarations a little harder to reach, but still available enough if there's a need to override them. Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3345)
2017-04-25Tapify libtestutil a bit betterRichard Levitte
This includes better signals of skips and subtests according to TAP 12, and flushing stdout and stderr at the end of every test function to make sure we get the output in good order. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3309)
2017-04-25TAPify testutilRichard Levitte
With the perl test framework comes the output format TAP (Test Anything Protocol, see http://testanything.org/) with extra extension for subtests. This change extends that same output format to any test program using testutil. In this implementation, each test program is seen as a full test that can be used as a subtest. The perl framework passes on the subtest level to the test programs with the environment variable HARNESS_OSSL_LEVEL. Furthermore, and series of tests added with ADD_ALL_TESTS is regarded as another subtest level. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3296)
2017-04-24Refactor the test framework testutilRichard Levitte
It's now built as a static library, and greatly simplified for test programs, which no longer need to include test_main_custom.h or test_main.h and link with the corresponding object files. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3243)
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-17 08:25:45 +0000