summaryrefslogtreecommitdiffstats
path: root/test/run_tests.pl
AgeCommit message (Collapse)Author
2018-12-06Following the license change, modify the boilerplates in test/Richard Levitte
Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/7767)
2018-05-01Update copyright yearMatt Caswell
Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/6145)
2018-04-30Use the config file from the source not the host for the testsKurt Roeckx
Fixes: #6046 Reviewed-by: Rich Salz <rsalz@openssl.org> GH: #6125
2017-09-02test/run_tests.pl: don't use Module::Load::Conditional.Andy Polyakov
Ironically enough not all installations get Module::Load::Conditional installed by default... [It's a bit half-hearted, because such installations are likely to lack more stuffi that is needed, but nevertheless, it proved to be helpful.] Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4323)
2017-08-15File::Glob option ':bsd_glob' doesn't work everywhere, replace w/ a wrapperRichard Levitte
Reviewed-by: Andy Polyakov <appro@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4069)
2017-08-15Consolidate the locations where we have our internal perl modulesRichard Levitte
Instead of having perl modules under test/testlib, util and util/perl, consolidate them all to be inside util/perl. Reviewed-by: Andy Polyakov <appro@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4069)
2017-07-28Perl: Use File::Glob::bsd_glob rather than File::Glob::globRichard Levitte
File::Glob::glob is deprecated, it's use generates this kind of message: File::Glob::glob() will disappear in perl 5.30. Use File::Glob::bsd_glob() instead. at ../master/Configure line 277. So instead, use a construction that makes the caller glob() use File::Glob::bsd_glob(). Note that we're still excluding VMS, as it's directory specs use '[' and ']', which have a different meaning with bsd_glob and would need some extra quoting. This might change, but later. Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/4040)
2017-07-07test/run_tests.pl: Make sure to exit with a code that's understood universallyRichard Levitte
TAP::Parser::Aggregator::has_errors may return any number, not just 0 and 1. With Perl on VMS, any number from 2 and on is interpreted as a VMS status, the 3 lower bits are the encoded severity (1 = SUCCESS, for example), so depending on what has_errors returns, a test failure might be interpreted as a success. Therefore, it's better to make sure the exit code is 0 or 1, nothing else (they are special on VMS, and mean SUCCESS or FAILURE, to match Unix conventions). Reviewed-by: Tim Hudson <tjh@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3880)
2017-05-26Fix use of "can_load()" in run_tests.pl.Rainer Jung
CLA: Trivial Fixes #3563. Reviewed-by: Andy Polyakov <appro@openssl.org> Reviewed-by: Richard Levitte <levitte@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3564)
2017-05-21test/run_tests.pl: don't mask test failures.Andy Polyakov
Switch to TAP::Harness inadvertently masked test failures. Test::Harness::runtests was terminating with non-zero exit code in case of failure[s], while TAP::Harness apparently holds caller responsible for doing so. Reviewed-by: Rich Salz <rsalz@openssl.org>
2017-05-10Prefer TAP::Harness over Test::HarnessRichard Levitte
TAP:Harness came along in perl 5.10.1, and since we claim to support perl 5.10.0 in configuration and testing, we can only load it conditionally. The main reason to use TAP::Harness rather than Test::Harness is its capability to merge stdout and stderr output from the test recipes, which Test::Harness can't. The merge gives much more comprehensible output when testing verbosely. Reviewed-by: Rich Salz <rsalz@openssl.org> Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3424)
2017-03-10Make it possible to select or deselect test groups by numberRichard Levitte
Examples of possible expressions (adapt to your platform): make test TESTS=-99 make test TESTS=10 make test TESTS=-9? make test TESTS=-[89]0 make test TESTS=[89]0 Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/2887)
2016-09-01Add a more versatile test chooserRichard Levitte
So far, the test runner (test/run_tests.pl) could get a list of tests to run, and if non were given, it assumes all available tests should be performed. However, that makes skipping just one or two tests a bit of a pain. This change makes the possibilities more versatile, run_checker.pl takes these arguments and will process them in the given order, starting with an empty set of tests to perform: alltests The current set becomes the whole set of available tests. test_xxx Adds 'test_xxx' to the current set. -test_xxx Removes 'test_xxx' from the current set. If nothing has been added to the set before this argument, the current set is first initialised to the whole set of available tests, then 'test_xxx' is removed from the current set. list Display all available tests, then stop. If no arguments are given, 'alltests' is assumed. Reviewed-by: Emilia Käsper <emilia@openssl.org>
2016-05-30perl: use the 'if' module to conditionally load File::GlobRichard Levitte
Trying to use normal perl conditions to conditionally 'use' a perl module didn't quite work. Using the 'if' module to do so does work. Reviewed-by: Andy Polyakov <appro@openssl.org>
2016-05-29Configure,test/recipes: "pin" glob to File::Glob::glob.Andy Polyakov
As it turns out default glob's behaviour for quoted argument varies from version to version, making it impossible to Configure or run tests in some cases. The reason for quoting globs was to accommodate source path with spaces in its name, which was treated by default glob as multiple paths. File::Glob::glob on the other hand doesn't consider spaces as delimiters and therefore works with unquoted patterns. [Unfortunaltely File::Glob::glob, being too csh-ly, doesn't work on VMS, hence the "pinning" is conditional.] Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-05-27perl glob: make sure to put quotes around the pattern, in case of spacesRichard Levitte
RT#4486 Reviewed-by: Matt Caswell <matt@openssl.org>
2016-05-11Replace cipherlist testEmilia Kasper
The old cipherlist test in ssltest.c only tests the internal order of the cipher table, which is pretty useless. Replace this test with a test that catches inadvertent changes to the default cipherlist. Fix run_tests.pl to correctly filter tests that have "list" in their name. (Also includes a small drive-by fix in .gitignore.) Reviewed-by: Rich Salz <rsalz@openssl.org>
2016-05-09Recognise VERBOSE and V as well as HARNESS_VERBOSEDr. Stephen Henson
PR#4462 Reviewed-by: Rich Salz <rsalz@openssl.org>
2016-04-20Copyright consolidation: perl filesRich Salz
Add copyright to most .pl files This does NOT cover any .pl file that has other copyright in it. Most of those are Andy's but some are public domain. Fix typo's in some existing files. Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-02-09unified build scheme: adjust test framework for out of source build treeRichard Levitte
To be able to run tests when we've built in a directory other than the source tree, the testing framework needs a few adjustments. test/testlib/OpenSSL/Test.pm needs to know where it can find shlib_wrap.sh, and a number of other tests need to be told a different place to find engines than what they may be able to figure out on their own. Relying to $TOP is not enough, $SRCTOP and $BLDTOP can be used as an alternative. As part of this change, top_file and top_dir are removed and srctop_file, bldtop_file, srctop_dir and bldtop_dir take their place. Reviewed-by: Ben Laurie <ben@openssl.org>
2016-01-22few typo fixesFdaSilvaYY
Signed-off-by: Rich Salz <rsalz@openssl.org> Reviewed-by: Kurt Roeckx <kurt@openssl.org>
2015-09-15Add a method to list available testsRichard Levitte
Reviewed-by: Emilia Käsper <emilia@openssl.org>
2015-09-07Adapt the libssl test harness testing scripts to new testing frameworkRichard Levitte
This involves adding $TOP/util as perl library in test/run_tests.pl. Reviewed-by: Rich Salz <rsalz@openssl.org>
2015-09-07Groundwork for a perl based testing frameworkRichard Levitte
The idea with this perl based testing framework is to make use of what's delivered with perl and exists on all sorts of platforms. The choice came to using Test::More and Test::Harness, as that seems to be the most widely spread foundation, even if perl is aged. The main runner of the show is run_tests.pl. As it currently stands, it's designed to run from inside Makefile, but it's absolutely possible to run it from the command line as well, like so: cd test OPENSSL_SRCDIR=.. perl run_tests.pl The tester scripts themselves are stored in the subdirectory recipes/, and initially, we have two such scripts, recipes/00-check_testalltests.t and recipes/00-check_testexes.t. recipes/00-check_testalltests.t will pick out the dependencies of "alltests" in test/Makefile, and check if it can find recipes with corresponding names. recipes/00-check_testexes.t does something similar, but bases it on existing compiled test binaries. They make it easy to figure out what's to be added, and will be removed when this effort is finished. Individual recipes can be run as well, of course, as they are perl scripts in themselves. For example, you can run only recipes/00-check_testexes.t like so: cd test OPENSSL_SRCDIR=.. perl recipes/00-check_testexes.t To make coding easier, there's a routine library OpenSSL::Test, which is reachable in a perl script like so: use lib 'testlib'; use OpenSSL::Test; Reviewed-by: Rich Salz <rsalz@openssl.org>