1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
|
RADIX Test Framework
====================
Purpose
-------
This directory contains the RADIX test framework, which is a six-dimension
script-driven facility intended to facilitate execution of
- multi-stream
- multi-client
- multi-server
- multi-thread
- multi-process (in future)
- multi-node (in future)
test vignettes for network protocol testing applications. While it is currently
used for QUIC, it has been designed to be agnostic so that it can be adapted to
other protocols in future if desired.
In particular, unilke the older multistream test framework, it does not assume a
single client and a single server. Examples of vignettes designed to be
supported by the RADIX test framework in future include:
- single client ↔ single server
- multiple clients ↔ single server
- single client ↔ multiple servers
- multiple clients ↔ multiple servers
“Multi-process” and “multi-node” means there has been some consideration
given to support of multi-process and multi-node testing in the future, though
this is not currently supported.
Differences to `quic_multistream_test`
--------------------------------------
The RADIX test features the following improvements relative to the
`quic_multistream_test` framework:
- Due to usage of the new QUIC server API, opcodes are no longer duplicated
into “C” and “S” variants. There is symmetry between all endpoints in this
regard. The legacy `QUIC_TSERVER` facility is not used and will eventually
be retired once other test suites no longer rely on it.
- The framework is not limited to two objects (client and server),
but can instead instantiate an arbitrary number of SSL objects and pass these
to operations. Other kinds of object could be supported in future if needed.
- Scripts are now generated dynamically at launch time by functions rather
than hardcoding them into the executable. This has the advantage that scripts
can be generated dynamically. In particular, this allows very long
procedurally generated test scripts which would be impractical to write
by hand. This can be used for stress testing, for example.
- As a result of the fact that scripts are now generated dynamically, the
in-memory representation of script operations has been improved as the ability
to write a script operation as an initializer list is no longer required. The
new representation is simpler, more compact, and more flexible. A stack-based
approach allows arbitrary argument types to be passed as operands to an
operation, rather than having a fixed number of operands of fixed type for all
script operations.
- Scripts are now named, rather than numbered, giving more useful debug output,
and hopefully reducing the frequency of merge conflicts.
- Debug logging has in general been significantly improved and has been designed
to be accessible and useful.
- Logging of child threads is now buffered and printed after a test is complete,
which avoids non-deterministic interleaving of test output.
- The number of core opcodes for the interpreter has been dramatically reduced
and now the vast majority of test operations are performed via `OP_FUNC`,
which is similar to `OP_CHECK`. This change is largely transparent to the
test developer.
- In the future, multi-process or multi-node testing will be supported.
The expectation is that multi-node testing will be facilitated by having the
master process invoke a hook shell script which is responsible for propping up
and tearing down the additional nodes. Writing a suitable hook script will be
left as an exercise to the test infrastructure maintainer. This abstracts the
test framework from the details of a specific infrastructure environment and
its management tools (VMs, containers, etc.).
- The core test interpreter is designed to be agnostic to QUIC and could be
used for testing other protocols in the future. There is a clean, layered
design which draws a clear distinction between the core interpreter,
protocol-specific bindings and support code, test operation definitions, and
script definitions.
Architecture
------------
The RADIX test suite framework is built in four layers:
- **TERP** ([terp.c](./terp.c)), a protocol-agnostic stack-based script
interpreter for interruptible execution of test vignettes;
- the **QUIC bindings** ([quic_bindings.c](./quic_bindings.c)), which defines
QUIC-specific test framework;
- the **QUIC operations** ([quic_ops.c](./quic_ops.c)), which define specific
test operations for TERP which can be invoked by QUIC unit tests on top of the
basic infrastructure defined by the QUIC bindings;
- the QUIC unit tests ([quic_tests.c](./quic_tests.c)), which use the above
QUIC bindings.
|
