From f83133a5ed9757668c12d78256ab91c2ef78a921 Mon Sep 17 00:00:00 2001 From: Richard Levitte Date: Mon, 8 Feb 2016 05:11:25 +0100 Subject: unified build scheme: add a design document This documents describes the three steps from build.info files via the %unified_info database to the build-file templates, along with some examples showing how the data gets processed along the way. Reviewed-by: Viktor Dukhovni --- Configurations/README.design | 550 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 550 insertions(+) create mode 100644 Configurations/README.design (limited to 'Configurations/README.design') diff --git a/Configurations/README.design b/Configurations/README.design new file mode 100644 index 0000000000..ed2d25efa1 --- /dev/null +++ b/Configurations/README.design @@ -0,0 +1,550 @@ +Design document for the unified scheme data +=========================================== + +How are things connected? +------------------------- + +The unified scheme takes all its data from the build.info files seen +throughout the source tree. These files hold the minimum information +needed to build end product files from diverse sources. See the +section on build.info files below. + +From the information in build.info files, Configure builds up an +information database as a hash table called %unified_info, which is +stored in configdata.pm, found at the top of the build tree (which may +or may not be the same as the source tree). + +Configurations/common.tmpl uses the data from %unified_info to +generate the rules for building end product files as well as +intermediary files with the help of a few functions found in the +build-file templates. See the section on build-file templates further +down for more information. + +build.info files +---------------- + +As mentioned earlier, build.info files are meant to hold the minimum +information needed to build output files, and therefore only (with a +few possible exceptions [1]) have information about end products (such +as scripts, library files and programs) and source files (such as C +files, C header files, assembler files, etc). Intermediate files such +as object files are rarely directly refered to in build.info files (and +when they are, it's always with the file name extension .o), they are +infered by Configure. By the same rule of minimalism, end product +file name extensions (such as .so, .a, .exe, etc) are never mentioned +in build.info. Their file name extensions will be infered by the +build-file templates, adapted for the platform they are meant for (see +sections on %unified_info and build-file templates further down). + +The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare +end products. + +The variables SOURCE, DEPEND, INCLUDE and ORDINALS are indexed by a +produced file, and their values are the source used to produce that +particular produced file, extra dependencies, include directories +needed, and ordinal files (explained further below. + +All their values in all the build.info throughout the source tree are +collected together and form a set of programs, libraries, engines and +scripts to be produced, source files, dependencies, etc etc etc. + +Let's have a pretend example, a very limited contraption of OpenSSL, +composed of the program 'apps/openssl', the libraries 'libssl' and +'libcrypto', an engine 'engines/ossltest' and their sources and +dependencies. + + # build.info + LIBS=libcrypto libssl + ORDINALS[libcrypto]=crypto + ORDINALS[libssl]=ssl + INCLUDE[libcrypto]=include + INCLUDE[libssl]=include + DEPEND[libssl]=libcrypto + +This is the top directory build.info file, and it tells us that two +libraries are to be built, there are some ordinals to be used to +declare what symbols in those libraries are seen as public, the +include directory 'include/' shall be used throughout when building +anything that will end up in each library, and that the library +'libssl' depend on the library 'libcrypto' to function properly. + + # apps/build.info + PROGRAMS=openssl + SOURCE[openssl]=openssl.c + INCLUDE[openssl]=.. ../include + DEPEND[openssl]=../libssl + +This is the build.info file in 'apps/', one may notice that all file +paths mentioned are relative to the directory the build.info file is +located in. This one tells us that there's a program to be built +called 'apps/openssl' (the file name extension will depend on the +platform and is therefore not mentioned in the build.info file). It's +built from one source file, 'apps/openssl.c', and building it requires +the use of '.' and 'include' include directories (both are declared +from the point of view of the 'apps/' directory), and that the program +depends on the library 'libssl' to function properly. + + # crypto/build.info + LIBS=../libcrypto + SOURCE[../libcrypto]=aes.c evp.c cversion.c + DEPEND[cversion.o]=buildinf.h + + BEGINRAW[Makefile(unix)] + crypto/buildinf.h : Makefile + perl util/mkbuildinf.h "$(CC) $(CFLAGS)" "$(PLATFORM)" \ + > crypto/buildinf.h + ENDRAW[Makefile(unix)] + +This is the build.info file in 'crypto', and it tells us a little more +about what's needed to produce 'libcrypto'. LIBS is used again to +declare that 'libcrypto' is to be produced. This declaration is +really unnecessary as it's already mentioned in the top build.info +file, but can make the info file easier to understand. This is to +show that duplicate information isn't an issue. + +This build.info file informs us that 'libcrypto' is built from a few +source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'. +It also shows us that building the object file inferred from +'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it +also shows the possibility to include raw build-file statements in a +build.info file, in this case showing how 'buildinf.h' is built on +Unix-like operating systems. + +Two things are worth an extra note: + +'DEPEND[cversion.o]' mentiones an object file. DEPEND indexes is the +only location where it's valid to mention them + +Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as +seen from the top directory, no exception. + + # ssl/build.info + LIBS=../libssl + SOURCE[../libssl]=tls.c + +This is the build.info file in 'ssl/', and it tells us that the +library 'libssl' is built from the source file 'ssl/tls.c'. + + # engines/build.info + ENGINES=libossltest + SOURCE[libossltest]=e_ossltest.c + DEPEND[libossltest]=../libcrypto + INCLUDE[libossltest]=../include + +This is the build.info file in 'engines/', telling us that an engine +called 'engines/libossltest' shall be built, that it's source is +'engines/e_ossltest.c' and that the include directory 'include/' may +be used when building anything that will be part of this engine. +Finally, the engine 'engines/libossltest' depends on the library +'libcrypto' to function properly. + +When Configure digests these build.info files, the accumulated +information comes down to this: + + LIBS=libcrypto libssl + ORDINALS[libcrypto]=crypto + SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c + DEPEND[crypto/cversion.o]=crypto/buildinf.h + INCLUDE[libcrypto]=include + ORDINALS[libssl]=ssl + SOURCE[libssl]=ssl/tls.c + INCLUDE[libssl]=include + DEPEND[libssl]=libcrypto + + PROGRAMS=apps/openssl + SOURCE[apps/openssl]=apps/openssl.c + INCLUDE[apps/openssl]=. include + DEPEND[apps/openssl]=libssl + + ENGINES=engines/libossltest + SOURCE[engines/libossltest]=engines/e_ossltest.c + DEPEND[engines/libossltest]=libcrypto + INCLUDE[engines/libossltest]=include + + BEGINRAW[Makefile(unix)] + crypto/buildinf.h : Makefile + perl util/mkbuildinf.h "$(CC) $(CFLAGS)" "$(PLATFORM)" \ + > crypto/buildinf.h + ENDRAW[Makefile(unix)] + + +A few notes worth mentioning: + +LIBS may be used to declare routine libraries only. + +PROGRAMS may be used to declare programs only. + +ENGINES may be used to declare engines only. + +The indexes for SOURCE, INCLUDE and ORDINALS must only be end product +files, such as libraries, programs or engines. The values of SOURCE +variables must only be source files (possibly generated) + +DEPEND shows a relationship between different end product files, such +as a program depending on a library, or between an object file and +some extra source file. + +When Configure processes the build.info files, it will take it as +truth without question, and will therefore perform very few checks. +If the build tree is separate from the source tree, it will assume +that all built files and up in the build directory and that all source +files are to be found in the source tree, if they can be found there. +Configure will assume that source files that can't be found in the +source tree (such as 'crypto/bildinf.h' in the example above) are +generated and will be found in the build tree. + + +The %unified_info database +-------------------------- + +The information in all the build.info get digested by Configure and +collected into the %unified_info database, divided into the following +indexes: + + depends => a hash table containing 'file' => [ 'dependency' ... ] + pairs. These are directly inferred from the DEPEND + variables in build.info files. + + engines => a list of engines. These are directly inferred from + the ENGINES variable in build.info files. + + includes => a hash table containing 'file' => [ 'include' ... ] + pairs. These are directly inferred from the INCLUDE + variables in build.info files. + + libraries => a list of libraries. These are directly inferred from + the LIBS variable in build.info files. + + ordinals => a hash table containing 'file' => [ 'word', 'ordfile' ] + pairs. 'file' and 'word' are directly inferred from + the ORDINALS variables in build.info files, while the + file 'ofile' comes from internal knowledge in + Configure. + + programs => a list of programs. These are directly inferred from + the PROGRAMS variable in build.info files. + + rawlines => a list of build-file lines. These are a direct copy of + the BEGINRAW..ENDRAW lines in build.info files. Note: + only the BEGINRAW..ENDRAW section for the current + platform are copied, the rest are ignored. + + scripts => a list of scripts. There are directly inferred from + the SCRIPTS variable in build.info files. + + sources => a hash table containing 'file' => [ 'sourcefile' ... ] + pairs. These are indirectly inferred from the SOURCE + variables in build.info files. Object files are + mentioned in this hash table, with source files from + SOURCE variables, and AS source files for programs and + libraries. + +As an example, here is how the build.info files example from the +section above would be digested into a %unified_info table: + + our %unified_info = ( + "depends" => + { + "apps/openssl" => + [ + "libssl", + ], + "crypto/cversion.o" => + [ + "crypto/buildinf.h", + ], + "engines/libossltest" => + [ + "libcrypto", + ], + "libssl" => + [ + "libcrypto", + ], + }, + "engines" => + [ + "engines/libossltest", + ], + "includes" => + { + "apps/openssl" => + [ + ".", + "include", + ], + "engines/libossltest" => + [ + "include" + ], + "libcrypto" => + [ + "include", + ], + "libssl" => + [ + "include", + ], + } + "libraries" => + [ + "libcrypto", + "libssl", + ], + "ordinals" => + { + "libcrypto" => + [ + "crypto", + "util/libeay.num", + ], + "libssl" => + [ + "ssl", + "util/ssleay.num", + ], + }, + "programs" => + [ + "apps/openssl", + ], + "rawlines" => + [ + "crypto/buildinf.h : Makefile", + " perl util/mkbuildinf.h \"\$(CC) \$(CFLAGS)\" \"\$(PLATFORM)\" \\" + " > crypto/buildinf.h" + ], + "sources" => + { + "apps/openssl" => + [ + "apps/openssl.o", + ], + "apps/openssl.o" => + [ + "apps/openssl.c", + ], + "crypto/aes.o" => + [ + "crypto/aes.c", + ], + "crypto/cversion.o" => + [ + "crypto/cversion.c", + ], + "crypto/evp.o" => + [ + "crypto/evp.c", + ], + "engines/e_ossltest.o" => + [ + "engines/e_ossltest.c", + ], + "engines/libossltest" => + [ + "engines/e_ossltest.o", + ], + "libcrypto" => + [ + "crypto/aes.c", + "crypto/cversion.c", + "crypto/evp.c", + ], + "libssl" => + [ + "ssl/tls.c", + ], + "ssl/tls.o" => + [ + "ssl/tls.c", + ], + }, + ); + +As can be seen, everything in %unified_info is fairly simple nuggest +of information. Still, it tells us that to build all programs, we +must build 'apps/openssl', and to build the latter, we will need to +build all its sources ('apps/openssl.o' in this case) and all the +other things it depends on (such as 'libssl'). All those dependencies +need to be built as well, using the same logic, so to build 'libssl', +we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the +latter... + + +Build-file templates +-------------------- + +Build-file templates are essentially build-files (such as Makefile on +Unix) with perl code fragments mixed in. Those perl code fragment +will generate all the configuration dependent data, including all the +rules needed to build end product files and intermediary files alike. +At a minimum, there must be a perl code fragment that defines a set of +functions that are used to generates specific build-file rules, to +build static libraries from object files, to build shared libraries +from static libraries, to programs from object files and libraries, +etc. + + src2dep - function that produces build file lines to get the + dependencies for an object file into a dependency + file. + + It's called like this: + + src2dep(obj => "PATH/TO/objectfile", + srcs => [ "PATH/TO/sourcefile", ... ], + incs => [ "INCL/PATH", ... ]); + + 'obj' has the dependent object file as well as + object file the dependencies are for; it's *without* + extension, src2dep() is expected to add that. + 'srcs' has the list of source files to build the + object file, with the first item being the source + file that directly corresponds to the object file. + 'incs' is a list of include file directories. + + src2obj - function that produces build file lines to build an + object file from source files and associated data. + + It's called like this: + + src2obj(obj => "PATH/TO/objectfile", + srcs => [ "PATH/TO/sourcefile", ... ], + deps => [ "dep1", ... ], + incs => [ "INCL/PATH", ... ]); + + 'obj' has the intended object file *without* + extension, src2obj() is expected to add that. + 'srcs' has the list of source files to build the + object file, with the first item being the source + file that directly corresponds to the object file. + 'deps' is a list of dependencies. 'incs' is a list + of include file directories. + + obj2lib - function that produces build file lines to build a + static library file ("libfoo.a" in Unix terms) from + object files. + + called like this: + + obj2lib(lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ]); + + 'lib' has the intended library file name *without* + extension, obj2lib is expected to add that. 'objs' + has the list of object files (also *without* + extension) to build this library. + + libobj2shlib - function that produces build file lines to build a + shareable object library file ("libfoo.so" in Unix + terms) from the corresponding static library file + or object files. + + called like this: + + libobj2shlib(shlib => "PATH/TO/shlibfile", + lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/otherlibfile", ... ], + ordinals => [ "word", "/PATH/TO/ordfile" ]); + + 'lib' has the intended library file name *without* + extension, libobj2shlib is expected to add that. + 'shlib' has the correcponding shared library name + *without* extension. 'deps' has the list of other + libraries (also *without* extension) this library + needs to be linked with. 'objs' has the list of + object files (also *without* extension) to build + this library. 'ordinals' MAY be present, and when + it is, its value is an array where the word is + "crypto" or "ssl" and the file is one of the ordinal + files util/libeay.num or util/ssleay.num in the + source directory. + + This function has a choice; it can use the + corresponding static library as input to make the + shared library, or the list of object files. + + obj2dynlib - function that produces build file lines to build a + dynamically loadable library file ("libfoo.so" on + Unix) from object files. + + called like this: + + obj2dynlib(lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/otherlibfile", + ... ]); + + This is almost the same as libobj2shlib, but the + intent is to build a shareable library that can be + loaded in runtime (a "plugin"...). The differences + are subtle, one of the most visible ones is that the + resulting shareable library is produced from object + files only. + + obj2bin - function that produces build file lines to build an + executable file from object files. + + called like this: + + obj2bin(bin => "PATH/TO/binfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/libfile", ... ]); + + 'bin' has the intended executable file name + *without* extension, obj2bin is expected to add + that. 'objs' has the list of object files (also + *without* extension) to build this library. 'deps' + has the list of library files (also *without* + extension) that the programs needs to be linked + with. + + in2script - function that produces build file lines to build a + script file from some input. + + called like this: + + in2script(script => "PATH/TO/scriptfile", + sources => [ "PATH/TO/infile", ... ]); + + 'script' has the intended script file name. + 'sources' has the list of source files to build the + resulting script from. + +Along with the build-file templates is the driving engine +Configurations/common.tmpl, which looks through all the information in +%unified_info and generates all the rulesets to build libraries, +programs and all intermediate files, using the rule generating +functions defined in the build-file template. + +As an example with the smaller build.info set we've seen as an +example, producing the rules to build 'libssl' would result in the +following calls: + + # Note: libobj2shlib will only be called if shared libraries are + # to be produced. + # Note 2: libobj2shlib gets both the name of the static library + # and the names of all the object files that go into it. It's up + # to the implementation to decide which to use as input. + libobj2shlib(shlib => "libssl", + lib => "libssl", + objs => [ "ssl/tls.o" ], + deps => [ "libcrypto" ] + ordinals => [ "ssl", "util/ssleay.num" ]); + + obj2lib(lib => "libssl" + objs => [ "ssl/tls.o" ]); + + # Note 3: common.tmpl peals off the ".o" extension, as the + # platform at hand may have a different one. + src2obj(obj => "ssl/tls" + srcs => [ "ssl/tls.c" ], + deps => [ ], + incs => [ "include" ]); + + src2dep(obj => "ssl/tls" + srcs => [ "ssl/tls.c" ], + incs => [ "include" ]); + +The returned strings from all those calls are then concatenated +together and written to the resulting build-file. -- cgit v1.2.3