path: root/Configurations/README-design.md

Design document for the unified scheme data
===========================================

How are things connected?
-------------------------

The unified scheme takes all its data from the `build.info` files seen
throughout the source tree.  These files hold the minimum information
needed to build end product files from diverse sources.  See the
section on `build.info` files below.

From the information in `build.info` files, `Configure` builds up an
information database as a hash table called `%unified_info`, which is
stored in configdata.pm, found at the top of the build tree (which may
or may not be the same as the source tree).

[`Configurations/common.tmpl`](common.tmpl) uses the data from `%unified_info` to
generate the rules for building end product files as well as
intermediary files with the help of a few functions found in the
build-file templates.  See the section on build-file templates further
down for more information.

build.info files
----------------

As mentioned earlier, `build.info` files are meant to hold the minimum
information needed to build output files, and therefore only (with a
few possible exceptions [1]) have information about end products (such
as scripts, library files and programs) and source files (such as C
files, C header files, assembler files, etc).  Intermediate files such
as object files are rarely directly referred to in `build.info` files (and
when they are, it's always with the file name extension `.o`), they are
inferred by `Configure`.  By the same rule of minimalism, end product
file name extensions (such as `.so`, `.a`, `.exe`, etc) are never mentioned
in `build.info`.  Their file name extensions will be inferred by the
build-file templates, adapted for the platform they are meant for (see
sections on `%unified_info` and build-file templates further down).

The variables `PROGRAMS`, `LIBS`, `MODULES` and `SCRIPTS` are used to declare
end products.  There are variants for them with `_NO_INST` as suffix
(`PROGRAM_NO_INST` etc) to specify end products that shouldn't get installed.

The variables `SOURCE`, `DEPEND`, `INCLUDE` and `DEFINE` are indexed by a
produced file, and their values are the source used to produce that
particular produced file, extra dependencies, include directories
needed, or C macros to be defined.

All their values in all the `build.info` throughout the source tree are
collected together and form a set of programs, libraries, modules and
scripts to be produced, source files, dependencies, etc etc etc.

Let's have a pretend example, a very limited contraption of OpenSSL,
composed of the program `apps/openssl`, the libraries `libssl` and
`libcrypto`, an module `engines/ossltest` and their sources and
dependencies.

    # build.info
    LIBS=libcrypto libssl
    INCLUDE[libcrypto]=include
    INCLUDE[libssl]=include
    DEPEND[libssl]=libcrypto

This is the top directory `build.info` file, and it tells us that two
libraries are to be built, the include directory `include/` shall be
used throughout when building anything that will end up in each
library, and that the library `libssl` depend on the library
`libcrypto` to function properly.

    # apps/build.info
    PROGRAMS=openssl
    SOURCE[openssl]=openssl.c
    INCLUDE[openssl]=.. ../include
    DEPEND[openssl]=../libssl

This is the `build.info` file in `apps/`, one may notice that all file
paths mentioned are relative to the directory the `build.info` file is
located in.  This one tells us that there's a program to be built
called `apps/openss` (the file name extension will depend on the
platform and is therefore not mentioned in the `build.info` file).  It's
built from one source file, `apps/openssl.c`, and building it requires
the use of `.` and `include/` include directories (both are declared
from the point of view of the `apps/` directory), and that the program
depends on the library `libssl` to function properly.

    # crypto/build.info
    LIBS=../libcrypto
    SOURCE[../libcrypto]=aes.c evp.c cversion.c
    DEPEND[cversion.o]=buildinf.h

    GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
    DEPEND[buildinf.h]=../Makefile
    DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm

This is the `build.info` file in `crypto/`, and it tells us a little more
about what's needed to produce `libcrypto`.  LIBS is used again to
declare that `libcrypto` is to be produced.  This declaration is
really unnecessary as it's already mentioned in the top `build.info`
file, but can make the info file easier to understand.  This is to
show that duplicate information isn't an issue.

This `build.info` file informs us that `libcrypto` is built from a few
source files, `crypto/aes.c`, `crypto/evp.c` and `crypto/cversion.c`.
It also shows us that building the object file inferred from
`crypto/cversion.c` depends on `crypto/buildinf.h`.  Finally, it
also shows the possibility to declare how some files are generated
using some script, in this case a perl script, and how such scripts
can be declared to depend on other files, in this case a perl module.

Two things are worth an extra note:

`DEPEND[cversion.o]` mentions an object file.  DEPEND indexes is the
only location where it's valid to mention them

    # ssl/build.info
    LIBS=../libssl
    SOURCE[../libssl]=tls.c

This is the build.info file in `ssl/`, and it tells us that the
library `libssl` is built from the source file `ssl/tls.c`.

    # engines/build.info
    MODULES=dasync
    SOURCE[dasync]=e_dasync.c
    DEPEND[dasync]=../libcrypto
    INCLUDE[dasync]=../include

    MODULES_NO_INST=ossltest
    SOURCE[ossltest]=e_ossltest.c
    DEPEND[ossltest]=../libcrypto.a
    INCLUDE[ossltest]=../include

This is the `build.info` file in `engines/`, telling us that two modules
called `engines/dasync` and `engines/ossltest` shall be built, that
`dasync`'s source is `engines/e_dasync.c` and `ossltest`'s source is
`engines/e_ossltest.c` and that the include directory `include/` may
be used when building anything that will be part of these modules.
Also, both modules depend on the library `libcrypto` to function
properly.  `ossltest` is explicitly linked with the static variant of
the library `libcrypto`.  Finally, only `dasync` is being installed, as
`ossltest` is only for internal testing.

When `Configure` digests these `build.info` files, the accumulated
information comes down to this:

    LIBS=libcrypto libssl
    SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
    DEPEND[crypto/cversion.o]=crypto/buildinf.h
    INCLUDE[libcrypto]=include
    SOURCE[libssl]=ssl/tls.c
    INCLUDE[libssl]=include
    DEPEND[libssl]=libcrypto

    PROGRAMS=apps/openssl
    SOURCE[apps/openssl]=apps/openssl.c
    INCLUDE[apps/openssl]=. include
    DEPEND[apps/openssl]=libssl

    MODULES=engines/dasync
    SOURCE[engines/dasync]=engines/e_dasync.c
    DEPEND[engines/dasync]=libcrypto
    INCLUDE[engines/dasync]=include

    MODULES_NO_INST=engines/ossltest
    SOURCE[engines/ossltest]=engines/e_ossltest.c
    DEPEND[engines/ossltest]=libcrypto.a
    INCLUDE[engines/ossltest]=include

    GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
    DEPEND[crypto/buildinf.h]=Makefile
    DEPEND[util/mkbuildinf.pl]=util/Foo.pm

A few notes worth mentioning:

`LIBS` may be used to declare routine libraries only.

`PROGRAMS` may be used to declare programs only.

`MODULES` may be used to declare modules only.

The indexes for `SOURCE` must only be end product files, such as
libraries, programs or modules.  The values of `SOURCE` variables must
only be source files (possibly generated).

`INCLUDE` and `DEPEND` shows a relationship between different files
(usually produced files) or between files and directories, such as a
program depending on a library, or between an object file and some
extra source file.

When `Configure` processes the `build.info` files, it will take it as
truth without question, and will therefore perform very few checks.
If the build tree is separate from the source tree, it will assume
that all built files and up in the build directory and that all source
files are to be found in the source tree, if they can be found there.
`Configure` will assume that source files that can't be found in the
source tree (such as `crypto/bildinf.h` in the example above) are
generated and will be found in the build tree.

The `%unified_info` database
----------------------------

The information in all the `build.info` get digested by `Configure` and
collected into the `%unified_info` database, divided into the following
indexes:

    depends   => a hash table containing 'file' => [ 'dependency' ... ]
                 pairs.  These are directly inferred from the DEPEND
                 variables in build.info files.

    modules   => a list of modules.  These are directly inferred from
                 the MODULES variable in build.info files.

    generate  => a hash table containing 'file' => [ 'generator' ... ]
                 pairs.  These are directly inferred from the GENERATE
                 variables in build.info files.

    includes  => a hash table containing 'file' => [ 'include' ... ]
                 pairs.  These are directly inferred from the INCLUDE
                 variables in build.info files.

    install   => a hash table containing 'type' => [ 'file' ... ] pairs.
                 The types are 'programs', 'libraries', 'modules' and
                 'scripts', and the array of files list the files of
                 that type that should be installed.

    libraries => a list of libraries.  These are directly inferred from
                 the LIBS variable in build.info files.

    programs  => a list of programs.  These are directly inferred from
                 the PROGRAMS variable in build.info files.

    scripts   => a list of scripts.  There are directly inferred from
                 the SCRIPTS variable in build.info files.

    sources   => a hash table containing 'file' => [ 'sourcefile' ... ]
                 pairs.  These are indirectly inferred from the SOURCE
                 variables in build.info files.  Object files are
                 mentioned in this hash table, with source files from
                 SOURCE variables, and AS source files for programs and
                 libraries.

    shared_sources =>
                 a hash table just like 'sources', but only as source
                 files (object files) for building shared libraries.

As an example, here is how the `build.info` files example from the
section above would be digested into a `%unified_info` table:

    our %unified_info = (
        "depends" =>
            {
                "apps/openssl" =>
                    [
                        "libssl",
                    ],
                "crypto/buildinf.h" =>
                    [
                        "Makefile",
                    ],
                "crypto/cversion.o" =>
                    [
                        "crypto/buildinf.h",
                    ],
                "engines/dasync" =>
                    [
                        "libcrypto",
                    ],
                "engines/ossltest" =>
                    [
                        "libcrypto.a",
                    ],
                "libssl" =>
                    [
                        "libcrypto",
                    ],
                "util/mkbuildinf.pl" =>
                    [
                        "util/Foo.pm",
                    ],
            },
        "modules" =>
            [
                "engines/dasync",
                "engines/ossltest",
            ],
        "generate" =>
            {
                "crypto/buildinf.h" =>
                    [
                        "util/mkbuildinf.pl",
                        "\"\$(CC)",
                        "\$(CFLAGS)\"",
                        "\"$(PLATFORM)\"",
                    ],
            },
        "includes" =>
            {
                "apps/openssl" =>
                    [
                        ".",
                        "include",
                    ],
                "engines/ossltest" =>
                    [
                        "include"
                    ],
                "libcrypto" =>
                    [
                        "include",
                    ],
                "libssl" =>
                    [
                        "include",
                    ],
                "util/mkbuildinf.pl" =>
                    [
                        "util",
                    ],
            }
        "install" =>
            {
                "modules" =>
                    [
                        "engines/dasync",
                    ],
                "libraries" =>
                    [
                        "libcrypto",
                        "libssl",
                    ],
                "programs" =>
                    [
                        "apps/openssl",
                    ],
           },
        "libraries" =>
            [
                "libcrypto",
                "libssl",
            ],
        "programs" =>
            [
                "apps/openssl",
            ],
        "sources" =>
            {
                "apps/openssl" =>
                    [
                        "apps/openssl.o",
                    ],
                "apps/openssl.o" =>
                    [
                        "apps/openssl.c",
                    ],
                "crypto/aes.o" =>
                    [
                        "crypto/aes.c",
                    ],
                "crypto/cversion.o" =>
                    [
                        "crypto/cversion.c",
                    ],
                "crypto/evp.o" =>
                    [
                        "crypto/evp.c",
                    ],
                "engines/e_dasync.o" =>
                    [
                        "engines/e_dasync.c",
                    ],
                "engines/dasync" =>
                    [
                        "engines/e_dasync.o",
                    ],
                "engines/e_ossltest.o" =>
                    [
                        "engines/e_ossltest.c",
                    ],
                "engines/ossltest" =>
                    [
                        "engines/e_ossltest.o",
                    ],
                "libcrypto" =>
                    [
                        "crypto/aes.c",
                        "crypto/cversion.c",
                        "crypto/evp.c",
                    ],
                "libssl" =>
                    [
                        "ssl/tls.c",
                    ],
                "ssl/tls.o" =>
                    [
                        "ssl/tls.c",
                    ],
            },
    );

As can be seen, everything in `%unified_info` is fairly simple suggest
of information.  Still, it tells us that to build all programs, we
must build `apps/openssl`, and to build the latter, we will need to
build all its sources (`apps/openssl.o` in this case) and all the
other things it depends on (such as `libssl`).  All those dependencies
need to be built as well, using the same logic, so to build `libssl`,
we need to build `ssl/tls.o` as well as `libcrypto`, and to build the
latter...

Build-file templates
--------------------

Build-file templates are essentially build-files (such as `Makefile` on
Unix) with perl code fragments mixed in.  Those perl code fragment
will generate all the configuration dependent data, including all the
rules needed to build end product files and intermediary files alike.
At a minimum, there must be a perl code fragment that defines a set of
functions that are used to generates specific build-file rules, to
build static libraries from object files, to build shared libraries
from static libraries, to programs from object files and libraries,
etc.

    generatesrc - function that produces build file lines to generate
                  a source file from some input.

                  It's called like this:

                        generatesrc(src => "PATH/TO/tobegenerated",
                                    generator => [ "generatingfile", ... ]
                                    generator_incs => [ "INCL/PATH", ... ]
                                    generator_deps => [ "dep1", ... ]
                                    incs => [ "INCL/PATH", ... ],
                                    deps => [ "dep1", ... ],
                                    intent => one of "libs", "dso", "bin" );

                  'src' has the name of the file to be generated.
                  'generator' is the command or part of command to
                  generate the file, of which the first item is
                  expected to be the file to generate from.
                  generatesrc() is expected to analyse and figure out
                  exactly how to apply that file and how to capture
                  the result.  'generator_incs' and 'generator_deps'
                  are include directories and files that the generator
                  file itself depends on.  'incs' and 'deps' are
                  include directories and files that are used if $(CC)
                  is used as an intermediary step when generating the
                  end product (the file indicated by 'src').  'intent'
                  indicates what the generated file is going to be
                  used for.

    src2obj     - function that produces build file lines to build an
                  object file from source files and associated data.

                  It's called like this:

                        src2obj(obj => "PATH/TO/objectfile",
                                srcs => [ "PATH/TO/sourcefile", ... ],
                                deps => [ "dep1", ... ],
                                incs => [ "INCL/PATH", ... ]
                                intent => one of "lib", "dso", "bin" );

                  'obj' has the intended object file with `.o`
                  extension, src2obj() is expected to change it to
                  something more suitable for the platform.
                  'srcs' has the list of source files to build the
                  object file, with the first item being the source
                  file that directly corresponds to the object file.
                  'deps' is a list of explicit dependencies.  'incs'
                  is a list of include file directories.  Finally,
                  'intent' indicates what this object file is going
                  to be used for.

    obj2lib     - function that produces build file lines to build a
                  static library file ("libfoo.a" in Unix terms) from
                  object files.

                  called like this:

                        obj2lib(lib => "PATH/TO/libfile",
                                objs => [ "PATH/TO/objectfile", ... ]);

                  'lib' has the intended library file name *without*
                  extension, obj2lib is expected to add that.  'objs'
                  has the list of object files to build this library.

    libobj2shlib - backward compatibility function that's used the
                  same way as obj2shlib (described next), and was
                  expected to build the shared library from the
                  corresponding static library when that was suitable.
                  NOTE: building a shared library from a static
                  library is now DEPRECATED, as they no longer share
                  object files.  Attempting to do this will fail.

    obj2shlib   - function that produces build file lines to build a
                  shareable object library file ("libfoo.so" in Unix
                  terms) from the corresponding object files.

                  called like this:

                        obj2shlib(shlib => "PATH/TO/shlibfile",
                                  lib => "PATH/TO/libfile",
                                  objs => [ "PATH/TO/objectfile", ... ],
                                  deps => [ "PATH/TO/otherlibfile", ... ]);

                  'lib' has the base (static) library file name
                  *without* extension.  This is useful in case
                  supporting files are needed (such as import
                  libraries on Windows).
                  'shlib' has the corresponding shared library name
                  *without* extension.  'deps' has the list of other
                  libraries (also *without* extension) this library
                  needs to be linked with.  'objs' has the list of
                  object files to build this library.

    obj2dso     - function that produces build file lines to build a
                  dynamic shared object file from object files.

                  called like this:

                        obj2dso(lib => "PATH/TO/libfile",
                                objs => [ "PATH/TO/objectfile", ... ],
                                deps => [ "PATH/TO/otherlibfile",
                                ... ]);

                  This is almost the same as obj2shlib, but the
                  intent is to build a shareable library that can be
                  loaded in runtime (a "plugin"...).

    obj2bin     - function that produces build file lines to build an
                  executable file from object files.

                  called like this:

                        obj2bin(bin => "PATH/TO/binfile",
                                objs => [ "PATH/TO/objectfile", ... ],
                                deps => [ "PATH/TO/libfile", ... ]);

                  'bin' has the intended executable file name
                  *without* extension, obj2bin is expected to add
                  that.  'objs' has the list of object files to build
                  this library.  'deps' has the list of library files
                  (also *without* extension) that the programs needs
                  to be linked with.

    in2script   - function that produces build file lines to build a
                  script file from some input.

                  called like this:

                        in2script(script => "PATH/TO/scriptfile",
                                  sources => [ "PATH/TO/infile", ... ]);

                  'script' has the intended script file name.
                  'sources' has the list of source files to build the
                  resulting script from.

Along with the build-file templates is the driving template
[`Configurations/common.tmpl`](common.tmpl), which looks through all the
information in `%unified_info` and generates all the rulesets to build libraries,
programs and all intermediate files, using the rule generating
functions defined in the build-file template.

As an example with the smaller `build.info` set we've seen as an
example, producing the rules to build `libcrypto` would result in the
following calls:

    # Note: obj2shlib will only be called if shared libraries are
    # to be produced.
    # Note 2: obj2shlib must convert the '.o' extension to whatever
    # is suitable on the local platform.
    obj2shlib(shlib => "libcrypto",
              objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ],
              deps => [  ]);

    obj2lib(lib => "libcrypto"
            objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]);

    src2obj(obj => "crypto/aes.o"
            srcs => [ "crypto/aes.c" ],
            deps => [ ],
            incs => [ "include" ],
            intent => "lib");

    src2obj(obj => "crypto/evp.o"
            srcs => [ "crypto/evp.c" ],
            deps => [ ],
            incs => [ "include" ],
            intent => "lib");

    src2obj(obj => "crypto/cversion.o"
            srcs => [ "crypto/cversion.c" ],
            deps => [ "crypto/buildinf.h" ],
            incs => [ "include" ],
            intent => "lib");

    generatesrc(src => "crypto/buildinf.h",
                generator => [ "util/mkbuildinf.pl", "\"$(CC)",
                               "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
                generator_incs => [ "util" ],
                generator_deps => [ "util/Foo.pm" ],
                incs => [ ],
                deps => [ ],
                intent => "lib");

The returned strings from all those calls are then concatenated
together and written to the resulting build-file.