From 7a8432123bd05894b28e286308fbcb8e51fa0d7f Mon Sep 17 00:00:00 2001 From: Matthias Beyer Date: Fri, 8 Jul 2016 18:21:05 +0200 Subject: doc: Rewrite "The Store" chapter --- doc/src/02000-store.md | 200 +++++++++++++++++++++---------------------------- 1 file changed, 85 insertions(+), 115 deletions(-) (limited to 'doc') diff --git a/doc/src/02000-store.md b/doc/src/02000-store.md index 098a5411..04c96e09 100644 --- a/doc/src/02000-store.md +++ b/doc/src/02000-store.md @@ -1,64 +1,67 @@ # The Store {#sec:thestore} -## File Format {#sec:thestore:fileformat} - -The content in the store MUST BE encoded in either Unicode UTF-8 or ASCII. -Each "Entry" (File) MUST HAVE a "Header" component as well as a "Content" -component. -Each "Entry" in the store MUST start with three single dashes ("-") followed -by a newline character, named "initial marker" in the following chapter. -The Header follows the initial marker (@sec:thestore:fileformat:header). -The Header MUST BE followed by a line which contains three single dashes ("-") -and a newline character, called "header-close marker" in the following -chapter. -The content follows the header-close marker (@sec:thestore:fileformat:content). - -### Header Format {#sec:thestore:fileformat:header} - -The header format MUST BE "TOML". -The sections which MAY or MUST be in the header are defined in the following -chapters. +The store is where all the good things happen. +The store is basically just a directory on the filesystem imag manages and keeps +its state in. + +One could say that the store is simply a databases, and it really is. We opted +to go for plain text, though, as we believe that plain text is the only sane way +to do such a thing. +A user should always be able to read her data without great effort and putting +everything in a _real_ database like sqlite or even postgresql would need a user +to install additional software just to read his own data. We don't want that. +Text is readable until the worlds end and we think it is therefor better to +store the data in plain text. + +The following sections describe the store and the file format we use to store +data. One may skip the following sections, they are included for users who want +to dig into the store with their editors. -#### Header section: "imag" {#sec:thestore:fileformat:header:imag} +## File Format {#sec:thestore:fileformat} -The header MUST contain a section called "imag", where the automatically by the -program generated data is stored in. -The contents of this section is edited via commandline calls or by the -program implicitely and SHOULD NOT be edited by the user. +The contents of the store are encoded in either UTF-8 or ASCII. Either way, a +normal text editor (like `vim` or the other one) will always be sufficient to +dog into the store and modify files. For simple viewing even a pager (like +`less`) is sufficient. -This "imag" section MUST contain the following keys +Each entry in the store consists of two parts: -1. A "version" Key. The version stored here is the version of the Store, the - Entry was created with. +1. Header +1. Content -The "imag" section MAY contain +The following section describe their purpose. -1. A section "imag.links" where a module is allowed to store URIs in a flat - list -1. A section "imag.content", used for referring to external content. - The key "uri" is the only one which is required, it refers to external - content. - An explicitely suggested key is "file" for referring to a _local Mirror_ of - the content. +### Header Format {#sec:thestore:fileformat:header} -#### Header section: "custom" {#sec:thestore:fileformat:header:custom} +The header format is where imag stores its data. The header is an area at the +top of every file which is seperated from the content part by three dashes +(`---`). Between these three dashes there is structured data. imag uses `TOML` +as data format for this structured data, because it fits best and the available +`TOML` parser for the rust programming language is really good. -The header MAY contain a section named "custom". -The user is free to store arbitrary data here. -The user is also free to edit this section by either commandline or editor. +The header can contain any amount of data, but modules (see @sec:modules) are +restricted in their way of altering the data. -#### Module Header section {#sec:thestore:fileformat:header:module} +So normally there are several sections in the header. One section (`[imag]`) is +always present. It contains a `version` field, which tells imag which version +this file was created with (the version information is _also_ encoded in the +filename, just in case things change in the future). It also contains a `links` +field which is an Array of values. This `links` field is for linking (see +@sec:thestore:linking) to other entries in the store. -The header MAY contain a section named after a module. -The corrosponding module is allowed to store arbitrary data in this section. +Other sections are named like the modules which created them. Every module is +allowed to store arbitrary data under its own section and a module may never +read other sections than its own. This is not enforced by imag itself, though. ### Content Format {#sec:thestore:fileformat:content} -The content is the part of the file where the user is free to enter any -textual content. -The content MAY BE rendered as Markdown or other markup format for the users -convenience. -The Store library MUST NOT expect any particular markup format. +The content is the part of the file where the user is free to enter any textual +content. The content may be rendered as Markdown or other markup format for the +users convenience. The store does never expect and specific markup and actually +the markup implementation is not inside the very code of imag. + +Technically it would be possible that the content part of a file is used to +store binary data. We don't want this, though. ### Example {#sec:thestore:fileformat:example} @@ -69,17 +72,10 @@ An example for a file in the store follows. [imag] version = "0.1.0" -[imag.content] -url = "file://home/user/kittens.mpeg" - -imag.links = [ - "imag://home/user/more_kittens.mpeg" -] +imag.links = ["/home/user/more_kittens.mpeg"] [examplemodule] -arbitrary = data -[custom] -truth = 42 +arbitrary = "data" --- This is an example text, written by the user. @@ -89,23 +85,25 @@ This is an example text, written by the user. ## File organization {#sec:thestore:fileorganization} The "Entries" are stored as files in the "Store", which is a directory the -user has access to. -The store MAY exist in the users Home-directory or any other directory the -user has Read-Write-Access to. - -The Path of each File is shown as absolute path in this paper, while the root -is always the store directory. -This Path is named "Storepath". -So if the store exists in `/home/user/store/`, a file with the Storepath +user has access to. The store may exist in the users Home-directory or any +other directory the user has read-write-Access to. + +Each module stores its data in an own subdirectory in the store. This is because +we like to keep things ordered and clean, not because it is technically +necessary. + +We name the path to a file in the store "Store id" or "Storepath" and we often +refer to it by using the store location as root. +So if the store exists in `/home/user/store/`, a file with the storepath `/example.file` is (on the filesystem) located at `/home/user/store/example.file`. -A Storepath contains predefined parts: +A storepath contains predefined parts: -* The module name of the Module the Entry belongs to. - This part is a directory. -* The version (semantic versioning applies) of the module storing the Entry - This part is a postfix to the filename +* The module name of the Module the Entry belongs to, as said above. + This part is always a directory. +* The version (semantic versioning applies) of the module storing the entry. + This part is a postfix to the filename. The pattern for the storepath is @@ -113,66 +111,38 @@ The pattern for the storepath is ///~ ``` -So if a Module named "ExampleModule" with version "0.1" stores a file in the -Store, the Storepath for a file with the name "example" is -"/ExampleModule/example~0.1". +So if a module named "example-module" with version "0.1.0" stores a file in the +Store, the storepath for a file with the name "example" is +"/example-module/example~0.1.0". -Any number of subdirectories MAY BE used, so creating folder hierarchies is -possible and valid. -A file "example" for a module "module" in version "0.1" would be stored in -sub-folders like this: +Any number of subdirectories may be used, so creating folder hierarchies is +possible and valid. A file "example" for a module "module" in version "0.1.0" +would be stored in sub-folders like this: ``` -/module/some/sub/folder/example~0.1 +/module/some/sub/folder/example~0.1.0 ``` -A given file MUST only exist once with the same path. -For example, it is invalid if these files exist at the same time: +For example, it is valid if these files exist at the same time: * /foo/bar~0.2 * /foo/bar~1.3 -To future-proof the System it is thus necessary to create a disambiguation at -the store level. Thus if a library wants to retrieve a file from the Store -it MUST at least accept files from it's current advertised version. It MAY -accept older files and it MAY transform them and resubmit them in the newer -version. +It might not be sane, though. -For this there will be an enum returned for each given Entry. It will have these -members: - -- `Compatible`, for version matches -- `PossiblyIncompatible`, if the current version is at least a major number - further -- `Incompatible`, if the file is a at least a major number further +To future-proof the system it is necessary to provide a way for modules to +differentiate in their versions on the store level. Thus if a module wants to +retrieve a file from the store it must at least accept files from it's current +advertised version. It may accept older files and it may transform them and +resubmit them in the newer version. ## Store path links {#sec:thestore:links} -Linking entries MUST BE version independent. +Linking entries is version independent. This means if an entry "a" from a module "A" gets written to the store, it may -link to an entry "b" from a module "B", which is in version "0.1" at the moment. -If the module "B" gets updated, it might update its entries in the store as -well. -The link from the "a" MUST NOT get invalid in this case. - -This is accomplished by linking without the version number: So a link for the -entry - -``` -/module/some/sub/folder/example~0.1 -``` - -is - -``` -imag://module/some/sub/folder/example -``` - -As shown in the example, a link to imag-internal entries, the link is prefixed -with a "imag://" identifier. -This prefix is optional. -A link to external content MUST NEVER be prefixed this way. -The path of the internal link MUST NEVER be relative, but always absolute from -the root directory of the store. +link to an entry "b" from a module "B", which is in version "0.1.0" at the +moment. If the module "B" gets updated, it might update its entries in the store +as well. The link from the "a" should never get invalid in this case, though it +is not ensured by the core of imag itself. -- cgit v1.2.3