diff options
| author | pennae <github@quasiparticle.net> | 2023-01-25 00:33:40 +0100 |
|---|---|---|
| committer | pennae <github@quasiparticle.net> | 2023-01-27 20:07:34 +0100 |
| commit | 0a6e6cf7e698a6a08a62d8863e2c66b36d5db0d9 (patch) | |
| tree | 6e28cb9ba90a3d03d6efab999b958527af45f84c /nixos/modules/services/databases | |
| parent | 8b8670db100efed03a979f7ec24c353f72c0bdbd (diff) | |
nixos/manual: render module chapters with nixos-render-docs
this converts meta.doc into an md pointer, not an xml pointer. since we
no longer need xml for manual chapters we can also remove support for
manual chapters from md-to-db.sh
since pandoc converts smart quotes to docbook quote elements and our
nixos-render-docs does not we lose this distinction in the rendered
output. that's probably not that bad, our stylesheet didn't make use of
this anyway (and pre-23.05 versions of the chapters didn't use quote
elements either).
also updates the nixpkgs manual to clarify that option docs support all
extensions (although it doesn't support headings at all, so heading
anchors don't work by extension).
Diffstat (limited to 'nixos/modules/services/databases')
| -rw-r--r-- | nixos/modules/services/databases/foundationdb.nix | 2 | ||||
| -rw-r--r-- | nixos/modules/services/databases/foundationdb.xml | 425 | ||||
| -rw-r--r-- | nixos/modules/services/databases/postgresql.nix | 2 | ||||
| -rw-r--r-- | nixos/modules/services/databases/postgresql.xml | 250 |
4 files changed, 2 insertions, 677 deletions
diff --git a/nixos/modules/services/databases/foundationdb.nix b/nixos/modules/services/databases/foundationdb.nix index 16d539b661eb..48e9898a68c2 100644 --- a/nixos/modules/services/databases/foundationdb.nix +++ b/nixos/modules/services/databases/foundationdb.nix @@ -424,6 +424,6 @@ in }; }; - meta.doc = ./foundationdb.xml; + meta.doc = ./foundationdb.md; meta.maintainers = with lib.maintainers; [ thoughtpolice ]; } diff --git a/nixos/modules/services/databases/foundationdb.xml b/nixos/modules/services/databases/foundationdb.xml deleted file mode 100644 index 611535a9eb8a..000000000000 --- a/nixos/modules/services/databases/foundationdb.xml +++ /dev/null @@ -1,425 +0,0 @@ -<!-- Do not edit this file directly, edit its companion .md instead - and regenerate this file using nixos/doc/manual/md-to-db.sh --> -<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb"> - <title>FoundationDB</title> - <para> - <emphasis>Source:</emphasis> - <filename>modules/services/databases/foundationdb.nix</filename> - </para> - <para> - <emphasis>Upstream documentation:</emphasis> - <link xlink:href="https://apple.github.io/foundationdb/">https://apple.github.io/foundationdb/</link> - </para> - <para> - <emphasis>Maintainer:</emphasis> Austin Seipp - </para> - <para> - <emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x - </para> - <para> - FoundationDB (or <quote>FDB</quote>) is an open source, distributed, - transactional key-value store. - </para> - <section xml:id="module-services-foundationdb-configuring"> - <title>Configuring and basic setup</title> - <para> - To enable FoundationDB, add the following to your - <filename>configuration.nix</filename>: - </para> - <programlisting> -services.foundationdb.enable = true; -services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x -</programlisting> - <para> - The <option>services.foundationdb.package</option> option is - required, and must always be specified. Due to the fact - FoundationDB network protocols and on-disk storage formats may - change between (major) versions, and upgrades must be explicitly - handled by the user, you must always manually specify this - yourself so that the NixOS module will use the proper version. - Note that minor, bugfix releases are always compatible. - </para> - <para> - After running <command>nixos-rebuild</command>, you can verify - whether FoundationDB is running by executing - <command>fdbcli</command> (which is added to - <option>environment.systemPackages</option>): - </para> - <programlisting> -$ sudo -u foundationdb fdbcli -Using cluster file `/etc/foundationdb/fdb.cluster'. - -The database is available. - -Welcome to the fdbcli. For help, type `help'. -fdb> status - -Using cluster file `/etc/foundationdb/fdb.cluster'. - -Configuration: - Redundancy mode - single - Storage engine - memory - Coordinators - 1 - -Cluster: - FoundationDB processes - 1 - Machines - 1 - Memory availability - 5.4 GB per process on machine with least available - Fault Tolerance - 0 machines - Server time - 04/20/18 15:21:14 - -... - -fdb> -</programlisting> - <para> - You can also write programs using the available client libraries. - For example, the following Python program can be run in order to - grab the cluster status, as a quick example. (This example uses - <command>nix-shell</command> shebang support to automatically - supply the necessary Python modules). - </para> - <programlisting> -a@link> cat fdb-status.py -#! /usr/bin/env nix-shell -#! nix-shell -i python -p python pythonPackages.foundationdb52 - -import fdb -import json - -def main(): - fdb.api_version(520) - db = fdb.open() - - @fdb.transactional - def get_status(tr): - return str(tr['\xff\xff/status/json']) - - obj = json.loads(get_status(db)) - print('FoundationDB available: %s' % obj['client']['database_status']['available']) - -if __name__ == "__main__": - main() -a@link> chmod +x fdb-status.py -a@link> ./fdb-status.py -FoundationDB available: True -a@link> -</programlisting> - <para> - FoundationDB is run under the <command>foundationdb</command> user - and group by default, but this may be changed in the NixOS - configuration. The systemd unit - <command>foundationdb.service</command> controls the - <command>fdbmonitor</command> process. - </para> - <para> - By default, the NixOS module for FoundationDB creates a single - SSD-storage based database for development and basic usage. This - storage engine is designed for SSDs and will perform poorly on - HDDs; however it can handle far more data than the alternative - <quote>memory</quote> engine and is a better default choice for - most deployments. (Note that you can change the storage backend - on-the-fly for a given FoundationDB cluster using - <command>fdbcli</command>.) - </para> - <para> - Furthermore, only 1 server process and 1 backup agent are started - in the default configuration. See below for more on scaling to - increase this. - </para> - <para> - FoundationDB stores all data for all server processes under - <filename>/var/lib/foundationdb</filename>. You can override this - using <option>services.foundationdb.dataDir</option>, e.g. - </para> - <programlisting> -services.foundationdb.dataDir = "/data/fdb"; -</programlisting> - <para> - Similarly, logs are stored under - <filename>/var/log/foundationdb</filename> by default, and there - is a corresponding <option>services.foundationdb.logDir</option> - as well. - </para> - </section> - <section xml:id="module-services-foundationdb-scaling"> - <title>Scaling processes and backup agents</title> - <para> - Scaling the number of server processes is quite easy; simply - specify <option>services.foundationdb.serverProcesses</option> to - be the number of FoundationDB worker processes that should be - started on the machine. - </para> - <para> - FoundationDB worker processes typically require 4GB of RAM - per-process at minimum for good performance, so this option is set - to 1 by default since the maximum amount of RAM is unknown. You’re - advised to abide by this restriction, so pick a number of - processes so that each has 4GB or more. - </para> - <para> - A similar option exists in order to scale backup agent processes, - <option>services.foundationdb.backupProcesses</option>. Backup - agents are not as performance/RAM sensitive, so feel free to - experiment with the number of available backup processes. - </para> - </section> - <section xml:id="module-services-foundationdb-clustering"> - <title>Clustering</title> - <para> - FoundationDB on NixOS works similarly to other Linux systems, so - this section will be brief. Please refer to the full FoundationDB - documentation for more on clustering. - </para> - <para> - FoundationDB organizes clusters using a set of - <emphasis>coordinators</emphasis>, which are just - specially-designated worker processes. By default, every - installation of FoundationDB on NixOS will start as its own - individual cluster, with a single coordinator: the first worker - process on <command>localhost</command>. - </para> - <para> - Coordinators are specified globally using the - <command>/etc/foundationdb/fdb.cluster</command> file, which all - servers and client applications will use to find and join - coordinators. Note that this file <emphasis>can not</emphasis> be - managed by NixOS so easily: FoundationDB is designed so that it - will rewrite the file at runtime for all clients and nodes when - cluster coordinators change, with clients transparently handling - this without intervention. It is fundamentally a mutable file, and - you should not try to manage it in any way in NixOS. - </para> - <para> - When dealing with a cluster, there are two main things you want to - do: - </para> - <itemizedlist spacing="compact"> - <listitem> - <para> - Add a node to the cluster for storage/compute. - </para> - </listitem> - <listitem> - <para> - Promote an ordinary worker to a coordinator. - </para> - </listitem> - </itemizedlist> - <para> - A node must already be a member of the cluster in order to - properly be promoted to a coordinator, so you must always add it - first if you wish to promote it. - </para> - <para> - To add a machine to a FoundationDB cluster: - </para> - <itemizedlist spacing="compact"> - <listitem> - <para> - Choose one of the servers to start as the initial coordinator. - </para> - </listitem> - <listitem> - <para> - Copy the <command>/etc/foundationdb/fdb.cluster</command> file - from this server to all the other servers. Restart - FoundationDB on all of these other servers, so they join the - cluster. - </para> - </listitem> - <listitem> - <para> - All of these servers are now connected and working together in - the cluster, under the chosen coordinator. - </para> - </listitem> - </itemizedlist> - <para> - At this point, you can add as many nodes as you want by just - repeating the above steps. By default there will still be a single - coordinator: you can use <command>fdbcli</command> to change this - and add new coordinators. - </para> - <para> - As a convenience, FoundationDB can automatically assign - coordinators based on the redundancy mode you wish to achieve for - the cluster. Once all the nodes have been joined, simply set the - replication policy, and then issue the - <command>coordinators auto</command> command - </para> - <para> - For example, assuming we have 3 nodes available, we can enable - double redundancy mode, then auto-select coordinators. For double - redundancy, 3 coordinators is ideal: therefore FoundationDB will - make <emphasis>every</emphasis> node a coordinator automatically: - </para> - <programlisting> -fdbcli> configure double ssd -fdbcli> coordinators auto -</programlisting> - <para> - This will transparently update all the servers within seconds, and - appropriately rewrite the <command>fdb.cluster</command> file, as - well as informing all client processes to do the same. - </para> - </section> - <section xml:id="module-services-foundationdb-connectivity"> - <title>Client connectivity</title> - <para> - By default, all clients must use the current - <command>fdb.cluster</command> file to access a given FoundationDB - cluster. This file is located by default in - <command>/etc/foundationdb/fdb.cluster</command> on all machines - with the FoundationDB service enabled, so you may copy the active - one from your cluster to a new node in order to connect, if it is - not part of the cluster. - </para> - </section> - <section xml:id="module-services-foundationdb-authorization"> - <title>Client authorization and TLS</title> - <para> - By default, any user who can connect to a FoundationDB process - with the correct cluster configuration can access anything. - FoundationDB uses a pluggable design to transport security, and - out of the box it supports a LibreSSL-based plugin for TLS - support. This plugin not only does in-flight encryption, but also - performs client authorization based on the given endpoint’s - certificate chain. For example, a FoundationDB server may be - configured to only accept client connections over TLS, where the - client TLS certificate is from organization <emphasis>Acme - Co</emphasis> in the <emphasis>Research and Development</emphasis> - unit. - </para> - <para> - Configuring TLS with FoundationDB is done using the - <option>services.foundationdb.tls</option> options in order to - control the peer verification string, as well as the certificate - and its private key. - </para> - <para> - Note that the certificate and its private key must be accessible - to the FoundationDB user account that the server runs under. These - files are also NOT managed by NixOS, as putting them into the - store may reveal private information. - </para> - <para> - After you have a key and certificate file in place, it is not - enough to simply set the NixOS module options – you must also - configure the <command>fdb.cluster</command> file to specify that - a given set of coordinators use TLS. This is as simple as adding - the suffix <command>:tls</command> to your cluster coordinator - configuration, after the port number. For example, assuming you - have a coordinator on localhost with the default configuration, - simply specifying: - </para> - <programlisting> -XXXXXX:XXXXXX@127.0.0.1:4500:tls -</programlisting> - <para> - will configure all clients and server processes to use TLS from - now on. - </para> - </section> - <section xml:id="module-services-foundationdb-disaster-recovery"> - <title>Backups and Disaster Recovery</title> - <para> - The usual rules for doing FoundationDB backups apply on NixOS as - written in the FoundationDB manual. However, one important - difference is the security profile for NixOS: by default, the - <command>foundationdb</command> systemd unit uses <emphasis>Linux - namespaces</emphasis> to restrict write access to the system, - except for the log directory, data directory, and the - <command>/etc/foundationdb/</command> directory. This is enforced - by default and cannot be disabled. - </para> - <para> - However, a side effect of this is that the - <command>fdbbackup</command> command doesn’t work properly for - local filesystem backups: FoundationDB uses a server process - alongside the database processes to perform backups and copy the - backups to the filesystem. As a result, this process is put under - the restricted namespaces above: the backup process can only write - to a limited number of paths. - </para> - <para> - In order to allow flexible backup locations on local disks, the - FoundationDB NixOS module supports a - <option>services.foundationdb.extraReadWritePaths</option> option. - This option takes a list of paths, and adds them to the systemd - unit, allowing the processes inside the service to write (and - read) the specified directories. - </para> - <para> - For example, to create backups in - <command>/opt/fdb-backups</command>, first set up the paths in the - module options: - </para> - <programlisting> -services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ]; -</programlisting> - <para> - Restart the FoundationDB service, and it will now be able to write - to this directory (even if it does not yet exist.) Note: this path - <emphasis>must</emphasis> exist before restarting the unit. - Otherwise, systemd will not include it in the private FoundationDB - namespace (and it will not add it dynamically at runtime). - </para> - <para> - You can now perform a backup: - </para> - <programlisting> -$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups -$ sudo -u foundationdb fdbbackup status -t default -</programlisting> - </section> - <section xml:id="module-services-foundationdb-limitations"> - <title>Known limitations</title> - <para> - The FoundationDB setup for NixOS should currently be considered - beta. FoundationDB is not new software, but the NixOS compilation - and integration has only undergone fairly basic testing of all the - available functionality. - </para> - <itemizedlist spacing="compact"> - <listitem> - <para> - There is no way to specify individual parameters for - individual <command>fdbserver</command> processes. Currently, - all server processes inherit all the global - <command>fdbmonitor</command> settings. - </para> - </listitem> - <listitem> - <para> - Ruby bindings are not currently installed. - </para> - </listitem> - <listitem> - <para> - Go bindings are not currently installed. - </para> - </listitem> - </itemizedlist> - </section> - <section xml:id="module-services-foundationdb-options"> - <title>Options</title> - <para> - NixOS’s FoundationDB module allows you to configure all of the - most relevant configuration options for - <command>fdbmonitor</command>, matching it quite closely. A - complete list of options for the FoundationDB module may be found - <link linkend="opt-services.foundationdb.enable">here</link>. You - should also read the FoundationDB documentation as well. - </para> - </section> - <section xml:id="module-services-foundationdb-full-docs"> - <title>Full documentation</title> - <para> - FoundationDB is a complex piece of software, and requires careful - administration to properly use. Full documentation for - administration can be found here: - <link xlink:href="https://apple.github.io/foundationdb/">https://apple.github.io/foundationdb/</link>. - </para> - </section> -</chapter> diff --git a/nixos/modules/services/databases/postgresql.nix b/nixos/modules/services/databases/postgresql.nix index 6665e7a088fc..7bbe1ad22595 100644 --- a/nixos/modules/services/databases/postgresql.nix +++ b/nixos/modules/services/databases/postgresql.nix @@ -585,6 +585,6 @@ in }; - meta.doc = ./postgresql.xml; + meta.doc = ./postgresql.md; meta.maintainers = with lib.maintainers; [ thoughtpolice danbst ]; } diff --git a/nixos/modules/services/databases/postgresql.xml b/nixos/modules/services/databases/postgresql.xml deleted file mode 100644 index 2f62d5d80b19..000000000000 --- a/nixos/modules/services/databases/postgresql.xml +++ /dev/null @@ -1,250 +0,0 @@ -<!-- Do not edit this file directly, edit its companion .md instead - and regenerate this file using nixos/doc/manual/md-to-db.sh --> -<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-postgresql"> - <title>PostgreSQL</title> - <para> - <emphasis>Source:</emphasis> - <filename>modules/services/databases/postgresql.nix</filename> - </para> - <para> - <emphasis>Upstream documentation:</emphasis> - <link xlink:href="http://www.postgresql.org/docs/">http://www.postgresql.org/docs/</link> - </para> - <para> - PostgreSQL is an advanced, free relational database. - </para> - <section xml:id="module-services-postgres-configuring"> - <title>Configuring</title> - <para> - To enable PostgreSQL, add the following to your - <filename>configuration.nix</filename>: - </para> - <programlisting> -services.postgresql.enable = true; -services.postgresql.package = pkgs.postgresql_11; -</programlisting> - <para> - Note that you are required to specify the desired version of - PostgreSQL (e.g. <literal>pkgs.postgresql_11</literal>). Since - upgrading your PostgreSQL version requires a database dump and - reload (see below), NixOS cannot provide a default value for - <xref linkend="opt-services.postgresql.package" /> such as the - most recent release of PostgreSQL. - </para> - <para> - By default, PostgreSQL stores its databases in - <filename>/var/lib/postgresql/$psqlSchema</filename>. You can - override this using - <xref linkend="opt-services.postgresql.dataDir" />, e.g. - </para> - <programlisting> -services.postgresql.dataDir = "/data/postgresql"; -</programlisting> - </section> - <section xml:id="module-services-postgres-upgrading"> - <title>Upgrading</title> - <note> - <para> - The steps below demonstrate how to upgrade from an older version - to <literal>pkgs.postgresql_13</literal>. These instructions are - also applicable to other versions. - </para> - </note> - <para> - Major PostgreSQL upgrades require a downtime and a few imperative - steps to be called. This is the case because each major version - has some internal changes in the databases’ state during major - releases. Because of that, NixOS places the state into - <filename>/var/lib/postgresql/<version></filename> where - each <literal>version</literal> can be obtained like this: - </para> - <programlisting> -$ nix-instantiate --eval -A postgresql_13.psqlSchema -"13" -</programlisting> - <para> - For an upgrade, a script like this can be used to simplify the - process: - </para> - <programlisting> -{ config, pkgs, ... }: -{ - environment.systemPackages = [ - (let - # XXX specify the postgresql package you'd like to upgrade to. - # Do not forget to list the extensions you need. - newPostgres = pkgs.postgresql_13.withPackages (pp: [ - # pp.plv8 - ]); - in pkgs.writeScriptBin "upgrade-pg-cluster" '' - set -eux - # XXX it's perhaps advisable to stop all services that depend on postgresql - systemctl stop postgresql - - export NEWDATA="/var/lib/postgresql/${newPostgres.psqlSchema}" - - export NEWBIN="${newPostgres}/bin" - - export OLDDATA="${config.services.postgresql.dataDir}" - export OLDBIN="${config.services.postgresql.package}/bin" - - install -d -m 0700 -o postgres -g postgres "$NEWDATA" - cd "$NEWDATA" - sudo -u postgres $NEWBIN/initdb -D "$NEWDATA" - - sudo -u postgres $NEWBIN/pg_upgrade \ - --old-datadir "$OLDDATA" --new-datadir "$NEWDATA" \ - --old-bindir $OLDBIN --new-bindir $NEWBIN \ - "$@" - '') - ]; -} -</programlisting> - <para> - The upgrade process is: - </para> - <orderedlist numeration="arabic"> - <listitem> - <para> - Rebuild nixos configuration with the configuration above added - to your <filename>configuration.nix</filename>. Alternatively, - add that into separate file and reference it in - <literal>imports</literal> list. - </para> - </listitem> - <listitem> - <para> - Login as root (<literal>sudo su -</literal>) - </para> - </listitem> - <listitem> - <para> - Run <literal>upgrade-pg-cluster</literal>. It will stop old - postgresql, initialize a new one and migrate the old one to - the new one. You may supply arguments like - <literal>--jobs 4</literal> and <literal>--link</literal> to - speedup migration process. See - <link xlink:href="https://www.postgresql.org/docs/current/pgupgrade.html">https://www.postgresql.org/docs/current/pgupgrade.html</link> - for details. - </para> - </listitem> - <listitem> - <para> - Change postgresql package in NixOS configuration to the one - you were upgrading to via - <xref linkend="opt-services.postgresql.package" />. Rebuild - NixOS. This should start new postgres using upgraded data - directory and all services you stopped during the upgrade. - </para> - </listitem> - <listitem> - <para> - After the upgrade it’s advisable to analyze the new cluster. - </para> - <itemizedlist> - <listitem> - <para> - For PostgreSQL ≥ 14, use the <literal>vacuumdb</literal> - command printed by the upgrades script. - </para> - </listitem> - <listitem> - <para> - For PostgreSQL < 14, run (as - <literal>su -l postgres</literal> in the - <xref linkend="opt-services.postgresql.dataDir" />, in - this example <filename>/var/lib/postgresql/13</filename>): - </para> - <programlisting> -$ ./analyze_new_cluster.sh -</programlisting> - </listitem> - </itemizedlist> - <warning> - <para> - The next step removes the old state-directory! - </para> - </warning> - <programlisting> -$ ./delete_old_cluster.sh -</programlisting> - </listitem> - </orderedlist> - </section> - <section xml:id="module-services-postgres-options"> - <title>Options</title> - <para> - A complete list of options for the PostgreSQL module may be found - <link linkend="opt-services.postgresql.enable">here</link>. - </para> - </section> - <section xml:id="module-services-postgres-plugins"> - <title>Plugins</title> - <para> - Plugins collection for each PostgreSQL version can be accessed - with <literal>.pkgs</literal>. For example, for - <literal>pkgs.postgresql_11</literal> package, its plugin - collection is accessed by - <literal>pkgs.postgresql_11.pkgs</literal>: - </para> - <programlisting> -$ nix repl '<nixpkgs>' - -Loading '<nixpkgs>'... -Added 10574 variables. - -nix-repl> postgresql_11.pkgs.<TAB><TAB> -postgresql_11.pkgs.cstore_fdw postgresql_11.pkgs.pg_repack -postgresql_11.pkgs.pg_auto_failover postgresql_11.pkgs.pg_safeupdate -postgresql_11.pkgs.pg_bigm postgresql_11.pkgs.pg_similarity -postgresql_11.pkgs.pg_cron postgresql_11.pkgs.pg_topn -postgresql_11.pkgs.pg_hll postgresql_11.pkgs.pgjwt -postgresql_11.pkgs.pg_partman postgresql_11.pkgs.pgroonga -... -</programlisting> - <para> - To add plugins via NixOS configuration, set - <literal>services.postgresql.extraPlugins</literal>: - </para> - <programlisting> -services.postgresql.package = pkgs.postgresql_11; -services.postgresql.extraPlugins = with pkgs.postgresql_11.pkgs; [ - pg_repack - postgis -]; -</programlisting> - <para> - You can build custom PostgreSQL-with-plugins (to be used outside - of NixOS) using function <literal>.withPackages</literal>. For - example, creating a custom PostgreSQL package in an overlay can - look like: - </para> - <programlisting> -self: super: { - postgresql_custom = self.postgresql_11.withPackages (ps: [ - ps.pg_repack - ps.postgis - ]); -} -</programlisting> - <para> - Here’s a recipe on how to override a particular plugin through an - overlay: - </para> - <programlisting> -self: super: { - postgresql_11 = super.postgresql_11.override { this = self.postgresql_11; } // { - pkgs = super.postgresql_11.pkgs // { - pg_repack = super.postgresql_11.pkgs.pg_repack.overrideAttrs (_: { - name = "pg_repack-v20181024"; - src = self.fetchzip { - url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz"; - sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5"; - }; - }); - }; - }; -} -</programlisting> - </section> -</chapter> |