1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
|
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb">
<title>FoundationDB</title>
<para>
<emphasis>Source:</emphasis>
<filename>modules/services/databases/foundationdb.nix</filename>
</para>
<para>
<emphasis>Upstream documentation:</emphasis>
<link xlink:href="https://apple.github.io/foundationdb/">https://apple.github.io/foundationdb/</link>
</para>
<para>
<emphasis>Maintainer:</emphasis> Austin Seipp
</para>
<para>
<emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
</para>
<para>
FoundationDB (or <quote>FDB</quote>) is an open source, distributed,
transactional key-value store.
</para>
<section xml:id="module-services-foundationdb-configuring">
<title>Configuring and basic setup</title>
<para>
To enable FoundationDB, add the following to your
<filename>configuration.nix</filename>:
</para>
<programlisting>
services.foundationdb.enable = true;
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
</programlisting>
<para>
The <option>services.foundationdb.package</option> option is
required, and must always be specified. Due to the fact
FoundationDB network protocols and on-disk storage formats may
change between (major) versions, and upgrades must be explicitly
handled by the user, you must always manually specify this
yourself so that the NixOS module will use the proper version.
Note that minor, bugfix releases are always compatible.
</para>
<para>
After running <command>nixos-rebuild</command>, you can verify
whether FoundationDB is running by executing
<command>fdbcli</command> (which is added to
<option>environment.systemPackages</option>):
</para>
<programlisting>
$ sudo -u foundationdb fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.
The database is available.
Welcome to the fdbcli. For help, type `help'.
fdb> status
Using cluster file `/etc/foundationdb/fdb.cluster'.
Configuration:
Redundancy mode - single
Storage engine - memory
Coordinators - 1
Cluster:
FoundationDB processes - 1
Machines - 1
Memory availability - 5.4 GB per process on machine with least available
Fault Tolerance - 0 machines
Server time - 04/20/18 15:21:14
...
fdb>
</programlisting>
<para>
You can also write programs using the available client libraries.
For example, the following Python program can be run in order to
grab the cluster status, as a quick example. (This example uses
<command>nix-shell</command> shebang support to automatically
supply the necessary Python modules).
</para>
<programlisting>
a@link> cat fdb-status.py
#! /usr/bin/env nix-shell
#! nix-shell -i python -p python pythonPackages.foundationdb52
import fdb
import json
def main():
fdb.api_version(520)
db = fdb.open()
@fdb.transactional
def get_status(tr):
return str(tr['\xff\xff/status/json'])
obj = json.loads(get_status(db))
print('FoundationDB available: %s' % obj['client']['database_status']['available'])
if __name__ == "__main__":
main()
a@link> chmod +x fdb-status.py
a@link> ./fdb-status.py
FoundationDB available: True
a@link>
</programlisting>
<para>
FoundationDB is run under the <command>foundationdb</command> user
and group by default, but this may be changed in the NixOS
configuration. The systemd unit
<command>foundationdb.service</command> controls the
<command>fdbmonitor</command> process.
</para>
<para>
By default, the NixOS module for FoundationDB creates a single
SSD-storage based database for development and basic usage. This
storage engine is designed for SSDs and will perform poorly on
HDDs; however it can handle far more data than the alternative
<quote>memory</quote> engine and is a better default choice for
most deployments. (Note that you can change the storage backend
on-the-fly for a given FoundationDB cluster using
<command>fdbcli</command>.)
</para>
<para>
Furthermore, only 1 server process and 1 backup agent are started
in the default configuration. See below for more on scaling to
increase this.
</para>
<para>
FoundationDB stores all data for all server processes under
<filename>/var/lib/foundationdb</filename>. You can override this
using <option>services.foundationdb.dataDir</option>, e.g.
</para>
<programlisting>
services.foundationdb.dataDir = "/data/fdb";
</programlisting>
<para>
Similarly, logs are stored under
<filename>/var/log/foundationdb</filename> by default, and there
is a corresponding <option>services.foundationdb.logDir</option>
as well.
</para>
</section>
<section xml:id="module-services-foundationdb-scaling">
<title>Scaling processes and backup agents</title>
<para>
Scaling the number of server processes is quite easy; simply
specify <option>services.foundationdb.serverProcesses</option> to
be the number of FoundationDB worker processes that should be
started on the machine.
</para>
<para>
FoundationDB worker processes typically require 4GB of RAM
per-process at minimum for good performance, so this option is set
to 1 by default since the maximum amount of RAM is unknown. You’re
advised to abide by this restriction, so pick a number of
processes so that each has 4GB or more.
</para>
<para>
A similar option exists in order to scale backup agent processes,
<option>services.foundationdb.backupProcesses</option>. Backup
agents are not as performance/RAM sensitive, so feel free to
experiment with the number of available backup processes.
</para>
</section>
<section xml:id="module-services-foundationdb-clustering">
<title>Clustering</title>
<para>
FoundationDB on NixOS works similarly to other Linux systems, so
this section will be brief. Please refer to the full FoundationDB
documentation for more on clustering.
</para>
<para>
FoundationDB organizes clusters using a set of
<emphasis>coordinators</emphasis>, which are just
specially-designated worker processes. By default, every
installation of FoundationDB on NixOS will start as its own
individual cluster, with a single coordinator: the first worker
process on <command>localhost</command>.
</para>
<para>
Coordinators are specified globally using the
<command>/etc/foundationdb/fdb.cluster</command> file, which all
servers and client applications will use to find and join
coordinators. Note that this file <emphasis>can not</emphasis> be
managed by NixOS so easily: FoundationDB is designed so that it
will rewrite the file at runtime for all clients and nodes when
cluster coordinators change, with clients transparently handling
this without intervention. It is fundamentally a mutable file, and
you should not try to manage it in any way in NixOS.
</para>
<para>
When dealing with a cluster, there are two main things you want to
do:
</para>
<itemizedlist spacing="compact">
<listitem>
<para>
Add a node to the cluster for storage/compute.
</para>
</listitem>
<listitem>
<para>
Promote an ordinary worker to a coordinator.
</para>
</listitem>
</itemizedlist>
<para>
A node must already be a member of the cluster in order to
properly be promoted to a coordinator, so you must always add it
first if you wish to promote it.
</para>
<para>
To add a machine to a FoundationDB cluster:
</para>
<itemizedlist spacing="compact">
<listitem>
<para>
Choose one of the servers to start as the initial coordinator.
</para>
</listitem>
<listitem>
<para>
Copy the <command>/etc/foundationdb/fdb.cluster</command> file
from this server to all the other servers. Restart
FoundationDB on all of these other servers, so they join the
cluster.
</para>
</listitem>
<listitem>
<para>
All of these servers are now connected and working together in
the cluster, under the chosen coordinator.
</para>
</listitem>
</itemizedlist>
<para>
At this point, you can add as many nodes as you want by just
repeating the above steps. By default there will still be a single
coordinator: you can use <command>fdbcli</command> to change this
and add new coordinators.
</para>
<para>
As a convenience, FoundationDB can automatically assign
coordinators based on the redundancy mode you wish to achieve for
the cluster. Once all the nodes have been joined, simply set the
replication policy, and then issue the
<command>coordinators auto</command> command
</para>
<para>
For example, assuming we have 3 nodes available, we can enable
double redundancy mode, then auto-select coordinators. For double
redundancy, 3 coordinators is ideal: therefore FoundationDB will
make <emphasis>every</emphasis> node a coordinator automatically:
</para>
<programlisting>
fdbcli> configure double ssd
fdbcli> coordinators auto
</programlisting>
<para>
This will transparently update all the servers within seconds, and
appropriately rewrite the <command>fdb.cluster</com
|