summaryrefslogtreecommitdiffstats
path: root/rfcs/0023-musl-libc.md
blob: d7e68511d8882326b2cfffd4950510e936b24cd7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
---
feature: musl-libc
start-date: 2018-02-19
author: Will Dietz (@dtzWill)
co-authors: Shea Levy (@shlevy)
related-issues: 34645, 6221, ...
---



# Summary
[summary]: #summary

When targeting Linux platforms, Nixpkgs builds software against
the defacto standard of Linux libc implementations:
[glibc](https://www.gnu.org/software/libc/).

This RFC proposes adding **experimental** support in Nixpkgs for the use
of an alternative libc implementation, [musl](https://www.musl-libc.org/),
for the reasons outlined below.
Adding this support is similar to introducing support for an architecture,
and realistically will be limited in compatibility and features
compared to non-musl siblings such as `x86_64-unknown-linux-gnu`.

Initial support is already available in nixpkgs master, capable
of building up through important packages such as nixUnstable itself
and LLVM.  This is not to be taken as an endorsement: discussing
and ultimately deciding whether support for musl should be part of nixpkgs
is the subject of this RFC.  That said, this initial support is
a reasonable foundation for evaluating technical details discussed below,
and a convenient way for interested parties to explore the work so far.

To help ensure we're all on the same page, unless otherwise specified
assume references to musl support implementation are in reference
to this commit (latest master at time of writing):

[bd7d5b365799a145717d31122ebfd24b51fea117](https://github.com/NixOS/nixpkgs/commit/bd7d5b365799a145717d31122ebfd24b51fea117)

# Motivation
[motivation]: #motivation

## Why Musl?
There are many reasons to prefer the use of musl.

musl is advertised as being:
* lightweight
* fast
* simple
* free
* correctness: standards-conforming
* correctness: safety

Additionally it is very popular when statically linking software,
creating binaries capable of executing most anywhere.

In fact it is for this reason that Nixpkgs itself builds
the bootstrap busybox using musl.

A somewhat outdated overview comparing musl against other
implementations is available [here](http://www.etalabs.net/compare_libcs.html).
Note this comparison is maintained by the (primary) author of musl,
(as indicated at the top of the page).

I'm unable to find good numbers but musl is "arguably" the second
most popular libc implementation on Linux, and is used
by a number of important projects you may be familiar with
large userbases, including:
* Alpine Linux - [#70 on Distrowatch](https://distrowatch.com/table.php?distribution=alpine) but very popular amongst docker users for producing slim container images.
* [OpenWRT/LEDE](https://openwrt.org/) - #1 open-source Linux router firmware project; foundation of most other projects targetting routers.
More projects and details of how they use musl can be found here:

https://wiki.musl-libc.org/projects-using-musl.html

## Why Nixpkgs?

The importance of musl is not the primary point of contention in this RFC,
instead perhaps the main question is whether such support belongs in Nixpkgs or not.

The main arguments for inclusion are:
* **convenience of use**
* **foundation for exciting future work**: musl is widely used by high-level languages
  as the libc implementation used to produce statically linked programs:
  these are easy to deploy, launch quickly, and only include the code required.
  (NOTE: currently musl support prefers dynamic linking and shared libraries
   as is the strong preference in Nixpkgs)
* Software sometimes must be patched to compile or run with musl; in @dtzWill's experience,
  these changes are largely fixes improving compliance and correctness resulting in
  higher-quality programs.  Recent versions of glibc have started taking stronger stances
  on enforcing compliance (look at patch fallout folllowing any glibc upgrade in last year or so)
  resulting in overlapping work from both sides.
  (NOTE: use of glibc extensions or reliance on non-standard behavior is still common and unlikely to go away soon)

And to a large extent:
"Why not?" -- similar to including support for architectures such as Aarch64 or RISC-V,
  and just like support for those architectures it's relatively clear that pushing them
  into private forks would be detrimental to the nixpkgs project as well as all users
  interested in using Nixpkgs on those platforms/architectures.

musl is clearly useful for a variety of important use cases,
however including support has a few costs (see Drawbacks, below):
do folks believe the costs are too high?


## Additional Resources

* [musl FAQ](https://www.musl-libc.org/faq.html)
* [projects using musl](https://wiki.musl-libc.org/projects-using-musl.html)
* [Slides from a talk discussing various libcs, 2014](http://events17.linuxfoundation.org/sites/events/files/slides/libc-talk.pdf)

## Related Isssues

* [big musl PR](https://github.com/NixOS/nixpkgs/pull/34645)
* [issues matching "musl", newest first](https://github.com/NixOS/nixpkgs/search?o=desc&q=musl&s=created&type=Issues&utf8=%E2%9C%93)
* [2015 libc discussion](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223)

# Detailed design
[design]: #detailed-design

## Goals
### Laying the Foundation

Implement the following in nixpkgs:

* [x] musl-based bootstrap
* [x] stdenv for native musl building
* [x] cross-musl stdenv

These are already implemented and are currently tested
to build and provide basic functionality as part
of release-cross.nix.

These features would be very difficult to implement
or maintain externally, and near impossible as an overlay.

## Package Compatibility

For a variety of reasons many packages do not work out-of-the-box
in musl-based environments.

### "Normalization"

Vast majority of the problems here are "minor" and are the
sort of problem we regularly encounter and address when
bumping to a new glibc version, new gcc version, or using
a clang-based stdenv (such as on Darwin).

I'm calling these fixes "normalization".
These are changes like "adding a missing include" or
"don't assume compiler is invoked 'gcc'".

These changes usually can be safely applied on all platforms
(although sometimes they are not for rebuild reasons)
and are easy to check for correctness or at least "couldn't-possibly-hurt".

### Big Incompatibilities

Some packages are very much not portable and require significant
and invasive changes to work with environments they don't expect.

In the context of this RFC's proposed musl support,
there are a number of packages that are known to be in this category:

* systemd
* ...

This RFC proposes ignoring those for the immediate future,
to be revisited later, and focuses on systemd.

#### Systemd

Currently many packages depend on systemd.
This dependency is indirect for all but a handful of packages,
with a few key pieces of software integrating with systemd.

As far as I know this dependency is generally optional,
and so we could easily avoid its use when using musl.

This makes it possible to build a great number of packages
(thousands) but more complicated software "ecosystems"
and "desktop environments" will not work without something
to tie them together with the various roles played by systemd.

Addressing this in any way is not in the scope of this RFC.

Similarly, NixOS itself (especially services) require systemd
and we do not propose altering this.

An early version of the "musl PR" patched systemd so that it
would build successfully, using patches from OpenEmbedded.org.

The result was never tested or reviewed in terms of providing
basic functionality or general suitability for Nixpkgs/NixOs.

(OE folks do great work, but they may expect rather different
things from systemd or workaround introduced shortcomings elsewhere
in various capacities)

## Scope

Primarily non-GUI packages for now, due to systemd blocker.

In the future these will be supported.

This RFC is primarily concerned with the groundwork for using musl at all.

## Testing and Maintenance

"Ideally" the answer would be an infinite number of builders would constantly
build all the things on all the platforms.

Unfortunately this is unrealistic due to capacity constraints and other reasons.

### Responsibility

"musl team" is reponsible, initially consisting of

* @dtzWill
* @shlevy
* @domenkozar
* @rasendubi

A team handle will be created to track this
and to ping the team on musl-related discussion or issues.

### Infrastructure

Build at least stdenv with more being added in the future.

Jobs may be given lower priority/shares.

# Drawbacks
[drawbacks]: #drawbacks

Why should we *not* do this?

Potential maintenance burden, particularly regarding collections of patches,
seems to be the primary concern.

## Additional Costs

* Maintenance
* Infrastructure (build pressure, storage, ...)

## Fractured Community

> Another issue: adding musl support fractures the Nixpkgs user/development community: some people will run musl-based packages and some will run glibc-based packages. As a result all of Nixpkgs/NixOS will end up being less tested. it doubles the test matrix on Linux, after all.

## Previous Discussion of drawbacks and concerns

This RFC was prompted by concerns about the drawbacks:
["I'm really not in favour of adding support to Nixpkgs"](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366789321).
This comment echoes very similar concerns expressed [back in 2015](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223).

# Alternatives
[alternatives]: #alternatives

* Maintain in a separate fork
  * [SLNOS project is willing to adopt](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366845015)
* Maintained as an overlay
* No musl libc support.
  * Not really an "alternative" :).

# Unresolved questions
[unresolved]: #unresolved-questions

What parts of the design are still TBD or unknowns?

## Support

We need to work on defining:

* What "Support" entails
* Responsibility
* Blame?

For now we leave it as an informal understanding
which we can improve on in the future.

## Impact

### Infrastructure

* Hydra
* ofborg

### Complexity

* evaluation complexity
  * cost of behind-the-scenes "magic" required
* keeping expressions avoidable
  * cyclomatic complexity

## How to Remove?

Is there a good way to move forward
without becoming impossibly intertwined?
Such that a future party could
* reduce nixpkgs to what it "would be" without musl support
* Do so confidently without worrying about subtle
  breakages?

Maintaining entirely as an overlay (or fork?)
is an obviously effective solution in this regard.
Clear separation and enforced use of carefully crafted
interfaces/abstractions may also help with this.

To some extent the importance of this depends
on how likely the community expects to find itself
"regretting" or wanting to be "rid" of musl support.

However the design and use of good abstractions
is valuable in all cases :).

# Future work
[future]: #future-work

### Fetch, Unpack, Patch

(TODO: Split to new RFC?)
It may be possible to leverage proper use of "phases" so that
we can provide reasonable coverage of the unpack and patch
phases for all "supported" configurations.

As an example, this would make it possible for our x86_64 builders and users to
get feedback ensuring that changes didn't break hashes or patch application
elsewhere without requiring builders of each configuration.

The benefit of this would be in avoiding most of the burden of building everything
while making it easy to catch the most common sort of problems
so they can be addressed ("oops I didn't update the hash for darwin")
or flagged for investigation.

I believe there's a branch or PR trying this somewhere.
Regardless, out of scope for this RFC.