nixpkgs-docs: when to prefer passthru.tests over installCheckPhase

And mention you can have either lightweight 'package' or
more heavyweight 'NixOS' (module) tests.

This was suggested at
https://github.com/ryantm/nixpkgs-update/issues/260#issuecomment-821287971
and discussed further at
https://github.com/NixOS/nixpkgs/pull/119731

3 files changed, 46 insertions, 2 deletions

diff --git a/doc/contributing/coding-conventions.chapter.md b/doc/contributing/coding-conventions.chapter.md
index 7199fc63c8d0..85c8626bd99c 100644
--- a/doc/contributing/coding-conventions.chapter.md
+++ b/doc/contributing/coding-conventions.chapter.md
@@ -545,7 +545,26 @@ The following types of tests exists:
 
 Here in the nixpkgs manual we describe mostly _package tests_; for _module tests_ head over to the corresponding [section in the NixOS manual](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
 
-### Writing package tests {#ssec-package-tests-writing}
+### Writing inline package tests {#ssec-inline-package-tests-writing}
+
+For very simple tests, they can be written inline:
+
+```nix
+{ …, yq-go }:
+
+buildGoModule rec {
+  …
+
+  passthru.tests = {
+    simple = runCommand "${pname}-test" {} ''
+      echo "test: 1" | ${yq-go}/bin/yq eval -j > $out
+      [ "$(cat $out | tr -d $'\n ')" = '{"test":1}' ]
+    '';
+  };
+}
+```
+
+### Writing larger package tests {#ssec-package-tests-writing}
 
 This is an example using the `phoronix-test-suite` package with the current best practices.
 
diff --git a/doc/stdenv/meta.chapter.md b/doc/stdenv/meta.chapter.md
index f226a725480c..e4970f7e9649 100644
--- a/doc/stdenv/meta.chapter.md
+++ b/doc/stdenv/meta.chapter.md
@@ -134,7 +134,28 @@ Attribute Set `lib.platforms` defines [various common lists](https://github.com/
 This attribute is special in that it is not actually under the `meta` attribute set but rather under the `passthru` attribute set. This is due to how `meta` attributes work, and the fact that they are supposed to contain only metadata, not derivations.
 :::
 
-An attribute set with as values tests. A test is a derivation, which builds successfully when the test passes, and fails to build otherwise. A derivation that is a test needs to have `meta.timeout` defined.
+An attribute set with tests as values. A test is a derivation that builds when the test passes and fails to build otherwise.
+
+You can run these tests with:
+
+```ShellSession
+$ cd path/to/nixpkgs
+$ nix-build -A your-package.tests
+```
+
+#### Package tests
+
+Tests that are part of the source package are often executed in the `installCheckPhase`.
+
+Prefer `passthru.tests` for tests that are introduced in nixpkgs because:
+
+* `passthru.tests` tests the 'real' package, independently from the environment in which it was built
+* we can run `passthru.tests` independently
+* `installCheckPhase` adds overhead to each build
+
+For more on how to write and run package tests, see <xref linkend="sec-package-tests"/>.
+
+#### NixOS tests
 
 The NixOS tests are available as `nixosTests` in parameters of derivations. For instance, the OpenSMTPD derivation includes lines similar to:
 
@@ -148,6 +169,8 @@ The NixOS tests are available as `nixosTests` in parameters of derivations. For
 }
 ```
 
+NixOS tests run in a VM, so they are slower than regular package tests. For more information see [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
+
 ### `timeout` {#var-meta-timeout}
 
 A timeout (in seconds) for building the derivation. If the derivation takes longer than this time to build, it can fail due to breaking the timeout. However, all computers do not have the same computing power, hence some builders may decide to apply a multiplicative factor to this value. When filling this value in, try to keep it approximately consistent with other values already present in `nixpkgs`.
diff --git a/doc/stdenv/stdenv.chapter.md b/doc/stdenv/stdenv.chapter.md
index e3e7b4c850be..9befcaa51a9a 100644
--- a/doc/stdenv/stdenv.chapter.md
+++ b/doc/stdenv/stdenv.chapter.md
@@ -714,6 +714,8 @@ to `~/.gdbinit`. GDB will then be able to find debug information installed via `
 
 The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default `installCheck` calls `make installcheck`.
 
+It is often better to add tests that are not part of the source distribution to `passthru.tests` (see <xref linkend="var-meta-tests"/>). This avoids adding overhead to every build and enables us to run them independently.
+
 #### Variables controlling the installCheck phase {#variables-controlling-the-installcheck-phase}
 
 ##### `doInstallCheck` {#var-stdenv-doInstallCheck}