workflows: add check for docbook/md manual equality

we want to make sure that rendering the manual from markdown without
going through docbook produces (semantically) the same output as with
going through docbook. to ensure this we'll build the manual twice, run
each manual through html-tidy to generate a normalized form and diff
the normalized forms. we don't want to compare raw output because that
exposes us to a lot of whitespace we'd have to reproduce exactly in the
MD render.

this check may be relaxed even further in the future, but hopefully not
by much.

2 files changed, 84 insertions, 0 deletions

diff --git a/.github/workflows/compare-manuals.sh b/.github/workflows/compare-manuals.sh
new file mode 100755
index 000000000000..b2cc68c7831d
--- /dev/null
+++ b/.github/workflows/compare-manuals.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env nix-shell
+#! nix-shell -i bash -p html-tidy
+
+set -euo pipefail
+shopt -s inherit_errexit
+
+normalize() {
+  tidy \
+      --anchor-as-name no \
+      --coerce-endtags no \
+      --escape-scripts no \
+      --fix-backslash no \
+      --fix-style-tags no \
+      --fix-uri no \
+      --indent yes \
+      --wrap 0 \
+      < "$1" \
+      2> /dev/null
+}
+
+diff -U3 <(normalize "$1") <(normalize "$2")
diff --git a/.github/workflows/manual-rendering.yml b/.github/workflows/manual-rendering.yml
new file mode 100644
index 000000000000..6c4a9507c5ca
--- /dev/null
+++ b/.github/workflows/manual-rendering.yml
@@ -0,0 +1,63 @@
+name: "Check NixOS Manual DocBook rendering against MD rendering"
+
+
+on:
+  schedule:
+    # * is a special character in YAML so you have to quote this string
+    # Check every 24 hours
+    - cron:  '0 0 * * *'
+
+permissions:
+  contents: read
+
+jobs:
+  check-rendering-equivalence:
+    permissions:
+      issues: write  # for peter-evans/create-or-update-comment to create or update comment
+    if: github.repository_owner == 'NixOS'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: cachix/install-nix-action@v17
+        with:
+          # explicitly enable sandbox
+          extra_nix_config: sandbox = true
+      - uses: cachix/cachix-action@v10
+        with:
+          # This cache is for the nixpkgs repo checks and should not be trusted or used elsewhere.
+          name: nixpkgs-ci
+          signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
+
+      - name: Build DocBook and MD manuals
+        run: |
+          export NIX_PATH=nixpkgs=$(pwd)
+          nix-build \
+            --option restrict-eval true \
+            -o docbook nixos/release.nix \
+            -A manual.x86_64-linux
+          nix-build \
+            --option restrict-eval true \
+            --arg configuration '{ documentation.nixos.options.allowDocBook = false; }' \
+            -o md nixos/release.nix \
+            -A manual.x86_64-linux
+
+      - name: Compare DocBook and MD manuals
+        id: check
+        run: |
+          .github/workflows/compare-manuals.sh \
+            docbook/share/doc/nixos/options.html \
+            md/share/doc/nixos/options.html
+
+      # if the manual can't be built we don't want to notify anyone.
+      # while this may temporarily hide rendering failures it will be a lot
+      # less noisy until all nixpkgs pull requests have stopped using
+      # docbook for option docs.
+      - name: Comment on failure
+        uses: peter-evans/create-or-update-comment@v2
+        if: ${{ failure() && steps.check.conclusion == 'failure' }}
+        with:
+          issue-number: 189318
+          body: |
+            Markdown and DocBook manuals do not agree.
+
+            Check https://github.com/NixOS/nixpkgs/actions/runs/${{ github.run_id }} for details.