plugins: rework plugin loading (#1924)

* zellij: Move "populate_data_dir" to utils

and rewrite it to take a second, optional parameter. This allows
controlling whether only a specific asset should be installed. We do
this as preparation for being able to recover from a plugin version
mismatch error, where we will need to repopulate the data dir for
offending plugins.

* server/wasm_vm: Recover from PluginVersionMismatch

Adds a global plugin cache that stores, per plugin config, the wasmer
module associated with it. Make `start_plugin` take the pre-populated
module and create only the necessary modifications to the wasm env etc.

* utils: Fix formatting

* zellij: Delete non-existent module

* utils/shared: fix missing "set_permissions"

function when not on unix systems.

* server/wasm_vm: Don't populate cachedir

with serialized versions of the WASM plugins.

* utils/input/plugins: load wasm bytes from assets

for builtin plugin specifications. This foregoes any need to:

- Dump the plugin bytes to disk at all and
- subsequently read the plugin bytes from disk

* zellij: Disable default asset installation

which previously installed only the builtin plugins to disk. This is no
longer necessary because now we can load the builtin plugins directly
from the application binary.

* utils/input/plugins: Update docs

* utils/input/plugins: Add 'is_builtin' method

to `PluginConfig` that returns true if the plugin configuration refers
to a builtin plugin.

* wasm_vm: Remove plugin version mismatch handling

because a version mismatch in an internal plugin is now unfixable, with
the plugins being loaded from the running binary, and we have no control
over external plugins in the first place.

* cargo: Reintroduce feature flag

for `disable_automatic_asset_installation`

* utils/consts: Add `ASSET_MAP`

which currently contains the compiled WASM plugins.

* utils/shared: Fix clippy lint

* utils/errors: Add more `ZellijError` variants

* zellij: Make loading internal plugins optional

by reenabling the `disable_automatic_asset_installation` flag and
utilizing it for this purpose. Changes plugin search behavior to throw
better errors in case the builtin plugins cannot be found, depending on
the state of this feature.

* utils/errors: Apply rustfmt

* utils/setup: Allow dumping builtin plugins

to a specified folder on disk. This is meant to be an "escape hatch" for
users that have accidentally deleted the builtin plugins from disk (in
cases where the plugins aren't loaded from inside the zellij binary).

* utils/input/plugins: Update docs

* utils/setup: Add hint to `setup --check` output

when zellij was built without the `disable_automatic_asset_installation`
flag and will thus not read builtin plugins from the "PLUGIN DIR".

* utils/setup: Refine `setup --dump-plugins`

to dump to:

- The default "DATA DIR" when no option is provided with the argument,
  or
- The provided option, if existent

Also print a message to stdout with the destination folder that the
plugins are dumped to.

* server/wasm_vm: Ignore "NotFound" errors

when attempting to delete the non-existent plugin data directories. This
silences an error message that otherwise ends up in the logs when
quitting zellij.

* utils/errors: Extend "BuiltinPluginMissing" msg

to hint the user to the `zellij setup --dump-plugins` command to fix
their issues for them!

* utils/errors: Track caller in calls to `non_fatal`

which will hopefully, once closures can be annotated, allow us to
display the location of the call to `non_fatal` in log messages.

* utils/input/plugins: Fix plugin lookup

to prefer internal assets if available. It was previously broken because
sorting the paths vector before deduping it would bring the paths into a
wrong order, looking up in the plugin folder first.

Also print a log message when a plugin is being loaded from the internal
assets but exists on disk, too.

* Apply rustfmt

* make: build-e2e depends on wasm-opt-plugins

so it updates the assets when building the binary

* server/qwasm_vm: Remove var

* utils/consts: Add plugins from target folder

and include them in the asset map from there, too. Include plugins from
debug or release builds, depending on the build type.

* utils/consts: Take release plugins from assets

instead of the target/release folder. The latter will break
installations from crates.io, because we currently rely on including the
plugins we pre-compiled and distribute along with the binary.

* server/wasm_vm: Reintroduce .cache folder

to speedup subsequent application launches.

* cargo: Reorder workspace members

to improve behavior with `cargo make` with respect to compilation order.

* Makefile: restructure plugin tasks

* Makefile: Fix CI errors

* Makefile: More CI diagnosis

* github: Install wasm-opt in e2e test workflow

* Makefile: Build plugins for e2e-test target

* server/Wasm_vm: Reorder plugin folder creation

so no folders are created in the plugin cache when loading a plugin
fails due to not being present or similar.

* update plugins testcommit

* makefile: Change job order

* changelog: Add PR #1924

1 files changed, 2 insertions, 0 deletions

diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index d1750cb18..22c243e1f 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -43,6 +43,8 @@ jobs:
       run: rustup target add x86_64-unknown-linux-musl
     - name: Install cargo-make
       run: nix profile install nixpkgs#cargo-make
+    - name: Install wasm-opt
+      run: sudo apt-get install -y --no-install-recommends binaryen
       #run: cargo install --debug cargo-make
     - name: Build asset
       run: cargo make build-e2e