plugins: rework plugin loading (#1924)

* zellij: Move "populate_data_dir" to utils

and rewrite it to take a second, optional parameter. This allows
controlling whether only a specific asset should be installed. We do
this as preparation for being able to recover from a plugin version
mismatch error, where we will need to repopulate the data dir for
offending plugins.

* server/wasm_vm: Recover from PluginVersionMismatch

Adds a global plugin cache that stores, per plugin config, the wasmer
module associated with it. Make `start_plugin` take the pre-populated
module and create only the necessary modifications to the wasm env etc.

* utils: Fix formatting

* zellij: Delete non-existent module

* utils/shared: fix missing "set_permissions"

function when not on unix systems.

* server/wasm_vm: Don't populate cachedir

with serialized versions of the WASM plugins.

* utils/input/plugins: load wasm bytes from assets

for builtin plugin specifications. This foregoes any need to:

- Dump the plugin bytes to disk at all and
- subsequently read the plugin bytes from disk

* zellij: Disable default asset installation

which previously installed only the builtin plugins to disk. This is no
longer necessary because now we can load the builtin plugins directly
from the application binary.

* utils/input/plugins: Update docs

* utils/input/plugins: Add 'is_builtin' method

to `PluginConfig` that returns true if the plugin configuration refers
to a builtin plugin.

* wasm_vm: Remove plugin version mismatch handling

because a version mismatch in an internal plugin is now unfixable, with
the plugins being loaded from the running binary, and we have no control
over external plugins in the first place.

* cargo: Reintroduce feature flag

for `disable_automatic_asset_installation`

* utils/consts: Add `ASSET_MAP`

which currently contains the compiled WASM plugins.

* utils/shared: Fix clippy lint

* utils/errors: Add more `ZellijError` variants

* zellij: Make loading internal plugins optional

by reenabling the `disable_automatic_asset_installation` flag and
utilizing it for this purpose. Changes plugin search behavior to throw
better errors in case the builtin plugins cannot be found, depending on
the state of this feature.

* utils/errors: Apply rustfmt

* utils/setup: Allow dumping builtin plugins

to a specified folder on disk. This is meant to be an "escape hatch" for
users that have accidentally deleted the builtin plugins from disk (in
cases where the plugins aren't loaded from inside the zellij binary).

* utils/input/plugins: Update docs

* utils/setup: Add hint to `setup --check` output

when zellij was built without the `disable_automatic_asset_installation`
flag and will thus not read builtin plugins from the "PLUGIN DIR".

* utils/setup: Refine `setup --dump-plugins`

to dump to:

- The default "DATA DIR" when no option is provided with the argument,
  or
- The provided option, if existent

Also print a message to stdout with the destination folder that the
plugins are dumped to.

* server/wasm_vm: Ignore "NotFound" errors

when attempting to delete the non-existent plugin data directories. This
silences an error message that otherwise ends up in the logs when
quitting zellij.

* utils/errors: Extend "BuiltinPluginMissing" msg

to hint the user to the `zellij setup --dump-plugins` command to fix
their issues for them!

* utils/errors: Track caller in calls to `non_fatal`

which will hopefully, once closures can be annotated, allow us to
display the location of the call to `non_fatal` in log messages.

* utils/input/plugins: Fix plugin lookup

to prefer internal assets if available. It was previously broken because
sorting the paths vector before deduping it would bring the paths into a
wrong order, looking up in the plugin folder first.

Also print a log message when a plugin is being loaded from the internal
assets but exists on disk, too.

* Apply rustfmt

* make: build-e2e depends on wasm-opt-plugins

so it updates the assets when building the binary

* server/qwasm_vm: Remove var

* utils/consts: Add plugins from target folder

and include them in the asset map from there, too. Include plugins from
debug or release builds, depending on the build type.

* utils/consts: Take release plugins from assets

instead of the target/release folder. The latter will break
installations from crates.io, because we currently rely on including the
plugins we pre-compiled and distribute along with the binary.

* server/wasm_vm: Reintroduce .cache folder

to speedup subsequent application launches.

* cargo: Reorder workspace members

to improve behavior with `cargo make` with respect to compilation order.

* Makefile: restructure plugin tasks

* Makefile: Fix CI errors

* Makefile: More CI diagnosis

* github: Install wasm-opt in e2e test workflow

* Makefile: Build plugins for e2e-test target

* server/Wasm_vm: Reorder plugin folder creation

so no folders are created in the plugin cache when loading a plugin
fails due to not being present or similar.

* update plugins testcommit

* makefile: Change job order

* changelog: Add PR #1924

7 files changed, 257 insertions, 18 deletions

diff --git a/zellij-utils/Cargo.toml b/zellij-utils/Cargo.toml
index cf15ce212..021d90205 100644
--- a/zellij-utils/Cargo.toml
+++ b/zellij-utils/Cargo.toml
@@ -50,5 +50,9 @@ insta = { version = "1.6.0", features = ["backtrace"] }
 
 
 [features]
+# If this feature is NOT set (default):
+#   - builtin plugins (status-bar, tab-bar, ...) are loaded directly from the application binary
+# If this feature is set:
+#   - builtin plugins MUST be available from whatever is configured as `PLUGIN_DIR`
 disable_automatic_asset_installation = []
 unstable = []
diff --git a/zellij-utils/src/consts.rs b/zellij-utils/src/consts.rs
index 964938131..d8b60e351 100644
--- a/zellij-utils/src/consts.rs
+++ b/zellij-utils/src/consts.rs
@@ -35,6 +35,51 @@ pub const FEATURES: &[&str] = &[
     "disable_automatic_asset_installation",
 ];
 
+#[cfg(not(target_family = "wasm"))]
+pub use not_wasm::*;
+
+#[cfg(not(target_family = "wasm"))]
+mod not_wasm {
+    use lazy_static::lazy_static;
+    use std::collections::HashMap;
+    use std::path::PathBuf;
+
+    // Convenience macro to add plugins to the asset map (see `ASSET_MAP`)
+    macro_rules! add_plugin {
+        ($assets:expr, $plugin:literal) => {
+            $assets.insert(
+                PathBuf::from("plugins").join($plugin),
+                #[cfg(debug_assertions)]
+                include_bytes!(concat!(
+                    env!("CARGO_MANIFEST_DIR"),
+                    "/../target/wasm32-wasi/debug/",
+                    $plugin
+                ))
+                .to_vec(),
+                #[cfg(not(debug_assertions))]
+                include_bytes!(concat!(
+                    env!("CARGO_MANIFEST_DIR"),
+                    "/../assets/plugins/",
+                    $plugin
+                ))
+                .to_vec(),
+            );
+        };
+    }
+
+    lazy_static! {
+        // Zellij asset map
+        pub static ref ASSET_MAP: HashMap<PathBuf, Vec<u8>> = {
+            let mut assets = std::collections::HashMap::new();
+            add_plugin!(assets, "compact-bar.wasm");
+            add_plugin!(assets, "status-bar.wasm");
+            add_plugin!(assets, "tab-bar.wasm");
+            add_plugin!(assets, "strider.wasm");
+            assets
+        };
+    }
+}
+
 #[cfg(unix)]
 pub use unix_only::*;
 
diff --git a/zellij-utils/src/errors.rs b/zellij-utils/src/errors.rs
index 8cd3ab86e..86dac5777 100644
--- a/zellij-utils/src/errors.rs
+++ b/zellij-utils/src/errors.rs
@@ -14,6 +14,7 @@ use colored::*;
 use log::error;
 use serde::{Deserialize, Serialize};
 use std::fmt::{Display, Error, Formatter};
+use std::path::PathBuf;
 
 use miette::Diagnostic;
 use thiserror::Error as ThisError;
@@ -92,9 +93,16 @@ pub trait LoggableError<T>: Sized {
     ///     .print_error(|msg| println!("{msg}"))
     ///     .unwrap();
     /// ```
+    #[track_caller]
     fn print_error<F: Fn(&str)>(self, fun: F) -> Self;
 
     /// Convenienve function, calls `print_error` with the closure `|msg| log::error!("{}", msg)`.
+    // Dev note:
+    // Currently this hides the location of the caller, because it will show this very line as
+    // "source" of the logging call. This isn't correct, because it may have been called by other
+    // functions, too. To track this, we need to attach `#[track_caller]` to the closure below,
+    // which isn't stabilized yet: https://github.com/rust-lang/rust/issues/87417
+    #[track_caller]
     fn to_log(self) -> Self {
         self.print_error(|msg| log::error!("{}", msg))
     }
@@ -133,6 +141,7 @@ pub trait FatalError<T> {
     /// Discards the result type afterwards.
     ///
     /// [`to_log`]: LoggableError::to_log
+    #[track_caller]
     fn non_fatal(self);
 
     /// Mark results as being fatal.
@@ -394,6 +403,56 @@ pub enum ZellijError {
     #[error("failed to start PTY")]
     FailedToStartPty,
 
+    #[error(
+        "This version of zellij was built to load the core plugins from
+the globally configured plugin directory. However, a plugin wasn't found:
+
+    Plugin name: '{plugin_path}'
+    Plugin directory: '{plugin_dir}'
+
+If you're a user:
+    Please report this error to the distributor of your current zellij version
+
+If you're a developer:
+    Either make sure to include the plugins with the application (See feature
+    'disable_automatic_asset_installation'), or make them available in the
+    plugin directory.
+
+Possible fix for your problem:
+    Run `zellij setup --dump-plugins`, and optionally point it to your
+    'DATA DIR', visible in e.g. the output of `zellij setup --check`. Without
+    further arguments, it will use the default 'DATA DIR'.
+"
+    )]
+    BuiltinPluginMissing {
+        plugin_path: PathBuf,
+        plugin_dir: PathBuf,
+        #[source]
+        source: anyhow::Error,
+    },
+
+    #[error(
+        "It seems you tried to load the following builtin plugin:
+
+    Plugin name: '{plugin_path}'
+
+This is not a builtin plugin known to this version of zellij. If you were using
+a custom layout, please refer to the layout documentation at:
+
+    https://zellij.dev/documentation/creating-a-layout.html#plugin
+
+If you think this is a bug and the plugin is indeed an internal plugin, please
+open an issue on GitHub:
+
+    https://github.com/zellij-org/zellij/issues
+"
+    )]
+    BuiltinPluginNonexistent {
+        plugin_path: PathBuf,
+        #[source]
+        source: anyhow::Error,
+    },
+
     #[error("an error occured")]
     GenericError { source: anyhow::Error },
 }
diff --git a/zellij-utils/src/input/layout.rs b/zellij-utils/src/input/layout.rs
index 38806c21a..94b4f9322 100644
--- a/zellij-utils/src/input/layout.rs
+++ b/zellij-utils/src/input/layout.rs
@@ -168,7 +168,7 @@ pub struct RunPlugin {
     pub location: RunPluginLocation,
 }
 
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
 pub enum RunPluginLocation {
     File(PathBuf),
     Zellij(PluginTag),
diff --git a/zellij-utils/src/input/plugins.rs b/zellij-utils/src/input/plugins.rs
index 46e1a40fc..1ab3e831f 100644
--- a/zellij-utils/src/input/plugins.rs
+++ b/zellij-utils/src/input/plugins.rs
@@ -9,6 +9,8 @@ use serde::{Deserialize, Serialize};
 use url::Url;
 
 use super::layout::{RunPlugin, RunPluginLocation};
+#[cfg(not(target_family = "wasm"))]
+use crate::consts::ASSET_MAP;
 pub use crate::data::PluginTag;
 use crate::errors::prelude::*;
 
@@ -74,7 +76,7 @@ impl Default for PluginsConfig {
 }
 
 /// Plugin metadata
-#[derive(Clone, Debug, PartialEq, Deserialize, Serialize)]
+#[derive(Clone, Debug, PartialEq, Eq, Hash, Deserialize, Serialize)]
 pub struct PluginConfig {
     /// Path of the plugin, see resolve_wasm_bytes for resolution semantics
     pub path: PathBuf,
@@ -87,16 +89,20 @@ pub struct PluginConfig {
 }
 
 impl PluginConfig {
-    /// Resolve wasm plugin bytes for the plugin path and given plugin directory. Attempts to first
-    /// resolve the plugin path as an absolute path, then adds a ".wasm" extension to the path and
-    /// resolves that, finally we use the plugin directory joined with the path with an appended
-    /// ".wasm" extension. So if our path is "tab-bar" and the given plugin dir is
-    /// "/home/bob/.zellij/plugins" the lookup chain will be this:
+    /// Resolve wasm plugin bytes for the plugin path and given plugin directory.
+    ///
+    /// If zellij was built without the 'disable_automatic_asset_installation' feature, builtin
+    /// plugins (Starting with 'zellij:' in the layout file) are loaded directly from the
+    /// binary-internal asset map. Otherwise:
+    ///
+    /// Attempts to first resolve the plugin path as an absolute path, then adds a ".wasm"
+    /// extension to the path and resolves that, finally we use the plugin directory joined with
+    /// the path with an appended ".wasm" extension. So if our path is "tab-bar" and the given
+    /// plugin dir is "/home/bob/.zellij/plugins" the lookup chain will be this:
     ///
     /// ```bash
     ///   /tab-bar
     ///   /tab-bar.wasm
-    ///   /home/bob/.zellij/plugins/tab-bar.wasm
     /// ```
     ///
     pub fn resolve_wasm_bytes(&self, plugin_dir: &Path) -> Result<Vec<u8>> {
@@ -110,9 +116,8 @@ impl PluginConfig {
             &plugin_dir.join(&self.path).with_extension("wasm"),
         ];
         // Throw out dupes, because it's confusing to read that zellij checked the same plugin
-        // location multiple times
+        // location multiple times. Do NOT sort the vector here, because it will break the lookup!
         let mut paths = paths_arr.to_vec();
-        paths.sort_unstable();
         paths.dedup();
 
         // This looks weird and usually we would handle errors like this differently, but in this
@@ -122,8 +127,32 @@ impl PluginConfig {
         // spell it out right here.
         let mut last_err: Result<Vec<u8>> = Err(anyhow!("failed to load plugin from disk"));
         for path in paths {
+            // Check if the plugin path matches an entry in the asset map. If so, load it directly
+            // from memory, don't bother with the disk.
+            #[cfg(not(target_family = "wasm"))]
+            if !cfg!(feature = "disable_automatic_asset_installation") && self.is_builtin() {
+                let asset_path = PathBuf::from("plugins").join(path);
+                if let Some(bytes) = ASSET_MAP.get(&asset_path) {
+                    log::debug!("Loaded plugin '{}' from internal assets", path.display());
+
+                    if plugin_dir.join(path).with_extension("wasm").exists() {
+                        log::info!(
+                            "Plugin '{}' exists in the 'PLUGIN DIR' at '{}' but is being ignored",
+                            path.display(),
+                            plugin_dir.display()
+                        );
+                    }
+
+                    return Ok(bytes.to_vec());
+                }
+            }
+
+            // Try to read from disk
             match fs::read(&path) {
-                Ok(val) => return Ok(val),
+                Ok(val) => {
+                    log::debug!("Loaded plugin '{}' from disk", path.display());
+                    return Ok(val);
+                },
                 Err(err) => {
                     last_err = last_err.with_context(|| err_context(err, &path));
                 },
@@ -131,6 +160,29 @@ impl PluginConfig {
         }
 
         // Not reached if a plugin is found!
+        #[cfg(not(target_family = "wasm"))]
+        if self.is_builtin() {
+            // Layout requested a builtin plugin that wasn't found
+            let plugin_path = self.path.with_extension("wasm");
+
+            if cfg!(feature = "disable_automatic_asset_installation")
+                && ASSET_MAP.contains_key(&PathBuf::from("plugins").join(&plugin_path))
+            {
+                return Err(ZellijError::BuiltinPluginMissing {
+                    plugin_path,
+                    plugin_dir: plugin_dir.to_owned(),
+                    source: last_err.unwrap_err(),
+                })
+                .context("failed to load a plugin");
+            } else {
+                return Err(ZellijError::BuiltinPluginNonexistent {
+                    plugin_path,
+                    source: last_err.unwrap_err(),
+                })
+                .context("failed to load a plugin");
+            }
+        }
+
         return last_err;
     }
 
@@ -143,10 +195,14 @@ impl PluginConfig {
             PluginType::Headless => {},
         }
     }
+
+    pub fn is_builtin(&self) -> bool {
+        matches!(self.location, RunPluginLocation::Zellij(_))
+    }
 }
 
 /// Type of the plugin. Defaults to Pane.
-#[derive(Clone, Debug, PartialEq, Deserialize, Serialize)]
+#[derive(Clone, Debug, PartialEq, Deserialize, Serialize, Hash, Eq)]
 #[serde(rename_all = "kebab-case")]
 pub enum PluginType {
     // TODO: A plugin with output that's cloned across every pane in a tab, or across the entire
diff --git a/zellij-utils/src/setup.rs b/zellij-utils/src/setup.rs
index 3bce52375..ddbc59673 100644
--- a/zellij-utils/src/setup.rs
+++ b/zellij-utils/src/setup.rs
@@ -1,3 +1,5 @@
+#[cfg(not(target_family = "wasm"))]
+use crate::consts::ASSET_MAP;
 use crate::input::theme::Themes;
 use crate::{
     cli::{CliArgs, Command},
@@ -5,6 +7,7 @@ use crate::{
         FEATURES, SYSTEM_DEFAULT_CONFIG_DIR, SYSTEM_DEFAULT_DATA_DIR_PREFIX, VERSION,
         ZELLIJ_PROJ_DIR,
     },
+    errors::prelude::*,
     input::{
         config::{Config, ConfigError},
         layout::Layout,
@@ -172,6 +175,41 @@ pub fn dump_specified_layout(layout: &str) -> std::io::Result<()> {
     }
 }
 
+#[cfg(not(target_family = "wasm"))]
+pub fn dump_builtin_plugins(path: &PathBuf) -> Result<()> {
+    for (asset_path, bytes) in ASSET_MAP.iter() {
+        let plugin_path = path.join(asset_path);
+        plugin_path
+            .parent()
+            .with_context(|| {
+                format!(
+                    "failed to acquire parent path of '{}'",
+                    plugin_path.display()
+                )
+            })
+            .and_then(|parent_path| {
+                std::fs::create_dir_all(parent_path).context("failed to create parent path")
+            })
+            .with_context(|| {
+                format!(
+                    "failed to create folder '{}' to dump plugin '{}' to",
+                    path.display(),
+                    plugin_path.display()
+                )
+            })?;
+
+        std::fs::write(plugin_path, bytes)
+            .with_context(|| format!("failed to dump builtin plugin '{}'", asset_path.display()))?;
+    }
+
+    Ok(())
+}
+
+#[cfg(target_family = "wasm")]
+pub fn dump_builtin_plugins(_path: &PathBuf) -> Result<()> {
+    Ok(())
+}
+
 #[derive(Debug, Default, Clone, Args, Serialize, Deserialize)]
 pub struct Setup {
     /// Dump the default configuration file to stdout
@@ -192,6 +230,17 @@ pub struct Setup {
     #[clap(long, value_parser)]
     pub dump_layout: Option<String>,
 
+    /// Dump the builtin plugins to DIR or "DATA DIR" if unspecified
+    #[clap(
+        long,
+        value_name = "DIR",
+        value_parser,
+        exclusive = true,
+        min_values = 0,
+        max_values = 1
+    )]
+    pub dump_plugins: Option<Option<PathBuf>>,
+
     /// Generates completion for the specified shell
     #[clap(long, value_name = "SHELL", value_parser)]
     pub generate_completion: Option<String>,
@@ -263,7 +312,7 @@ impl Setup {
     }
 
     /// General setup helpers
-    pub fn from_cli(&self) -> std::io::Result<()> {
+    pub fn from_cli(&self) -> Result<()> {
         if self.clean {
             return Ok(());
         }
@@ -292,15 +341,24 @@ impl Setup {
     }
 
     /// Checks the merged configuration
-    pub fn from_cli_with_options(
-        &self,
-        opts: &CliArgs,
-        config_options: &Options,
-    ) -> std::io::Result<()> {
+    pub fn from_cli_with_options(&self, opts: &CliArgs, config_options: &Options) -> Result<()> {
         if self.check {
             Setup::check_defaults_config(opts, config_options)?;
             std::process::exit(0);
         }
+
+        if let Some(maybe_path) = &self.dump_plugins {
+            let data_dir = &opts.data_dir.clone().unwrap_or_else(get_default_data_dir);
+            let dir = match maybe_path {
+                Some(path) => path,
+                None => data_dir,
+            };
+
+            println!("Dumping plugins to '{}'", dir.display());
+            dump_builtin_plugins(&dir)?;
+            std::process::exit(0);
+        }
+
         Ok(())
     }
 
@@ -361,6 +419,18 @@ impl Setup {
         }
         writeln!(&mut message, "[DATA DIR]: {:?}", data_dir).unwrap();
         message.push_str(&format!("[PLUGIN DIR]: {:?}\n", plugin_dir));
+        if !cfg!(feature = "disable_automatic_asset_installation") {
+            writeln!(
+                &mut message,
+                " Builtin, default plugins will not be loaded from disk."
+            )
+            .unwrap();
+            writeln!(
+                &mut message,
+                " Create a custom layout if you require this behavior."
+            )
+            .unwrap();
+        }
         if let Some(layout_dir) = layout_dir {
             writeln!(&mut message, "[LAYOUT DIR]: {:?}", layout_dir).unwrap();
         } else {
diff --git a/zellij-utils/src/shared.rs b/zellij-utils/src/shared.rs
index f89acaf57..3ef24995d 100644
--- a/zellij-utils/src/shared.rs
+++ b/zellij-utils/src/shared.rs
@@ -24,6 +24,11 @@ mod unix_only {
     }
 }
 
+#[cfg(not(unix))]
+pub fn set_permissions(_path: &std::path::Path, _mode: u32) -> std::io::Result<()> {
+    Ok(())
+}
+
 pub fn ansi_len(s: &str) -> usize {
     from_utf8(&strip(s).unwrap()).unwrap().width()
 }