cli: replace clap with lexopt and supporting code

ripgrep began it's life with docopt for argument parsing. Then it moved
to Clap and stayed there for a number of years. Clap has served ripgrep
well, and it probably could continue to serve ripgrep well, but I ended
up deciding to move off of it.

Why?

The first time I had the thought of moving off of Clap was during the
2->3->4 transition. I thought the 3.x and 4.x releases were great, but
for me, it ended up moving a little too quickly. Since the release of
4.x was telegraphed around when 3.x came out, I decided to just hold off
and wait to migrate to 4.x instead of doing a 3.x migration followed
shortly by another 4.x migration. Of course, I just never ended up doing
the migration at all. I never got around to it and there just wasn't a
compelling reason for me to upgrade. While I never investigated it, I
saw an upgrade as a non-trivial amount of work in part because I didn't
encapsulate the usage of Clap enough.

The above is just what got me started thinking about it. It wasn't
enough to get me to move off of it on its own. What ended up pushing me
over the edge was a combination of factors:

* As mentioned above, I didn't want to run on the migration treadmill.
This has proven to not be much of an issue, but at the time of the
2->3->4 releases, I didn't know how long Clap 4.x would be out before a
5.x would come out.
* The release of lexopt[1] caught my eye. IMO, that crate demonstrates
exactly how something new can arrive on the scene and just thoroughly
solve a problem minimalistically. It has the docs, the reasoning, the
simple API, the tests and good judgment. It gets all the weird corner
cases right that Clap also gets right (and is part of why I was
originally attracted to Clap).
* I have an overall desire to reduce the size of my dependency tree. In
part because a smaller dependency tree tends to correlate with better
compile times, but also in part because it reduces my reliance and trust
on others. It lets me be the "master" of ripgrep's destiny by reducing
the amount of behavior that is the result of someone else's decision
(whether good or bad).
* I perceived that Clap solves a more general problem than what I
actually need solved. Despite the vast number of flags that ripgrep has,
its requirements are actually pretty simple. We just need simple
switches and flags that support one value. No multi-value flags. No
sub-commands. And probably a lot of other functionality that Clap has
that makes it so flexible for so many different use cases. (I'm being
hand wavy on the last point.)

With all that said, perhaps most importantly, the future of ripgrep
possibly demands a more flexible CLI argument parser. In today's world,
I would really like, for example, flags like `--type` and `--type-not`
to be able to accumulate their repeated values into a single sequence
while respecting the order they appear on the CLI. For example, prior
to this migration, `rg regex-automata -Tlock -ttoml` would not return
results in `Cargo.lock` in this repository because the `-Tlock` always
took priority even though `-ttoml` appeared after it. But with this
migration, `-ttoml` now correctly overrides `-Tlock`. We would like to
do similar things for `-g/--glob` and `--iglob` and potentially even
now introduce a `-G/--glob-not` flag instead of requiring users to use
`!` to negate a glob. (Which I had done originally to work-around this
problem.) And some day, I'd like to add some kind of boolean matching to
ripgrep perhaps similar to how `git grep` does it. (Although I haven't
thought too carefully on a design yet.) In order to do that, I perceive
it would be difficult to implement correctly in Clap.

I believe that this last point is possible to implement correctly in
Clap 2.x, although it is awkward to do so. I have not looked closely
enough at the Clap 4.x API to know whether it's still possible there. In
any case, these were enough reasons to move off of Clap and own more of
the argument parsing process myself.

This did require a few things:

* I had to write my own logic for how arguments are combined into one
single state object. Of course, I wanted this. This was part of the
upside. But it's still code I didn't have to write for Clap.
* I had to write my own shell completion generator.
* I had to write my own `-h/--help` output generator.
* I also had to write my own man page generator. Well, I had to do this
with Clap 2.x too, although my understanding is that Clap 4.x supports
this. With that said, without having tried it, my guess is that I
probably wouldn't have liked the output it generated because I
ultimately had to write most of the roff by hand myself to get the man
page I wanted. (This also had the benefit of dropping the build
dependency on asciidoc/asciidoctor.)

While this is definitely a fair bit of extra work, it overall only cost
me a couple days. IMO, that's a good trade off given that this code is
unlikely to change again in any substantial way. And it should also
allow for more flexible semantics going forward.

Fixes #884, Fixes #1648, Fixes #1701, Fixes #1814, Fixes #1966

[1]: https://docs.rs/lexopt/0.3.0/lexopt/index.html

1 files changed, 20 insertions, 239 deletions

diff --git a/build.rs b/build.rs
index eae7f856..db9584bf 100644
--- a/build.rs
+++ b/build.rs
@@ -1,70 +1,24 @@
-use std::{
-    env,
-    fs::{self, File},
-    io::{self, Read, Write},
-    path::Path,
-    process,
-};
-
-use clap::Shell;
-
-use crate::app::{RGArg, RGArgKind};
-
-#[allow(dead_code)]
-#[path = "crates/core/app.rs"]
-mod app;
-
 fn main() {
-    // OUT_DIR is set by Cargo and it's where any additional build artifacts
-    // are written.
-    let Some(outdir) = env::var_os("OUT_DIR") else {
-        eprintln!(
-            "OUT_DIR environment variable not defined. \
-             Please file a bug: \
-             https://github.com/BurntSushi/ripgrep/issues/new"
-        );
-        process::exit(1);
-    };
-    fs::create_dir_all(&outdir).unwrap();
-
-    let stamp_path = Path::new(&outdir).join("ripgrep-stamp");
-    if let Err(err) = File::create(&stamp_path) {
-        panic!("failed to write {}: {}", stamp_path.display(), err);
-    }
-    if let Err(err) = generate_man_page(&outdir) {
-        eprintln!("failed to generate man page: {}", err);
-    }
-
-    // Use clap to build completion files.
-    let mut app = app::app();
-    app.gen_completions("rg", Shell::Bash, &outdir);
-    app.gen_completions("rg", Shell::Fish, &outdir);
-    app.gen_completions("rg", Shell::PowerShell, &outdir);
-    // Note that we do not use clap's support for zsh. Instead, zsh completions
-    // are manually maintained in `complete/_rg`.
-
-    // Make the current git hash available to the build.
-    if let Some(rev) = git_revision_hash() {
-        println!("cargo:rustc-env=RIPGREP_BUILD_GIT_HASH={}", rev);
-    }
-    // Embed a Windows manifest and set some linker options. The main reason
-    // for this is to enable long path support on Windows. This still, I
-    // believe, requires enabling long path support in the registry. But if
-    // that's enabled, then this will let ripgrep use C:\... style paths that
-    // are longer than 260 characters.
+    set_git_revision_hash();
     set_windows_exe_options();
 }
 
+/// Embed a Windows manifest and set some linker options.
+///
+/// The main reason for this is to enable long path support on Windows. This
+/// still, I believe, requires enabling long path support in the registry. But
+/// if that's enabled, then this will let ripgrep use C:\... style paths that
+/// are longer than 260 characters.
 fn set_windows_exe_options() {
     static MANIFEST: &str = "pkg/windows/Manifest.xml";
 
-    let Ok(target_os) = env::var("CARGO_CFG_TARGET_OS") else { return };
-    let Ok(target_env) = env::var("CARGO_CFG_TARGET_ENV") else { return };
+    let Ok(target_os) = std::env::var("CARGO_CFG_TARGET_OS") else { return };
+    let Ok(target_env) = std::env::var("CARGO_CFG_TARGET_ENV") else { return };
     if !(target_os == "windows" && target_env == "msvc") {
         return;
     }
 
-    let Ok(mut manifest) = env::current_dir() else { return };
+    let Ok(mut manifest) = std::env::current_dir() else { return };
     manifest.push(MANIFEST);
     let Some(manifest) = manifest.to_str() else { return };
 
@@ -77,189 +31,16 @@ fn set_windows_exe_options() {
     println!("cargo:rustc-link-arg-bin=rg=/WX");
 }
 
-fn git_revision_hash() -> Option<String> {
-    let output = process::Command::new("git")
-        .args(&["rev-parse", "--short=10", "HEAD"])
-        .output()
-        .ok()?;
-    let v = String::from_utf8_lossy(&output.stdout).trim().to_string();
-    if v.is_empty() {
-        None
-    } else {
-        Some(v)
-    }
-}
-
-fn generate_man_page<P: AsRef<Path>>(outdir: P) -> io::Result<()> {
-    // If asciidoctor isn't installed, fallback to asciidoc.
-    if let Err(err) = process::Command::new("asciidoctor").output() {
-        eprintln!(
-            "Could not run 'asciidoctor' binary, falling back to 'a2x'."
-        );
-        eprintln!("Error from running 'asciidoctor': {}", err);
-        return legacy_generate_man_page::<P>(outdir);
-    }
-    // 1. Read asciidoctor template.
-    // 2. Interpolate template with auto-generated docs.
-    // 3. Save interpolation to disk.
-    // 4. Use asciidoctor to convert to man page.
-    let outdir = outdir.as_ref();
-    let cwd = env::current_dir()?;
-    let tpl_path = cwd.join("doc").join("rg.1.txt.tpl");
-    let txt_path = outdir.join("rg.1.txt");
-
-    let mut tpl = String::new();
-    File::open(&tpl_path)?.read_to_string(&mut tpl)?;
-    let options =
-        formatted_options()?.replace("&#123;", "{").replace("&#125;", "}");
-    tpl = tpl.replace("{OPTIONS}", &options);
-
-    let githash = git_revision_hash();
-    let githash = githash.as_ref().map(|x| &**x);
-    tpl = tpl.replace("{VERSION}", &app::long_version(githash, false));
-
-    File::create(&txt_path)?.write_all(tpl.as_bytes())?;
-    let result = process::Command::new("asciidoctor")
-        .arg("--doctype")
-        .arg("manpage")
-        .arg("--backend")
-        .arg("manpage")
-        .arg(&txt_path)
-        .spawn()?
-        .wait()?;
-    if !result.success() {
-        let msg =
-            format!("'asciidoctor' failed with exit code {:?}", result.code());
-        return Err(ioerr(msg));
-    }
-    Ok(())
-}
-
-fn legacy_generate_man_page<P: AsRef<Path>>(outdir: P) -> io::Result<()> {
-    // If asciidoc isn't installed, then don't do anything.
-    if let Err(err) = process::Command::new("a2x").output() {
-        eprintln!("Could not run 'a2x' binary, skipping man page generation.");
-        eprintln!("Error from running 'a2x': {}", err);
-        return Ok(());
-    }
-    // 1. Read asciidoc template.
-    // 2. Interpolate template with auto-generated docs.
-    // 3. Save interpolation to disk.
-    // 4. Use a2x (part of asciidoc) to convert to man page.
-    let outdir = outdir.as_ref();
-    let cwd = env::current_dir()?;
-    let tpl_path = cwd.join("doc").join("rg.1.txt.tpl");
-    let txt_path = outdir.join("rg.1.txt");
+/// Make the current git hash available to the build as the environment
+/// variable `RIPGREP_BUILD_GIT_HASH`.
+fn set_git_revision_hash() {
+    use std::process::Command;
 
-    let mut tpl = String::new();
-    File::open(&tpl_path)?.read_to_string(&mut tpl)?;
-    tpl = tpl.replace("{OPTIONS}", &formatted_options()?);
-
-    let githash = git_revision_hash();
-    let githash = githash.as_ref().map(|x| &**x);
-    tpl = tpl.replace("{VERSION}", &app::long_version(githash, false));
-
-    File::create(&txt_path)?.write_all(tpl.as_bytes())?;
-    let result = process::Command::new("a2x")
-        .arg("--no-xmllint")
-        .arg("--doctype")
-        .arg("manpage")
-        .arg("--format")
-        .arg("manpage")
-        .arg(&txt_path)
-        .spawn()?
-        .wait()?;
-    if !result.success() {
-        let msg = format!("'a2x' failed with exit code {:?}", result.code());
-        return Err(ioerr(msg));
-    }
-    Ok(())
-}
-
-fn formatted_options() -> io::Result<String> {
-    let mut args = app::all_args_and_flags();
-    args.sort_by(|x1, x2| x1.name.cmp(&x2.name));
-
-    let mut formatted = vec![];
-    for arg in args {
-        if arg.hidden {
-            continue;
-        }
-        // ripgrep only has two positional arguments, and probably will only
-        // ever have two positional arguments, so we just hardcode them into
-        // the template.
-        if let app::RGArgKind::Positional { .. } = arg.kind {
-            continue;
-        }
-        formatted.push(formatted_arg(&arg)?);
-    }
-    Ok(formatted.join("\n\n"))
-}
-
-fn formatted_arg(arg: &RGArg) -> io::Result<String> {
-    match arg.kind {
-        RGArgKind::Positional { .. } => {
-            panic!("unexpected positional argument")
-        }
-        RGArgKind::Switch { long, short, multiple } => {
-            let mut out = vec![];
-
-            let mut header = format!("--{}", long);
-            if let Some(short) = short {
-                header = format!("-{}, {}", short, header);
-            }
-            if multiple {
-                header = format!("*{}* ...::", header);
-            } else {
-                header = format!("*{}*::", header);
-            }
-            writeln!(out, "{}", header)?;
-            writeln!(out, "{}", formatted_doc_txt(arg)?)?;
-
-            Ok(String::from_utf8(out).unwrap())
-        }
-        RGArgKind::Flag { long, short, value_name, multiple, .. } => {
-            let mut out = vec![];
-
-            let mut header = format!("--{}", long);
-            if let Some(short) = short {
-                header = format!("-{}, {}", short, header);
-            }
-            if multiple {
-                header = format!("*{}* _{}_ ...::", header, value_name);
-            } else {
-                header = format!("*{}* _{}_::", header, value_name);
-            }
-            writeln!(out, "{}", header)?;
-            writeln!(out, "{}", formatted_doc_txt(arg)?)?;
-
-            Ok(String::from_utf8(out).unwrap())
-        }
-    }
-}
-
-fn formatted_doc_txt(arg: &RGArg) -> io::Result<String> {
-    let paragraphs: Vec<String> = arg
-        .doc_long
-        .replace("{", "&#123;")
-        .replace("}", r"&#125;")
-        // Hack to render ** literally in man page correctly. We can't put
-        // these crazy +++ in the help text directly, since that shows
-        // literally in --help output.
-        .replace("*-g 'foo/**'*", "*-g +++'foo/**'+++*")
-        .split("\n\n")
-        .map(|s| s.to_string())
-        .collect();
-    if paragraphs.is_empty() {
-        return Err(ioerr(format!("missing docs for --{}", arg.name)));
-    }
-    let first = format!("  {}", paragraphs[0].replace("\n", "\n  "));
-    if paragraphs.len() == 1 {
-        return Ok(first);
+    let args = &["rev-parse", "--short=10", "HEAD"];
+    let Ok(output) = Command::new("git").args(args).output() else { return };
+    let rev = String::from_utf8_lossy(&output.stdout).trim().to_string();
+    if rev.is_empty() {
+        return;
     }
-    Ok(format!("{}\n+\n{}", first, paragraphs[1..].join("\n+\n")))
-}
-
-fn ioerr(msg: String) -> io::Error {
-    io::Error::new(io::ErrorKind::Other, msg)
+    println!("cargo:rustc-env=RIPGREP_BUILD_GIT_HASH={}", rev);
 }