| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We look for similar flag names via Jaccard index on ngrams. In my
experience this tends to work better than Levenshtein or other edit
distance based metrics. Principally because it allows for out-of-order
suggestions. For example, --case-smart will result in a suggestion for
--smart-case, even though the edit distance between them is pretty big.
This is something Clap did for us. I initially thought it wasn't
necessary to add this back in, but I realized it wouldn't be much work
and might actually be helpful to folks.
|
|
Fixes #1622
|
|
Fixes #1635
|
|
Fixes #1709
|
|
Basically, unless the -a/--text flag is given, it is generally always an
error to search for an explicit NUL byte because the binary detection
will prevent it from matching.
Fixes #1838
|
|
Fixes #2046
|
|
Closes #2122
|
|
The way we do this is a little hokey but I believe it is correct.
Fixes #2246
|
|
Fixes #2337
|
|
The --vimgrep flag has some severe footguns when using a pattern that
matches very frequently. We had already written some docs to warn about
that, but now we also include a suggestion to avoid exorbitant heap
usage.
Closes #2505
|
|
This mimics what was written in the man page.
Closes #2401
|
|
This clarifies that the paths are not sorted in a fully lexicographic
order, but that / is treated specially.
Fixes #2418
|
|
This adds info about whether PCRE2 is available or not to the output of
--version. Essentially, --version now subsumes --pcre2-version, although
we do retain the former because it (usefully) emits an exit code based
on whether PCRE2 is available or not.
Closes #2645
|
|
Previously, we were applying the -M/--max-columns flag *before* triming
prefix ASCII whitespace. But this doesn't make a whole lot of sense. We
should be trimming first, but the result of trimming is ultimately what
we'll be printing and that's what -M/--max-columns should be applied to.
Fixes #2458
|
|
Closes #2479
|
|
When one does not provide any paths to ripgrep to search, it has to
guess between searching stdin and the current working directory. It is
possible for this guess to be wrong, and having the heuristics and the
choice in the debug logs is useful for diagnosing this.
The failure mode here is still pretty bad because you need to know to
reach for the `--debug` flag in the first place. Namely, the typical
failure mode is that ripgrep tries to search stdin while the intent is
for it to search the current working directory, and thus likely blocking
forever waiting for data on stdin.
(Arguably this is a problem with the process architecture that invokes
ripgrep. It shouldn't give ripgrep an open stdin handle that isn't
closed.)
Closes #2524
|
|
Previously, every worker would increment the shared num_pending count on
every new work item, and decrement it after finishing them, leading to
lots of contention. Now, we only track the number of workers actively
running, so there is no contention except when workers go to sleep or
wake up.
Closes #2642
|
|
Interestingly, ripgrep now only has two available ASCII letter short
flags remaining: -k and -y.
Closes #2643, Closes #2644
|
|
Instead, we just roll our own. A slow version of this is pretty simple
to do, and that's what we write here. The `base64` crate supports a lot
more functionality and is quite fast, but we care about neither of those
things for this particular aspect of ripgrep. (base64 is only used for
non-UTF-8 data or file paths, which are both quite rare.)
|
|
As suggested by @epage[1].
Ad hoc timings on my i7-12900K:
before cargo build: 4.91s
before cargo build release: 8.05s
after cargo build: 4.69s
after cargo build release: 7.83s
... pretty underwhelming if you ask me. Ah well. And on my M2 mac mini:
before cargo build: 6.18s
before cargo build release: 14.50s
after cargo build: 5.52s
after cargo build release: 13.44s
Still kind of underwhelming, but definitely better. It shaves a full
second off of compile times in release mode. I went back to my
i7-12900K, but passed `-j1` to `cargo build` to force single threaded
mode:
before cargo build: 19.44s
before cargo build release: 50.64s
after cargo build: 16.76s
after cargo build release: 48.00s
Which seems pretty consistent with the modest improvements above.
Looking at `cargo build --timings`, the beefiest chunk of time is spent
in compiling `regex-automata`, by far. This is fine because it's core
functionality. I wish a fast general purpose regex engine with its
internals exposed as a separately versioned library didn't require so
much code... Blech.
[1]: https://old.reddit.com/r/rust/comments/17rd8ww/faster_compilation_with_the_parallel_frontend_in/k8igjlg/
|
|
The idea is that by bringing derives in via serde's optional feature, it
was inhibiting compilation speed[1]. We try to fix that by depending on
`serde_derive` as a distinct dependency.
It does seem to improve overall compilation time, but only by about 0.5
seconds. With that said, my machine has a lot of cores, so it's possible
this will help more on less powerful CPUs.
[1]: https://old.reddit.com/r/rust/comments/17rd8ww/faster_compilation_with_the_parallel_frontend_in/k8igjlg/
|
|
I introduced a regression in the migration off of the clap by having
both the buffer writer and the printer be responsible for printing file
separators in multi-threaded search. The buffer writer owns that
responsibility in multi-threaded search.
|
|
ripgrep began it's life with docopt for argument parsing. Then it moved
to Clap and stayed there for a number of years. Clap has served ripgrep
well, and it probably could continue to serve ripgrep well, but I ended
up deciding to move off of it.
Why?
The first time I had the thought of moving off of Clap was during the
2->3->4 transition. I thought the 3.x and 4.x releases were great, but
for me, it ended up moving a little too quickly. Since the release of
4.x was telegraphed around when 3.x came out, I decided to just hold off
and wait to migrate to 4.x instead of doing a 3.x migration followed
shortly by another 4.x migration. Of course, I just never ended up doing
the migration at all. I never got around to it and there just wasn't a
compelling reason for me to upgrade. While I never investigated it, I
saw an upgrade as a non-trivial amount of work in part because I didn't
encapsulate the usage of Clap enough.
The above is just what got me started thinking about it. It wasn't
enough to get me to move off of it on its own. What ended up pushing me
over the edge was a combination of factors:
* As mentioned above, I didn't want to run on the migration treadmill.
This has proven to not be much of an issue, but at the time of the
2->3->4 releases, I didn't know how long Clap 4.x would be out before a
5.x would come out.
* The release of lexopt[1] caught my eye. IMO, that crate demonstrates
exactly how something new can arrive on the scene and just thoroughly
solve a problem minimalistically. It has the docs, the reasoning, the
simple API, the tests and good judgment. It gets all the weird corner
cases right that Clap also gets right (and is part of why I was
originally attracted to Clap).
* I have an overall desire to reduce the size of my dependency tree. In
part because a smaller dependency tree tends to correlate with better
compile times, but also in part because it reduces my reliance and trust
on others. It lets me be the "master" of ripgrep's destiny by reducing
the amount of behavior that is the result of someone else's decision
(whether good or bad).
* I perceived that Clap solves a more general problem than what I
actually need solved. Despite the vast number of flags that ripgrep has,
its requirements are actually pretty simple. We just need simple
switches and flags that support one value. No multi-value flags. No
sub-commands. And probably a lot of other functionality that Clap has
that makes it so flexible for so many different use cases. (I'm being
hand wavy on the last point.)
With all that said, perhaps most importantly, the future of ripgrep
possibly demands a more flexible CLI argument parser. In today's world,
I would really like, for example, flags like `--type` and `--type-not`
to be able to accumulate their repeated values into a single sequence
while respecting the order they appear on the CLI. For example, prior
to this migration, `rg regex-automata -Tlock -ttoml` would not return
results in `Cargo.lock` in this repository because the `-Tlock` always
took priority even though `-ttoml` appeared after it. But with this
migration, `-ttoml` now correctly overrides `-Tlock`. We would like to
do similar things for `-g/--glob` and `--iglob` and potentially even
now introduce a `-G/--glob-not` flag instead of requiring users to use
`!` to negate a glob. (Which I had done originally to work-around this
problem.) And some day, I'd like to add some kind of boolean matching to
ripgrep perhaps similar to how `git grep` does it. (Although I haven't
thought too carefully on a design yet.) In order to do that, I perceive
it would be difficult to implement correctly in Clap.
I believe that this last point is possible to implement correctly in
Clap 2.x, although it is awkward to do so. I have not looked closely
enough at the Clap 4.x API to know whether it's still possible there. In
any case, these were enough reasons to move off of Clap and own more of
the argument parsing process myself.
This did require a few things:
* I had to write my own logic for how arguments are combined into one
single state object. Of course, I wanted this. This was part of the
upside. But it's still code I didn't have to write for Clap.
* I had to write my own shell completion generator.
* I had to write my own `-h/--help` output generator.
* I also had to write my own man page generator. Well, I had to do this
with Clap 2.x too, although my understanding is that Clap 4.x supports
this. With that said, without having tried it, my guess is that I
probably wouldn't have liked the output it generated because I
ultimately had to write most of the roff by hand myself to get the man
page I wanted. (This also had the benefit of dropping the build
dependency on asciidoc/asciidoctor.)
While this is definitely a fair bit of extra work, it overall only cost
me a couple days. IMO, that's a good trade off given that this code is
unlikely to change again in any substantial way. And it should also
allow for more flexible semantics going forward.
Fixes #884, Fixes #1648, Fixes #1701, Fixes #1814, Fixes #1966
[1]: https://docs.rs/lexopt/0.3.0/lexopt/index.html
|
|
This avoids needing to import and call GlobSetBuilder::new explicitly.
Closes #2635
|
|
This permits the value to be surrounded in double quotes. It's still not
perfect, but probably better than it was. Getting this to be more
correct will likely require writing (or using) a real parser, which I'm
not particularly incliend to do at present.
Fixes #2392, Closes #2629
|
|
This commit adds `anyhow` as a dependency and switches over to it from
Box<dyn Error>.
It actually looks like I've kept all of my errors rather shallow, such
that we don't get a huge benefit from anyhow at present. But now that
anyhow is in use, I expect to use its "context" feature more going
forward.
|
|
There's no particular reason for this change. I happened to be looking
at the code again and realized that stealing from your left neighbour
or your right neighbour shouldn't make a difference (and indeed perf is
the same in my benchmarks).
Closes #2624
|
|
As a result of discussion in #2611, it seems prudent to disable
hyperlinks by default. Ideally they would be enabled, but it looks like
some environments may barf on them. Since this is the first release with
hyperlink support, it makes sense to me at least to make users opt into
them. This does not preclude enabling them by default in future
releases.
|
|
Previously, we had logic to skip our own inner literal optimization if
the regex itself was already (likely) accelerated. It turns out that the
presence of a Unicode word boundary can defeat acceleration to a point.
It's likely enough that even if the underlying regex is accelerated, it
would be prudent to do our own inner literal optimization if the pattern
has a Unicode word boundary.
Normally a Unicode word boundary doesn't defeat literal optimizations,
since even the slower engines can make use of *prefix* literal
optimizations. But a regex can be accelerated via its own inner or
suffix literal optimizations, and those require the use of a DFA (or
lazy DFA). Since DFAs crap out on haystacks that contain a non-ASCII
Unicode scalar value when the regex contains a Unicode word boundary, it
follows that an "accelerated" can still wind up being quite slow.
(An "accelerated" regex can also slow down because of restrictions on
avoiding quadratic behavior, but I believe this happens less frequently
and is not as severe as the slow down as a result of Unicode word
boundaries. Namely, avoiding quadratic behavior just means giving up on
the inner literal optimization for a single search. In which case, the
regex engine can still fall back to a normal forward DFA. That will
definitely be slower than an inner literal optimization done by ripgrep,
but not quite as dramatic as it would be when DFAs can't be used at
all.)
|
|
|
|
I did this in the course of trying to optimize it. I don't believe I
made it any faster, but the refactoring led to code that I think is
more readable.
|
