| Age | Commit message (Collapse) | Author |
|---|
|
Fixes #138
|
|
This also drops an ancient rand 0.3 dependency.
|
|
reverse command to reverse rows of CSV data.
Unlike sort command, it preserves order of rows with the same key.
PR #134
|
|
The `xsv partition` has been available for some time,
but is not visible in the help message or in `xsv --list`.
PR #131
|
|
This includes the update to docopt 0.8, which moves us over to Serde.
Woohoo!
We also bump the minimum Rust version supported to 1.20.0.
|
|
For the most part, this switches write_record to write_byte_record where
possible, and also moves some uses of the `byte_records` iterator to the
manual `read_byte_record` method, which permits us to amortize
allocation.
This also removes the use of normal selection for `xsv search`, which
was causing a huge slow down.
Also, bump to csv 1.0.0.beta.3.
|
|
This commit resists the urge to refactor/rewrite xsv and ports it over
to the new CSV API. It made a lot of things cleaner and even improved
the performance of core commands like `count`, `sample`, `search`,
`select` and `slice`.
This also removes the last remaining (dubious) uses of `unsafe` within
xsv.
Benchmarks before/after:
benchmark before after
count 0.26s 175.05 MB/sec 0.11 413.76 MB/sec
flatten 4.53s 10.04 MB/sec 4.54 10.02 MB/sec
flatten_condensed 4.72s 9.64 MB/sec 4.45 10.22 MB/sec
frequency 1.91s 23.82 MB/sec 1.82 25.00 MB/sec
index 0.28s 162.54 MB/sec 0.12 379.28 MB/sec
sample_10 0.43s 105.84 MB/sec 0.18 252.85 MB/sec
sample_1000 0.44s 103.44 MB/sec 0.18 252.85 MB/sec
sample_100000 0.50s 91.02 MB/sec 0.29 156.94 MB/sec
search 0.59s 77.14 MB/sec 0.27 168.56 MB/sec
select 0.41s 111.00 MB/sec 0.14 325.09 MB/sec
sort 2.59s 17.57 MB/sec 2.18 20.87 MB/sec
slice_one_middle 0.22s 206.88 MB/sec 0.08 568.92 MB/sec
slice_one_middle_index 0.01s 4551.36 MB/sec 0.01 4551.36 MB/sec
stats 1.26s 36.12 MB/sec 1.09 41.75 MB/sec
stats_index 0.19s 239.54 MB/sec 0.15 303.42 MB/sec
stats_everything 2.13s 21.36 MB/sec 1.94 23.46 MB/sec
stats_everything_index 1.00s 45.51 MB/sec 0.93 48.93 MB/sec
|
|
This patch adds a new `xsv partition` subcommand, and makes a few minor
modifications to `xsv split` for consistency. The new subcommand is
used as follows:
xsv partition state out/ cities.csv
This will take the data in `cities.csv` and use the `state` column to
create new file names, creating files like:
out/CA.csv
out/TX.csv
If the `states` column is empty, we'll put the data into:
out/empty.csv
There's an option for specifying the filename template:
xsv partition --filename=state-{}.csv state out/ cities.csv
This will create:
out/state-CA.csv
out/state-TX.csv
This `--filename` option may also be used for `xsv split`.
There's also a `--prefix-length` argument, which can be used to limit
the number of files created:
xsv partition --prefix-length=3 zip out/ customers.csv
This will create files using the first three digits of the zip code:
out/000.csv
out/010.csv
out/011.csv
out/213.csv
Note that if you try to split into more than roughly 500 files on
Windows or 1000 files on Linux systems, you may need to adjust `ulimit`
or the local equivalent. (This was discussed in the original proposal.)
You can also write `--filename={}/file.csv`. This supports two use
cases:
xsv partition --filename {}/cities.csv state . all-cities.csv
xsv partition --filename {}/monuments.csv state . all-monuments.csv
xsv partition --filename {}/parks.csv state . all-parks.csv
Above, we want to partition our records by state into separate
directories, but we have multiple kinds of data.
xsv partition --filename {}/$(uuidgen).csv . input1.csv
xsv partition --filename {}/$(uuidgen).csv . input2.csv
Above, we're running multiple (possibly parallel) copies of xsv and we
want to parition the data into multiple directories without a filename
clash.
Fixes #51.
|
|
|
|
This in particular updates to the latest packages available and fixes a
few visibility warnings produced by rustc (that will eventually become
hard errors).
|
|
|
|
Fixes #26.
|
|
|
|
There's no particular advantage in these simplistic cases, but
dogfooding is A Good Thing.
|
|
We still can't run tests on stable (because we need to be able to set
file times), but that's fine for now.
|
|
Essentially, use the new `filetime` crate to do stuff with last
modified/accessed file times.
|
|
Not quite on Rust stable yet, but close.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Also, benchmarks seem OK... no major regressions. Weird.
|
|
|
|
We seem to have suffered some major perf regressions. Ug.
|
|
Basically reducing the number of unstable features being used.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This makes it work better with tab completion.
|
|
|
|
|
