path: root/CHANGELOG.md

VERSION NEXT
============

## Dependency changes

* Minimum required Python version bumped to 3.8
  ([#552](https://github.com/papis/papis/pull/552)).
* Moved to `pyproject.toml` and removed `setup.py` completely. We use
  [hatchling](https://github.com/pypa/hatch/tree/master/backend) as the build
  backend.
* Removed [arxiv2bib](https://github.com/nathangrigg/arxiv2bib) in favor of
  [arxiv.py](https://github.com/lukasschwab/arxiv.py).
* Removed [tqdm](https://github.com/tqdm/tqdm) dependency (using a progress bar
  from `prompt_toolkit` instead).
* Made `chardet` an optional dependency. This is an optional dependency of
  `bs4`, `requests` and `feedparser` and should be installed if possible.
* Added [markdownify](https://github.com/matthewwithanm/python-markdownify) as
  an optional dependency for the Zenodo downloader.
* Added [platformdirs](https://github.com/platformdirs/platformdirs) dependency
  for platform-specific config locations.

## Features

### New: Improved Windows support

The support for Windows has been significantly improved in this release. On top of
a slew of fixes, there is also a build that creates a Windows executable and
installer using [PyInstaller](https://pyinstaller.org/en/stable).

These can currently be found only as artifacts under the
[Windows release workflow](https://github.com/papis/papis/actions/workflows/windows.yml).
We do urge anyone knowledgeable and interested to try them out and report any
issues, so that they can be improved before they make it into regular releases.

Huge thanks to @kiike for all the work on this!

### New: Add `cache` command ([#603](https://github.com/papis/papis/pull/603))

The `cache` command has been added in order to provide more control over the
papis cache. Accordingly, the equivalent commands `papis --cc` and
`papis --clear-cache` have been removed and can be replaced by the equivalent
```sh
papis cache clear
```

The command can now also update only specific documents using
```sh
papis cache update QUERY
```

You can learn more about the cache command in the documentation.

### New: EPUB support for the web application

Now you can read EPUB files from the comfort of the web application. The
workflow is similar to the existing one for PDFs and it uses the
[epubjs-reader](https://github.com/futurepress/epubjs-reader) library.

### New: Exporter for the Typst Hayagriva format ([#559](https://github.com/papis/papis/pull/559))

[Typst](https://github.com/typst/typst) is a new typesetting system with some
very cool features and a more modern outlook compared to LaTeX. While it
supports BibTeX, it also has its own bibliography format called Hayagriva
(a YAML file). Papis can now export directly to this format using
```
papis export --format typst QUERY
```

### New: Add `init` command ([#620](https://github.com/papis/papis/pull/620))

A new `papis init` command was added to initialize a new configuration file
and add additional libraries. The command is mainly interactive and sets some
standard default options. It can be used as
```sh
papis init /path/to/my/library
```

**Warning**: We currently use the standard Python `configparser` which does not
preserve comments. This means that updating a configuration file using `papis init`
will remove any comments and possibly reorder some options.

### New: Add `tag` command ([#648](https://github.com/papis/papis/pull/648))

A new `papis tag` command was added to handle adding and removing tags
(or keywords) for a document. For example, to add a few tags to a set of
documents
```
papis tag --add cool-project --add from-steve --all QUERY
```
or change a tag for a mislabeled document
```
papis tag --remove biology --add neuroscience QUERY
```

For more examples see the documentation.

**Warning**: The `tag` command expects that the tags are a list, not a single
string with a separator. You can quickly transform your tags into a list using
`papis doctor` e.g.
```sh
papis \
    --set doctor-key-type-check-keys '["tags:list"]' \
    --set doctor-key-type-check-separator ' ' \
    doctor --fix --all --explain -t key-type QUERY
```
where you may need to change the separator to match your choice.

### Major: Expand `update` command ([#681](https://github.com/papis/papis/pull/681))

The update command got some major improvements this release. It now allows much
more fine-grained modifications of a document without requiring the user to open
a full editor. For example, it can now

* Append to a key using
    ```sh
    papis update --append title ' (Volume 2)' <QUERY>
    ```

* Remove items from a list using
    ```sh
    papis update --remove tags 'physics' <QUERY>
    ```

* Remove a key completely from the document
    ```sh
    papis update --drop eprinttype <QUERY>
    ```

For more examples see the documentation.

### Major: Expand `doctor` command

The `papis doctor` command has seen many additions and modifications in this
release. First, the `papis add` and `papis update` commands can now automatically
run selected fixed with the `--auto-doctor` flag
([#598](https://github.com/papis/papis/pull/598)). This makes it less likely for
ill-formatted documents to make it into the library in the first place!

Many checks for BibTeX have been added: `biblatex-type-alias`, `biblatex-key-alias`
and `biblatex-required-keys` ([#663](https://github.com/papis/papis/pull/663)).
These checks ensure that the document has all the keys required by the BibLaTeX
engine to output well-formatted entries for each document type.

Other smaller noteworthy changes:

* `key-type`: added fixers that automatically convert some types
  ([#652](https://github.com/papis/papis/pull/652)
   [#656](https://github.com/papis/papis/pull/656)).
* `keys-exist`: added fixers for `author` and `author_list`
  ([#655](https://github.com/papis/papis/pull/655))
* `duplicated-values`: new check for duplicate values inside lists
  ([#695](https://github.com/papis/papis/pull/695))
* `bibtex-type`: add fixer to automatically convert known document types
  ([#732](https://github.com/papis/papis/pull/732))

### Minor: Plugin helpers ([#680](https://github.com/papis/papis/pull/680) and [#752](https://github.com/papis/papis/pull/752))

A new module `papis.testing` was introduced to help with testing Papis-related
functionality. They allow easy setup of temporary configurations and temporary
libraries for testing purposes.

A new module `papis.sphinx_ext` was introduced to help with Papis-related
documentation. This allows defining configuration options in the Sphinx documentation
that describe their types and default values in sync with the ones used in the
code.

These have mostly been promoted to separate module so that plugins can easily
make use of the same infrastructure.

### Minor: Configuration file location ([#745](https://github.com/papis/papis/pull/745))

The location of the configuration file has been standardized using
[platformdirs](https://github.com/platformdirs/platformdirs). On Linux-y systems,
this should not make any difference, since Papis has been using the XDG
environment variables to get the proper location.

However, on other systems, especially Windows and macOS, the **location of the
configuration files will change**. Papis has implemented a simple migration, so
this should be fairly automatic.

## Other noteworthy features

* Add more BibTeX key conversions
  ([#561](https://github.com/papis/papis/pull/561)
   [#562](https://github.com/papis/papis/pull/562)).
* Allow setting refs in `papis update`
  ([#593](https://github.com/papis/papis/pull/539)).
* Add proper extensions to temporary downloaded files to better differentiate
  ([#548](https://github.com/papis/papis/pull/548)).
* Updated `Dockerfile` and added `release.Dockerfile`
  ([#597](https://github.com/papis/papis/pull/597)).
* Allow `papis.format.format` to set default values and not leak exceptions
  into document fields.
  ([#596](https://github.com/papis/papis/pull/596)).
* Added a `flake.nix` script for easier use with the nix package manager
  ([#600](https://github.com/papis/papis/pull/600)).
* Updated default `opentool` on Windows
  ([#569](https://github.com/papis/papis/pull/569)).
* `papis` command: when invoking `papis` without a subcommand the help message
  is printed. This should avoid some confusions for new users
  ([#603](https://github.com/papis/papis/pull/603)).
* Allow multiple `--doc-folder` arguments to commands that support it
  ([#635](https://github.com/papis/papis/pull/635)).
* Improve ScienceDirect abstract extraction
  ([#637](https://github.com/papis/papis/pull/637))
* Better support for `--batch` in `papis add`
  ([#630](https://github.com/papis/papis/pull/630)).
* Add `--[no-]download-files` flags to `papis add`
  ([#641](https://github.com/papis/papis/pull/641)).
* Add a downloader for ACL Anthology documents
  ([#575](https://github.com/papis/papis/pull/575))
* Expand `papis list` to list all plugins and other extensions
  ([#716](https://github.com/papis/papis/pull/716))
* Add `biblatex-software` types to our BibTeX module
  ([#719](https://github.com/papis/papis/pull/719))
* Sort tags alphabetically in `papis serve`
  ([#730](https://github.com/papis/papis/pull/730))
* Add a `--move` flag to `papis add`
  ([#740](https://github.com/papis/papis/pull/740))
* Enhance `python` formatter with some additional conversions
  ([#709](https://github.com/papis/papis/pull/709))
* Fix the `these.fr` downloader
  ([#729](https://github.com/papis/papis/pull/792)).
* Add a Zenodo downloader
  ([#770](https://github.com/papis/papis/pull/770)). This uses some of the
  fields from [biblatex-software](https://ctan.org/pkg/biblatex-software?lang=en).
* Make file and folder cleanup more configurable
  ([#803](https://github.com/papis/papis/pull/803)).
* Expand the `rename` command
  ([#810](https://github.com/papis/papis/pull/810)).
* Allow configuration for marked lines margins and make marked and unmarked
  margins span the whole document entry.
  ([#820](https://github.com/papis/papis/pull/820))

## Bug fixes

* Fixed Papis to BibTeX key conversions
  ([#555](https://github.com/papis/papis/pull/555)).
* Fix encoding of `info.yaml` files on Windows
  ([#571](https://github.com/papis/papis/pull/571)).
* Fixed quoting in some calls to external commands
  ([#580](https://github.com/papis/papis/pull/580)).
* Warn and specify how external scripts are loaded.
  ([#594](https://github.com/papis/papis/pull/594)).
* Fix manpage generation from Sphinx
  ([#609](https://github.com/papis/papis/pull/609)).
* Ensure that user provided data in `papis add` using `--set key value` is not
  overwritten by importers
  ([#616](https://github.com/papis/papis/pull/616)).
* Fix some boolean flags working in an unexpected manner
  ([#636](https://github.com/papis/papis/pull/636)).
* Fix opening multiple files in the default picker
  ([#693](https://github.com/papis/papis/pull/693))
* Do not escape verbatim BibTeX fields like `url`
  ([#739](https://github.com/papis/papis/pull/739))

VERSION v0.13 (May 7, 2023)
===========================

## Features

### New: Special `papis_id` key ([#449](https://github.com/papis/papis/pull/449))

In order to make plugin writing easier we have decided to introduce a
`papis_id` key in the document's info files. This key essentially functions as
a UUID for the document. Many commands have gained support for this, e.g.

    papis list --id query
    papis open papis_id:someid

For more information see the documentation.

**Important**: This change requires updating the database backend. This can be
done by simply clearing the cache using `papis --cc`.

### New: `papis doctor` command ([#421](https://github.com/papis/papis/pull/421))

A new `papis doctor` command was introduced that can be used to check that
document information is correct, up to date, or is nicely linted. The command
also supports fixing incorrect information for many cases!

It can be used as

    papis doctor --checks files --explain query

There are several useful checks implemented already (non-exhaustive list):

* `files`: checks that the document files actually exist in the file system.
* `keys-exist`: checks that the provided keys exist in the document.
* `bibtex-type`: checks that the document type is a valid BibTeX type.
* `refs`: checks that the document ref exists and does not contain invalid characters.

For more information see the documentation.

### New: `papis citations` command ([#451](https://github.com/papis/papis/pull/451))

A new `papis citations` command was added to handle retrieving citation
information for a document. This includes both papers cited by the document
and those which cite the document itself.

For more information see the documentation.


### New: Add DBLP support ([#489](https://github.com/papis/papis/pull/489))

Add support for the [DBLP](dblp.org/) database. The database can now be explored
using

    papis explore dblp -q query pick

and documents can be imported directly using the DBLP key or URL

    papis add --from dblp 'conf/iccg/EncarnacaoAFFGM93'

### Major improvements to the web application ([#424](https://github.com/papis/papis/pull/424))

The Papis web application, which can be accessed with `papis serve`, has seen
major development since the last version. It now has support for

* editing of `info.yaml` file of a document (requires `ace.js`).
* viewing document PDF files directly in the browser (requires `pdfjs`).
* viewing LaTeX in titles, abstracts and others (requires `KaTeX`).
* exporting the document to BibTeX.
* showing citations for the document.
* showing errors from `papis doctor`.
* showing a timeline for queried documents.

And many other general interface and robustness improvements.

### Other noteworthy features

* A major overhaul of the README ([#415](https://github.com/papis/papis/pull/415)).
* Consistent use of ANSI colors ([#462](https://github.com/papis/papis/pull/462)).
* Better notes handling in
  `papis update` ([#404](https://github.com/papis/papis/pull/404)),
  `papis edit` ([#391](https://github.com/papis/papis/pull/391)) and
  the Papis picker ([#319](https://github.com/papis/papis/pull/319)).
* Improvements to BiBTeX export (
  [#412](https://github.com/papis/papis/pull/412)
  [#444](https://github.com/papis/papis/pull/444)
  [#443](https://github.com/papis/papis/pull/443)
  [#468](https://github.com/papis/papis/pull/468)).
* Support [NO_COLOR](https://no-color.org/)
  ([#437](https://github.com/papis/papis/pull/437)).
* Major overhaul of the `papis config` command, which can now show defaults
  and select specific sections ([#454](https://github.com/papis/papis/pull/454)).
* Fix shell completion on `bash`, `zsh`, and `fish`
  ([#478](https://github.com/papis/papis/pull/478)).
* Update the logging format ([#465](https://github.com/papis/papis/pull/465)).
* Updated multiple downloaders to support the latest version of the website
  and for better data extraction (
  [#441](https://github.com/papis/papis/pull/441)
  [#447](https://github.com/papis/papis/pull/447)).
* Avoid downloading files on `papis add` in certain scenarios
  ([#505](https://github.com/papis/papis/pull/505)).
* Recognize `DjVu` files correctly ([#522](https://github.com/papis/papis/pull/522)).
* Add USENIX downloader ([#523](https://github.com/papis/papis/pull/523)).
* Support remote URLs in `papis addto` ([#541](https://github.com/papis/papis/pull/541)).
* Add `sort-reverse` configuration option ([#543](https://github.com/papis/papis/pull/543)).

## Bug fixes

* Properly detach processes on `papis open` ([#476](https://github.com/papis/papis/pull/476)).
* Fix link extraction in `crossref` ([#480](https://github.com/papis/papis/pull/480)).
* Do not generate refs in downloaders ([#483](https://github.com/papis/papis/pull/483)).
* Fix empty config file handling ([#497](https://github.com/papis/papis/pull/497)).
* Fix database query for `jinja2` ([#499](https://github.com/papis/papis/pull/499)).
* Fix crash on trying to add a duplicate document ([#510](https://github.com/papis/papis/pull/510)).
* Fix many typos ([#540](https://github.com/papis/papis/pull/540)).

VERSION v0.12 (May 23, 2022)
============================

Many issues were resolved and new (and old) contributors
made the following changes possible.

## Add hook infrastructure
A basic hook infrastructure has been added to be able to
use emacs-like hooks for some commands.

## Add `additional` keyword for the formatter plugin system

## `papis bibtex`
- Add `import` command to import bibtex files as papis documents
  into the library.
- Add `filter-cited`.

## `papis exec`
- Add command `exec` to run python scripts in the environment of the
  papis executable.

## papis picker
- You can now pick several elements with the key binding `c-t`
- Add support for fzf picker,
  check out the [documentation](https://papis.readthedocs.io/en/latest/configuration.html#fzf-integration).

## `papis merge`
- Add the command `papis merge` to merge documents in pairs.

## `papis browse`
- Add `-n` and `--print` to just print the url to be opened.

## Downloaders

- Add an `acm` downloader
- Add an `Project Euclid` downloader

## Notes

Now you can remove notes files with `papis rm --notes`
or have templates for the notes.
Check out the
[notes-template variable](https://papis.readthedocs.io/en/latest/configuration.html#config-settings-notes-template).

## Web application `papis serve`
- The new, simple and experimental web application is available
  through the `papis serve` command.
  Feel free to make suggestions.

## MacOS
- disable multiprocessing by default on mac due to lack of performance and source
  of strange behaviour.

## Community
- usage of github discussions and #papis channel on libera.

VERSION v0.11 (October 17, 2020)
================================

## `papis explore`
- Add `add` command to simply add documents retrieved.

## `papis.export`
- Add the key `_papis_local_folder` so that third-party apps
  can get the documents' paths without having to go again through papis.

## `papis bibtex`
- Add `unique` command to be able to merge different bib files
  and filter out repetitions.
- Add `doctor` to check for fields in a bibfile.
- Add `iscited` to check which bib items are cited in a text file.

## Add Format plugin
- Sébastien Popoff has added a format plugin architecture, so now `jinja2`
  is available again as a plugin.

## Bibtex
- Improve the reference building routine.
- Change the default `ref-format` to
  ```
  "{doc[title]:.15} {doc[author]:.6} {doc[year]}",
  ```
- The default ref if no reference could be built will not be
  using the folder name as before, but using the values in the `info.yaml`
  limited to 30 characters.

## `papis add`
- Create a reference at the time of adding if no reference exists.

## `papis browse`
- Add more documentation.
- Add an `ads` handler to jump into the `ads` website of the paper
  using a doi.


VERSION v0.10 (May 2, 2020)
===========================

- Fix several bugs.
- Typecheck the whole codebase
  and drop support for python versions up to 3.4, the latter version included.
- Add `--sort` and `--reverse` flags to most command line commands, together
  with the `sort-file` configuration option.
  see [doc](https://papis.readthedocs.io/en/latest/configuration.html#config-settings-sort-field).
- Add `time-stamps` in order to sort chronologically documents
  see [doc](https://papis.readthedocs.io/en/latest/configuration.html#config-settings-time-stamp).
- Add `--doc-folder` to most command line commands for better bash-scripting
  capabilities.
- Add `--logfile` flag in order to dump log messages to a file.
- Move `papis.utils.format_doc` to `papis.document.format_doc`
  in order to minimize circular dependencies.
- Add a `--profile` flag to profile the papis run.
- Define a new API for general pickers. (TBD)
- Add a citeseerx downloader.

## `papis add`
- Add an experimental importer selector when smart matching inputs.
  See [here](https://asciinema.org/a/i2kXyZMNaT8n7YRz7DcVIfqm5).

## `papis bibtex`
- Add `papis bibtex browse`.

## `Run` and `Git` command
- Add `--pick`, `--doc-folder`, `--all` and `--sort`
  flags so that we can choose a document to run the shell or git command
  in that folder or in all folders matching a given query introduced by
  `--pick`.



VERSION v0.9 (October 21, 2019)
===============================

## Plugin architecture ##

A new plugin architecture is in place.
For more information please refer to
[the documentation](https://papis.readthedocs.io/en/latest/plugin.html)

## Git interface ##

Now some usual commands have a `--git` flag that lets the command work
alongside git. For instance if the `--git` flag is passed to `papis-edit`,
it will add and commit the `info.yaml` file automatically. The same applies
to `papis-add`, `papis-addto`, `papis-update` and `papis-rm`.

You can activate by default the `--git` flag using the `use-git` configuration
option.
**For devs**: The main functions to implement this interface are found in the
papis module `papis.git`.

## `papis add` ##

- The configuration settings `file-name` and `folder-name` are now
  ```
  add-file-name
  add-folder-name
  ```
  so that they become more readable and understandable.
  Also the flag `--name` is now `--folder-name`.
- The flag `--commit` now has the name `--git`.
- The flag `--dir` now has the more descriptive name `--subfolder`.
- The flag `--no-docu