path: root/doc/languages-frameworks/python.section.md

# Python

## User Guide

### Using Python

#### Overview

Several versions of the Python interpreter are available on Nix, as well as a
high amount of packages. The attribute `python` refers to the default
interpreter, which is currently CPython 2.7. It is also possible to refer to
specific versions, e.g. `python35` refers to CPython 3.5, and `pypy` refers to
the default PyPy interpreter.

Python is used a lot, and in different ways. This affects also how it is
packaged. In the case of Python on Nix, an important distinction is made between
whether the package is considered primarily an application, or whether it should
be used as a library, i.e., of primary interest are the modules in
`site-packages` that should be importable.

In the Nixpkgs tree Python applications can be found throughout, depending on
what they do, and are called from the main package set. Python libraries,
however, are in separate sets, with one set per interpreter version.

The interpreters have several common attributes. One of these attributes is
`pkgs`, which is a package set of Python libraries for this specific
interpreter. E.g., the `toolz` package corresponding to the default interpreter
is `python.pkgs.toolz`, and the CPython 3.5 version is `python35.pkgs.toolz`.
The main package set contains aliases to these package sets, e.g.
`pythonPackages` refers to `python.pkgs` and `python35Packages` to
`python35.pkgs`.

#### Installing Python and packages

The Nix and NixOS manuals explain how packages are generally installed. In the
case of Python and Nix, it is important to make a distinction between whether the
package is considered an application or a library.

Applications on Nix are typically installed into your user
profile imperatively using `nix-env -i`, and on NixOS declaratively by adding the
package name to `environment.systemPackages` in `/etc/nixos/configuration.nix`.
Dependencies such as libraries are automatically installed and should not be
installed explicitly.

The same goes for Python applications and libraries. Python applications can be
installed in your profile. But Python libraries you would like to use for
development cannot be installed, at least not individually, because they won't
be able to find each other resulting in import errors. Instead, it is possible
to create an environment with `python.buildEnv` or `python.withPackages` where
the interpreter and other executables are able to find each other and all of the
modules.

In the following examples we create an environment with Python 3.5, `numpy` and
`toolz`. As you may imagine, there is one limitation here, and that's that
you can install only one environment at a time. You will notice the complaints
about collisions when you try to install a second environment.

##### Environment defined in separate `.nix` file

Create a file, e.g. `build.nix`, with the following expression
```nix
with import <nixpkgs> {};

python35.withPackages (ps: with ps; [ numpy toolz ])
```
and install it in your profile with
```shell
nix-env -if build.nix
```
Now you can use the Python interpreter, as well as the extra packages (`numpy`,
`toolz`) that you added to the environment.

##### Environment defined in `~/.config/nixpkgs/config.nix`

If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g.
using `config.nix`,
```nix
{ # ...

  packageOverrides = pkgs: with pkgs; {
    myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
  };
}
```
and install it in your profile with
```shell
nix-env -iA nixpkgs.myEnv
```
The environment is is installed by referring to the attribute, and considering
the `nixpkgs` channel was used.

##### Environment defined in `/etc/nixos/configuration.nix`

For the sake of completeness, here's another example how to install the environment system-wide.

```nix
{ # ...

  environment.systemPackages = with pkgs; [
    (python35.withPackages(ps: with ps; [ numpy toolz ]))
  ];
}
```

#### Temporary Python environment with `nix-shell`

The examples in the previous section showed how to install a Python environment
into a profile. For development you may need to use multiple environments.
`nix-shell` gives the possibility to temporarily load another environment, akin
to `virtualenv`.

There are two methods for loading a shell with Python packages. The first and recommended method
is to create an environment with `python.buildEnv` or `python.withPackages` and load that. E.g.
```sh
$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'
```
opens a shell from which you can launch the interpreter
```sh
[nix-shell:~] python3
```
The other method, which is not recommended, does not create an environment and requires you to list the packages directly,

```sh
$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz
```
Again, it is possible to launch the interpreter from the shell.
The Python interpreter has the attribute `pkgs` which contains all Python libraries for that specific interpreter.

##### Load environment from `.nix` expression
As explained in the Nix manual, `nix-shell` can also load an
expression from a `.nix` file. Say we want to have Python 3.5, `numpy`
and `toolz`, like before, in an environment. Consider a `shell.nix` file
with
```nix
with import <nixpkgs> {};

(python35.withPackages (ps: [ps.numpy ps.toolz])).env
```
Executing `nix-shell` gives you again a Nix shell from which you can run Python.

What's happening here?

1. We begin with importing the Nix Packages collections. `import <nixpkgs>` imports the `<nixpkgs>` function, `{}` calls it and the `with` statement brings all attributes of `nixpkgs` in the local scope. These attributes form the main package set.
2. Then we create a Python 3.5 environment with the `withPackages` function.
3. The `withPackages` function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages `numpy` and `toolz` from the package set.

##### Execute command with `--run`
A convenient option with `nix-shell` is the `--run`
option, with which you can execute a command in the `nix-shell`. We can
e.g. directly open a Python shell
```sh
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"
```
or run a script
```sh
$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
```

##### `nix-shell` as shebang
In fact, for the second use case, there is a more convenient method. You can
add a [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) to your script
specifying which dependencies `nix-shell` needs. With the following shebang, you
can just execute `./myscript.py`, and it will make available all dependencies and
run the script in the `python3` shell.

```py
#! /usr/bin/env nix-shell
#! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])"

import numpy

print(numpy.__version__)
```

### Developing with Python

Now that you know how to get a working Python environment with Nix, it is time
to go forward and start actually developing with Python. We will first have a
look at how Python packages are packaged on Nix. Then, we will look at how you
can use development mode with your code.

#### Packaging a library

With Nix all packages are built by functions. The main function in Nix for
building Python libraries is `buildPythonPackage`. Let's see how we can build the
`toolz` package.

```nix
{ lib, buildPythonPackage, fetchPypi }:

buildPythonPackage rec {
  pname = "toolz";
  version = "0.7.4";

  src = fetchPypi {
    inherit pname version;
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  doCheck = false;

  meta = with lib; {
    homepage = https://github.com/pytoolz/toolz;
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}
```

What happens here? The function `buildPythonPackage` is called and as argum