diff options
28 files changed, 30732 insertions, 3498 deletions
diff --git a/doc/beam-users-guide.xml b/doc/beam-users-guide.xml new file mode 100644 index 000000000000..c1a4c90bc27c --- /dev/null +++ b/doc/beam-users-guide.xml @@ -0,0 +1,376 @@ +<chapter xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" + xml:id="users-guide-to-the-erlang-infrastructure"> + + <title>User's Guide to the Beam Infrastructure</title> + <section xml:id="beam-introduction"> + <title>Beam Languages (Erlang & Elixir) on Nix</title> + <para> + In this document and related Nix expressions we use the term + <emphasis>Beam</emphasis> to describe the environment. Beam is + the name of the Erlang Virtial Machine and, as far as we know, + from a packaging perspective all languages that run on Beam are + interchangable. The things that do change, like the build + system, are transperant to the users of the package. So we make + no distinction. + </para> + </section> +<section xml:id="build-tools"> + <title>Build Tools</title> + <section xml:id="build-tools-rebar3"> + <title>Rebar3</title> + <para> + By default Rebar3 wants to manage it's own dependencies. In the + normal non-Nix, this is perfectly acceptable. In the Nix world it + is not. To support this we have created two versions of rebar3, + <literal>rebar3</literal> and <literal>rebar3-open</literal>. The + <literal>rebar3</literal> version has been patched to remove the + ability to download anything from it. If you are not running it a + nix-shell or a nix-build then its probably not going to work for + you. <literal>rebar3-open</literal> is the normal, un-modified + rebar3. It should work exactly as would any other version of + rebar3. Any Erlang package should rely on + <literal>rebar3</literal> and thats really what you should be + using too. + </para> + </section> + <section xml:id="build-tools-other"> + <title>Mix & Erlang.mk</title> + <para> + Both Mix and Erlang.mk work exactly as you would expect. There + is a bootstrap process that needs to be run for both of + them. However, that is supported by the + <literal>buildMix</literal> and <literal>buildErlangMk</literal> derivations. + </para> + </section> + +</section> + +<section xml:id="how-to-install-beam-packages"> + <title>How to install Beam packages</title> + <para> + Beam packages are not registered in the top level simply because + they are not relevant to the vast majority of Nix users. They are + installable using the <literal>beamPackages</literal> attribute + set. + + You can list the avialable packages in the + <literal>beamPackages</literal> with the following command: + </para> + + <programlisting> +$ nix-env -f "<nixpkgs>" -qaP -A beamPackages +beamPackages.esqlite esqlite-0.2.1 +beamPackages.goldrush goldrush-0.1.7 +beamPackages.ibrowse ibrowse-4.2.2 +beamPackages.jiffy jiffy-0.14.5 +beamPackages.lager lager-3.0.2 +beamPackages.meck meck-0.8.3 +beamPackages.rebar3-pc pc-1.1.0 + </programlisting> + <para> + To install any of those packages into your profile, refer to them by + their attribute path (first column): + </para> + <programlisting> +$ nix-env -f "<nixpkgs>" -iA beamPackages.ibrowse + </programlisting> + <para> + The attribute path of any Beam packages corresponds to the name + of that particular package in Hex or its OTP Application/Release name. + </para> +</section> +<section xml:id="packaging-beam-applications"> + <title>Packaging Beam Applications</title> + <section xml:id="packaging-erlang-applications"> + <title>Erlang Applications</title> + <section xml:id="rebar3-packages"> + <title>Rebar3 Packages</title> + <para> + There is a Nix functional called + <literal>buildRebar3</literal>. We use this function to make a + derivation that understands how to build the rebar3 project. For + example, the epression we use to build the <link + xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link> + project follows. + </para> + <programlisting> + {stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }: + + buildRebar3 rec { + name = "hex2nix"; + version = "0.0.1"; + + src = fetchFromGitHub { + owner = "ericbmerritt"; + repo = "hex2nix"; + rev = "${version}"; + sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg"; + }; + + beamDeps = [ ibrowse jsx erlware_commons ]; + } + </programlisting> + <para> + The only visible difference between this derivation and + something like <literal>stdenv.mkDerivation</literal> is that we + have added <literal>erlangDeps</literal> to the derivation. If + you add your Beam dependencies here they will be correctly + handled by the system. + </para> + <para> + If your package needs to compile native code via Rebar's port + compilation mechenism. You should add <literal>compilePort = + true;</literal> to the derivation. + </para> + </section> + <section xml:id="erlang-mk-packages"> + <title>Erlang.mk Packages</title> + <para> + Erlang.mk functions almost identically to Rebar. The only real + difference is that <literal>buildErlangMk</literal> is called + instead of <literal>buildRebar3</literal> + </para> + <programlisting> + { buildErlangMk, fetchHex, cowlib, ranch }: + buildErlangMk { + name = "cowboy"; + version = "1.0.4"; + src = fetchHex { + pkg = "cowboy"; + version = "1.0.4"; + sha256 = + "6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac"; + }; + beamDeps = [ cowlib ranch ]; + + meta = { + description = ''Small, fast, modular HTTP server written in + Erlang.''; + license = stdenv.lib.licenses.isc; + homepage = "https://github.com/ninenines/cowboy"; + }; + } + </programlisting> + </section> + <section xml:id="mix-packages"> + <title>Mix Packages</title> + <para> + Mix functions almost identically to Rebar. The only real + difference is that <literal>buildMix</literal> is called + instead of <literal>buildRebar3</literal> + </para> + <programlisting> + { buildMix, fetchHex, plug, absinthe }: + buildMix { + name = "absinthe_plug"; + version = "1.0.0"; + src = fetchHex { + pkg = "absinthe_plug"; + version = "1.0.0"; + sha256 = + "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33"; + }; + beamDeps = [ plug absinthe]; + + meta = { + description = ''A plug for Absinthe, an experimental GraphQL + toolkit''; + license = stdenv.lib.licenses.bsd3; + homepage = "https://github.com/CargoSense/absinthe_plug"; + }; + } + </programlisting> + </section> + </section> +</section> +<section xml:id="how-to-develop"> + <title>How to develop</title> + <section xml:id="accessing-an-environment"> + <title>Accessing an Environment</title> + <para> + Often, all you want to do is be able to access a valid + environment that contains a specific package and its + dependencies. we can do that with the <literal>env</literal> + part of a derivation. For example, lets say we want to access an + erlang repl with ibrowse loaded up. We could do the following. + </para> + <programlisting> + ~/w/nixpkgs ❯❯❯ nix-shell -A beamPackages.ibrowse.env --run "erl" + Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] + + Eshell V7.0 (abort with ^G) + 1> m(ibrowse). + Module: ibrowse + MD5: 3b3e0137d0cbb28070146978a3392945 + Compiled: January 10 2016, 23:34 + Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam + Compiler options: [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"}, + debug_info,debug_info,nowarn_shadow_vars, + warn_unused_import,warn_unused_vars,warnings_as_errors, + {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}] + Exports: + add_config/1 send_req_direct/7 + all_trace_off/0 set_dest/3 + code_change/3 set_max_attempts/3 + get_config_value/1 set_max_pipeline_size/3 + get_config_value/2 set_max_sessions/3 + get_metrics/0 show_dest_status/0 + get_metrics/2 show_dest_status/1 + handle_call/3 show_dest_status/2 + handle_cast/2 spawn_link_worker_process/1 + handle_info/2 spawn_link_worker_process/2 + init/1 spawn_worker_process/1 + module_info/0 spawn_worker_process/2 + module_info/1 start/0 + rescan_config/0 start_link/0 + rescan_config/1 stop/0 + send_req/3 stop_worker_process/1 + send_req/4 stream_close/1 + send_req/5 stream_next/1 + send_req/6 terminate/2 + send_req_direct/4 trace_off/0 + send_req_direct/5 trace_off/2 + send_req_direct/6 trace_on/0 + trace_on/2 + ok + 2> + </programlisting> + <para> + Notice the <literal>-A beamPackages.ibrowse.env</literal>.That + is the key to this functionality. + </para> + </section> + <section xml:id="creating-a-shell"> + <title>Creating a Shell</title> + <para> + Getting access to an environment often isn't enough to do real + development. Many times we need to create a + <literal>shell.nix</literal> file and do our development inside + of the environment specified by that file. This file looks a lot + like the packageing described above. The main difference is that + <literal>src</literal> points to project root and we call the + package directly. + </para> + <programlisting> +{ pkgs ? import "<nixpkgs"> {} }: + +with pkgs; + +let + + f = { buildRebar3, ibrowse, jsx, erlware_commons }: + buildRebar3 { + name = "hex2nix"; + version = "0.1.0"; + src = ./.; + erlangDeps = [ ibrowse jsx erlware_commons ]; + }; + drv = beamPackages.callPackage f {}; + +in + drv + </programlisting> + <section xml:id="building-in-a-shell"> + <title>Building in a shell</title> + <para> + We can leveral the support of the Derivation, regardless of + which build Derivation is called by calling the commands themselv.s + </para> + <programlisting> +# ============================================================================= +# Variables +# ============================================================================= + +NIX_TEMPLATES := "$(CURDIR)/nix-templates" + +TARGET := "$(PREFIX)" + +PROJECT_NAME := thorndyke + +NIXPKGS=../nixpkgs +NIX_PATH=nixpkgs=$(NIXPKGS) +NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure +# ============================================================================= +# Rules +# ============================================================================= +.PHONY= all test clean repl shell build test analyze configure install \ + test-nix-install publish plt analyze + +all: build + +guard-%: + @ if [ "${${*}}" == "" ]; then \ + echo "Environment variable $* not set"; \ + exit 1; \ + fi + +clean: + rm -rf _build + rm -rf .cache + +repl: + $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'" + +shell: + $(NIX_SHELL) + +configure: + $(NIX_SHELL) --command 'eval "$$configurePhase"' + +build: configure + $(NIX_SHELL) --command 'eval "$$buildPhase"' + +install: + $(NIX_SHELL) --command 'eval "$$installPhase"' + +test: + $(NIX_SHELL) --command 'mix test --no-start --no-deps-check' + +plt: + $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check" + +analyze: build plt + $(NIX_SHELL) --run "mix dialyzer --no-compile" + + </programlisting> + <para> + If you add the <literal>shell.nix</literal> as described and + user rebar as follows things should simply work. Aside from the + <literal>test</literal>, <literal>plt</literal>, and + <literal>analyze</literal> the talks work just fine for all of + the build Derivations. + </para> + </section> +</section> +</section> +<section xml:id="generating-packages-from-hex-with-hex2nix"> + <title>Generating Packages from Hex with Hex2Nix</title> + <para> + Updating the Hex packages requires the use of the + <literal>hex2nix</literal> tool. Given the path to the Erlang + modules (usually + <literal>pkgs/development/erlang-modules</literal>). It will + happily dump a file called + <literal>hex-packages.nix</literal>. That file will contain all + the packages that use a recognized build system in Hex. However, + it can't know whether or not all those packages are buildable. + </para> + <para> + To make life easier for our users, it makes good sense to go + ahead and attempt to build all those packages and remove the + ones that don't build. To do that, simply run the command (in + the root of your <literal>nixpkgs</literal> repository). that follows. + </para> + <programlisting> +$ nix-build -A beamPackages + </programlisting> + <para> + That will build every package in + <literal>beamPackages</literal>. Then you can go through and + manually remove the ones that fail. Hopefully, someone will + improve <literal>hex2nix</literal> in the future to automate + that. + </para> +</section> +</chapter> diff --git a/doc/erlang-users-guide.xml b/doc/erlang-users-guide.xml deleted file mode 100644 index 074ae50b1c05..000000000000 --- a/doc/erlang-users-guide.xml +++ /dev/null @@ -1,305 +0,0 @@ -<chapter xmlns="http://docbook.org/ns/docbook" - xmlns:xlink="http://www.w3.org/1999/xlink" - xml:id="users-guide-to-the-erlang-infrastructure"> - -<title>User's Guide to the Erlang Infrastructure</title> -<section xml:id="build-tools"> - <title>Build Tools</title> - <para> - By default Rebar3 wants to manage it's own dependencies. In the - normal non-Nix, this is perfectly acceptable. In the Nix world it - is not. To support this we have created two versions of rebar3, - <literal>rebar3</literal> and <literal>rebar3-open</literal>. The - <literal>rebar3</literal> version has been patched to remove the - ability to download anything from it. If you are not running it a - nix-shell or a nix-build then its probably not going to work for - you. <literal>rebar3-open</literal> is the normal, un-modified - rebar3. It should work exactly as would any other version of - rebar3. Any Erlang package should rely on - <literal>rebar3</literal> and thats really what you should be - using too. - </para> -</section> - -<section xml:id="how-to-install-erlang-packages"> - <title>How to install Erlang packages</title> - <para> - Erlang packages are not registered in the top level simply because - they are not relevant to the vast majority of Nix users. They are - installable using the <literal>erlangPackages</literal> attribute set. - - You can list the avialable packages in the - <literal>erlangPackages</literal> with the following command: - </para> - - <programlisting> -$ nix-env -f "<nixpkgs>" -qaP -A erlangPackages -erlangPackages.esqlite esqlite-0.2.1 -erlangPackages.goldrush goldrush-0.1.7 -erlangPackages.ibrowse ibrowse-4.2.2 -erlangPackages.jiffy jiffy-0.14.5 -erlangPackages.lager lager-3.0.2 -erlangPackages.meck meck-0.8.3 -erlangPackages.rebar3-pc pc-1.1.0 - </programlisting> - <para> - To install any of those packages into your profile, refer to them by - their attribute path (first column): - </para> - <programlisting> -$ nix-env -f "<nixpkgs>" -iA erlangPackages.ibrowse - </programlisting> - <para> - The attribute path of any Erlang packages corresponds to the name - of that particular package in Hex or its OTP Application/Release name. - </para> -</section> -<section xml:id="packaging-erlang-applications"> - <title>Packaging Erlang Applications</title> - <section xml:id="rebar3-packages"> - <title>Rebar3 Packages</title> - <para> - There is a Nix functional called - <literal>buildRebar3</literal>. We use this function to make a - derivation that understands how to build the rebar3 project. For - example, the epression we use to build the <link - xlink:href="https://github.com/erlang-nix/hex2nix">hex2nix</link> - project follows. - </para> - <programlisting> -{stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }: - -buildRebar3 rec { - name = "hex2nix"; - version = "0.0.1"; - - src = fetchFromGitHub { - owner = "ericbmerritt"; - repo = "hex2nix"; - rev = "${version}"; - sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg"; - }; - - erlangDeps = [ ibrowse jsx erlware_commons ]; -} - </programlisting> - <para> - The only visible difference between this derivation and - something like <literal>stdenv.mkDerivation</literal> is that we - have added <literal>erlangDeps</literal> to the derivation. If - you add your Erlang dependencies here they will be correctly - handled by the system. - </para> - <para> - If your package needs to compile native code via Rebar's port - compilation mechenism. You should add <literal>compilePort = - true;</literal> to the derivation. - </para> - </section> - - <section xml:id="hex-packages"> - <title>Hex Packages</title> - <para> - Hex packages are based on Rebar packages. In fact, at the moment - we can only compile Hex packages that are buildable with - Rebar3. Packages that use Mix and other build systems are not - supported. That being said, we know a lot more about Hex and can - do more for you. - </para> - <programlisting> -{ buildHex }: - buildHex { - name = "esqlite"; - version = "0.2.1"; - sha256 = "1296fn1lz4lz4zqzn4dwc3flgkh0i6n4sydg501faabfbv8d3wkr"; - compilePort = true; -} - </programlisting> - <para> - For Hex packages you need to provide the name, the version, and - the Sha 256 digest of the package and use - <literal>buildHex</literal> to build it. Obviously, the package - needs to have already been published to Hex. - </para> - </section> -</section> -<section xml:id="how-to-develop"> - <title>How to develop</title> - <section xml:id="accessing-an-environment"> - <title>Accessing an Environment</title> - <para> - Often, all you want to do is be able to access a valid - environment that contains a specific package and its - dependencies. we can do that with the <literal>env</literal> - part of a derivation. For example, lets say we want to access an - erlang repl with ibrowse loaded up. We could do the following. - </para> - <programlisting> - ~/w/nixpkgs ❯❯❯ nix-shell -A erlangPackages.ibrowse.env --run "erl" - Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] - - Eshell V7.0 (abort with ^G) - 1> m(ibrowse). - Module: ibrowse - MD5: 3b3e0137d0cbb28070146978a3392945 - Compiled: January 10 2016, 23:34 - Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam - Compiler options: [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"}, - debug_info,debug_info,nowarn_shadow_vars, - warn_unused_import,warn_unused_vars,warnings_as_errors, - {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}] - Exports: - add_config/1 send_req_direct/7 - all_trace_off/0 set_dest/3 - code_change/3 set_max_attempts/3 - get_config_value/1 set_max_pipeline_size/3 - get_config_value/2 set_max_sessions/3 - get_metrics/0 show_dest_status/0 - get_metrics/2 show_dest_status/1 - handle_call/3 show_dest_status/2 - handle_cast/2 spawn_link_worker_process/1 - handle_info/2 spawn_link_worker_process/2 - init/1 spawn_worker_process/1 - module_info/0 spawn_worker_process/2 - module_info/1 start/0 - rescan_config/0 start_link/0 - rescan_config/1 stop/0 - send_req/3 stop_worker_process/1 - send_req/4 stream_close/1 - send_req/5 stream_next/1 - send_req/6 terminate/2 - send_req_direct/4 trace_off/0 - send_req_direct/5 trace_off/2 - send_req_direct/6 trace_on/0 - trace_on/2 - ok - 2> - </programlisting> - <para> - Notice the <literal>-A erlangPackages.ibrowse.env</literal>.That - is the key to this functionality. - </para> - </section> - <section xml:id="creating-a-shell"> - <title>Creating a Shell</title> - <para> - Getting access to an environment often isn't enough to do real - development. Many times we need to create a - <literal>shell.nix</literal> file and do our development inside - of the environment specified by that file. This file looks a lot - like the packageing described above. The main difference is that - <literal>src</literal> points to project root and we call the - package directly. - </para> - <programlisting> -{ pkgs ? import "<nixpkgs"> {} }: - -with pkgs; - -let - - f = { buildHex, ibrowse, jsx, erlware_commons }: - buildHex { - name = "hex2nix"; - version = "0.1.0"; - src = ./.; - erlangDeps = [ ibrowse jsx erlware_commons ]; - }; - drv = erlangPackages.callPackage f {}; - -in - drv - </programlisting> - <section xml:id="building-in-a-shell"> - <title>Building in a shell</title> - <para> - Unfortunatly for us users of Nix, Rebar isn't very cooperative - with us from the standpoint of building a hermetic - environment. When building the rebar3 support we had to do some - sneaky things to get it not to go out and pull packages on its - own. Also unfortunately, you have to do some of the same things - when building a project inside of a Nix shell. - - <orderedlist numeration="arabic"> - <listitem> - <para>Run <literal>rebar3-nix-bootstrap</literal> every time - dependencies change</para> - </listitem> - <listitem> - <para>Set Home to the current directory.</para> - </listitem> - </orderedlist> - - If you do these two things then Rebar will be happy with you. I - codify these into a makefile. Forunately, rebar3-nix-bootstrap - is idempotent and fairly quick. so you can run it as often as - you like. - </para> - <programlisting> -# ============================================================================= -# Rules -# ============================================================================= -.PHONY= all test clean repl shell build test analyze bootstrap - -all: test - -clean: - rm -rf _build - rm -rf .cache - -repl: - nix-shell --run "erl" - -shell: - nix-shell --run "bash" - -bootstrap: - nix-shell --pure --run "rebar3-nix-bootstrap" - -build: bootstrap - nix-shell --pure --run "HOME=$(CURDIR) rebar3 compile" - -analyze: bootstrap - nix-shell --pure --run "HOME=$(CURDIR) rebar3 do compile,dialyzer" - -test: bootstrap - nix-shell --pure --run "HOME=$(CURDIR) rebar3 do compile,dialyzer,eunit" - - </programlisting> - <para> - If you add the <literal>shell.nix</literal> as described and - user rebar as follows things should simply work. - </para> - </section> -</section> -</section> -<section xml:id="generating-packages-from-hex-with-hex2nix"> - <title>Generating Packages from Hex with Hex2Nix</title> - <para> - Updating the Hex packages requires the use of the - <literal>hex2nix</literal> tool. Given the path to the Erlang - modules (usually - <literal>pkgs/development/erlang-modules</literal>). It will - happily dump a file called - <literal>hex-packages.nix</literal>. That file will contain all - the packages that use a recognized build system in Hex. However, - it can't know whether or not all those packages are buildable. - </para> - <para> - To make life easier for our users, it makes good sense to go - ahead and attempt to build all those packages and remove the - ones that don't build. To do that, simply run the command (in - the root of your <literal>nixpkgs</literal> repository). that follows. - </para> - <programlisting> -$ nix-build -A erlangPackages - </programlisting> - <para> - That will build every package in - <literal>erlangPackages</literal>. Then you can go through and - manually remove the ones that fail. Hopefully, someone will - improve <literal>hex2nix</literal> in the future to automate - that. - </para> -</section> -</chapter> diff --git a/doc/manual.xml b/doc/manual.xml index de663fcd5b64..1045d0d4f815 100644 --- a/doc/manual.xml +++ b/doc/manual.xml @@ -21,7 +21,7 @@ <xi:include href="coding-conventions.xml" /> <xi:include href="submitting-changes.xml" /> <xi:include href="haskell-users-guide.xml" /> - <xi:include href="erlang-users-guide.xml" /> + <xi:include href="beam-users-guide.xml" /> <xi:include href="contributing.xml" /> </book> diff --git a/pkgs/development/beam-modules/build-erlang-mk.nix b/pkgs/development/beam-modules/build-erlang-mk.nix new file mode 100644 index 000000000000..8c2b72aa43cf --- /dev/null +++ b/pkgs/development/beam-modules/build-erlang-mk.nix @@ -0,0 +1,86 @@ +{ stdenv, writeText, erlang, perl, which, gitMinimal, wget }: + +{ name, version +, src +, setupHook ? null +, buildInputs ? [] +, beamDeps ? [] +, postPatch ? "" +, compilePorts ? false +, installPhase ? null +, meta ? {} |