haskell/stackage
1
0
Fork 0
mirror of https://github.com/commercialhaskell/stackage.git synced 2026-01-12 23:38:29 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #1 from fpco/master

Browse Source 
Merge upstream changes
This commit is contained in:
Daniel Díaz 2017-09-02 08:53:50 +02:00 committed by GitHub
commit db8928a037
13 changed files with 3095 additions and 1480 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.travis.yml
View File

@ -2,7 +2,7 @@
sudo: false
# Choose a lightweight base image; we provide our own build tools.
language: c
language: generic
# GHC depends on GMP. You can add other dependencies here as well.
addons:
@ -11,7 +11,7 @@ addons:
- libgmp-dev
env:
- GHCVER=8.0.1
- GHCVER=8.2.1
install:
# Download and unpack the stack executable

288
CURATORS.md
View File

@ -1,10 +1,10 @@
This is a collection of instructions covering the processes that the Stackage curators - the
guys who maintain the Stackage project itself - should be doing on a regular basis.
Originally this was handled largely by Michael Snoyman,
but now we are a team of 4 people handling requests weekly in rotation.
but now we are a team of 5 people handling requests weekly in rotation.
Curation activities are mostly automated, and do not take up a significant amount of time.
## Workflow
## Workflow overview
This section sketches out at a high level how the entire Stackage build/curation
process works:
@ -21,9 +21,34 @@ process works:
## Pull requests
The typical story on pull requests is: if Travis accepts it, and the author
only added packages under his/her own name, merge it. If the build later fails
(see below), then block the package until it's fixed.
The typical story on pull requests is: If Travis accepts it and the
author only added packages under his/her own name, merge it. If the
build later fails (see [Adding Debian packages]), then block the
package until it's fixed.
[Adding Debian packages]: https://github.com/fpco/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
If benchmarks, haddocks, or test suites fails at this point we
typically also block the package until these issues are fixed. This in
order to add packages with a clean slate.
Optionally we can check if packdeps says the package is up to date.
Visit http://packdeps.haskellers.com/feed?needle=<package-name>
Builds may fail because of unrelated bounds changes. If this happens,
first add any version bounds to get master into a passing state (see
"Fixing bounds issues"), then re-run the travis build.
A common issue is that authors submit newly uploaded packages, it can
take up to an hour before this has synced across the stack
infrastructure. You can usually compare the versions of the package in
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
to what's on hackage to see if this is the case. Wait an hour and
re-run the pull request.
Tests also commonly fail due to missing test files, and sometimes due
to doctest limitations. You can point the maintainer to
https://github.com/bergmark/blog/blob/master/2016/package-faq.md
## Fixing bounds issues
@ -33,24 +58,146 @@ issue on the Stackage repo about the problem, and modifying the
build-constraints.yaml file to work around it in one of the ways below. Be sure
to refer to the issue for workarounds added to that file.
* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
### Temporary upper bounds
Most common technique, just prevent a new version of a library from
being included immediately. This also applies to when only benchmarks
and tests are affected.
* Copy the stackage-curator output and create a new issue, see e.g
https://github.com/fpco/stackage/issues/2108
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
```yaml
"Stackage upper bounds":
# https://github.com/fpco/stackage/issues/2108
- pipes < 4.3.0
```
* Commit (message e.g. "Upper bound for #2108")
* Optionally: Verify with `stackage-curator check` locally
* Push
* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
Sometimes releases for different packages are tightly coupled. Then it
can make sense to combine them into one issue, as in
https://github.com/fpco/stackage/issues/2143.
If a dependency that is not explicitly in stackage is causing test or
benchmark failures you can skip or expect them to fail (see "Skipping
tests and benchmarks" and "Expecting test/benchmark/haddock
failures"). Bonus points for reporting this upstream to that packages'
maintainer.
### Lifting upper bounds
You can try this when you notice that a package has been updated. You
can also periodically try to lift bounds (I think it's good to do this
at the start of your week /@bergmark)
If not all packages have been updated check if any of them are missing
from the original issue and if so add a new comment mentioning them. A
new package may appear if its dependencies were part of this issue but
have been updated since the last time we checked. We want to give
these new packages ample time to be upgraded.
If stackage-curator is happy commit the change ("Remove upper bounds
and close #X"). After doing this the next nightly build may fail
because some packages didn't have an upper bound in place, but
compilation failed. In this case revert the previous commit so any
disabled packages are enabled again, re-open the issue, and add a new
comment with the failing packages. This is to give all maintainers
enough time to upgrade for this case as well.
### Amending upper bounds
With the `pipes` example above there was later a new release of
`pipes-safe` that required the **newer** version of `pipes`. You can
add that package to the same upper bounds section,
(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
as we want to lift them simultaneously.
### Skipping tests and benchmarks
Sometimes tests and benchmark dependencies are forgotten or not cared
for. To disable compilation for them add them to `skipped-tests` or
`skipped-benchmarks`. If a package is added to these sections they
won't be compiled, and their dependencies won't be taken into account.
There are sub sections under these headers that is used to group types
of failures together, and also to document what type of failures
exist.
### Expecting test/benchmark/haddock failures
The difference from the `skipped` sections is that items listed here
are compiled and their dependencies are taken into account. These
sections also have sub sections with groups and descriptions.
One big category of test suites in this section are those requiring
running services. We don't want to run those, but we do want to check
dependencies and compile them.
If there are no version bounds that would fix the issue or if you
can't figure it out, file it
(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
maintainer for help.
### Waiting for new releases
Sometimes there is a failure reported on a (now possibly closed) issue
on an external tracker. If an issue gets resolved but there is no
hackage release yet we'd like to get notified when it's uploaded.
Add the package with its current version to the
`tell-me-when-its-released` section. This will cause the build to stop
when the new version is out.
### Excluding packages
In an extreme case of a non-responsive maintainer, you can remove the
package entirely from Stackage. We try to avoid that whenever
possible.
This typically happens when we move to a new major GHC release or when
there are only a few packages waiting for updates on an upper bounds
issue.
Comment out the offending packages from the "packages" section and add
a comment saying why it was disabled:
```
# - swagger # bounds: aeson 1.0
```
If this causes reverse dependencies to be disabled we should notify
the maintainers of those packages.
## Updating the content of the Docker image used for building
### Adding Debian packages for required system tools or libraries
Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
since new packages are not added to the current lts release.
After you've committed those changes, merging them into the `nightly` branch should
trigger a DockerHub build. Simply run:
Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to trusty which is the same version as the server).
```bash
$ git checkout nightly
$ git merge master
$ git push
```
Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
### Upgrading GHC version
The Dockerfile contains information on which GHC versions should be used. You
can modify it and push it to Github to trigger a DockerHub build. The master
can modify it and push it to Github to trigger a DockerHub build. The nightly
branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the
major version number (e.g., lts3 for lts-3.\*).
@ -76,14 +223,34 @@ rm -r nightly/work/builds/nightly/
```
This should also be done when moving the Nightly docker image to a new version of Ubuntu.
If you're impatient and would like to build the Docker image on the
build server instead of waiting for Docker Hub, you can run the
following command:
```
DIR=$(mktemp -d)
(cd $DIR \
&& git clone https://github.com/fpco/stackage \
&& cd stackage \
&& docker build --tag snoyberg/stackage:nightly .)
rm -rf $DIR
```
Note that we do a clean clone of the `stackage` repo instead of using
the existing checkout because of how `docker build` works: it will
send the entire local directory contents as context to the Docker
daemon, which in the case of the build tree is a _lot_ of content. (We
can discuss the wisdom&mdash;or lack thereof&mdash;of Docker's
approach separately.)
## stackage-build server
You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
```
Host stackage-build
User ubuntu
Hostname ec2-52-5-20-252.compute-1.amazonaws.com
User curators
Hostname build.stackage.org
```
### Running the build script
@ -103,7 +270,7 @@ we're just not there yet.
/opt/stackage-build/stackage/automated/build.sh lts-3.0
```
Recommended: run these from inside a `screen` session. If you get version bound
Recommended: run these from inside a `tmux` session. If you get version bound
problems on nightly or LTS major, you need to fix build-constraints.yaml (see
info above). For an LTS minor bump, you'll typically want to use the
`CONSTRAINTS` environment variable, e.g.:
@ -123,8 +290,14 @@ If a build fails for bounds reasons, see all of the advice above. If the code
itself doesn't build, or tests fail, open up an issue and then either put in a
version bound to avoid that version or something else. It's difficult to give
universal advice on how to solve things, since each situation is unique. Let's
develop this advice over time. For now: if you're not sure, ask Michael for
guidance.
develop this advice over time. For now: if you're not sure, ask for guidance.
__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
build plan, you can set the environment variable `NOPLAN=1`. This is
useful for such cases as an intermittent test failure, out of memory
condition, or manually tweaking the plan file.
Note LTS builds inherit the current Hackage data (stack updated for Nigthly) to avoid excess extra rebuilding.
### Timing
@ -138,9 +311,82 @@ LTS minor bumps typically are run on Sundays.
### Website sync debugging (and other out of disk space errors)
* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
### Wiping the cache
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
You can wipe the nightly cache and rebuild everything by doing
`rm -rf /var/stackage/stackage/automated/nightly`.
Replace nightly with `lts7` to wipe the LTS 7 cache.
### Force a single package rebuild
You can force a single package to rebuild by deleting its "previous result"
file, e.g.:
```
$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
```
## Local curator setup
We do not run the full stackage build locally as that might take too
much time. However, some steps on the other hand are much faster to do
yourself, e.g. verifying constraints without building anything.
To get started, install `stackage-curator` via Git, or [the Linux binary]:
```
$ git clone git@github.com:fpco/stackage-curator.git
$ cd stackage-curator && stack install
```
It is a good idea to upgrade `stackage-curator` at the start of your week.
Then, clone the stackage repo, get the latest packages and run dependency
resolution:
```
$ git clone git@github.com:fpco/stackage.git
$ stack update && stackage-curator check
```
This can be used to make sure all version bounds are in place, including for
test suites and benchmarks, to check whether bounds can be lifted, and to get
[tell-me-when-its-released] notifications.
`stackage-curator` does not build anything, so you wont see any compilation
errors for builds, tests and benchmarks.
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
## Adding new curators
1. Add public ssh key to `~/.ssh/authorized_keys` on build server
2. Add to fpco/stackage project.
## Dealing with a new GHC release
As mentioned in the [GHC upgrade note], the major impact of a new GHC release
is on the packages that are causing upper bounds to be put in place. In order
to minimise out-of-date breakage and allow maintainers to have a solid chance
of getting their packages into the newest LTS, we try to do the following:
Make an early announcement (in the form of a blog post, typically) of the new
GHC release on the nightly build and the planned deadline for the new LTS release.
Make it clear, that in the time coming up to this, we hope package maintainers
will upgrade their packages to allow for the new GHC release.
We prefer to prune packages causing upper bounds constraints **after** the LTS
release to allow the maximum amount of packages to get into the newest LTS.
You will almost always need to update the Win32 package version listed in the
build-constraints.yaml file.
After the first LTS release, the package pruning process may begin in the
nightly build in order to move forward with getting the latest versions of
packages compatible with the new GHC release.
[GHC upgrade note]: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version

8
Dockerfile
View File

@ -1,8 +1,12 @@
FROM ubuntu:16.04
FROM fpco/pid1:16.04
ENV HOME /home/stackage
ENV LANG en_US.UTF-8
ENV PATH /opt/ghc/8.0.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
# NOTE: also update debian-bootstrap.sh when cuda version changes
ENV PATH /usr/local/cuda-8.0/bin:/opt/ghc/8.2.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV CUDA_PATH /usr/local/cuda-8.0
ENV LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/nvvm/lib64
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh

72
MAINTAINERS.md
View File

@ -2,58 +2,89 @@ This project is built around the concept of maintainers taking responsibility fo
The idea behind Stackage is that, if all packages work with the newest versions of dependencies, we avoid dependency hell. Specifically, we aim for:
* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
* All packages are compatible with the newest versions of all dependencies.
* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
* All packages are compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
* All packages in a snapshot are compatible with the versions of libraries that ship with the GHC used in the snapshot ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
Packages in Stackage are not patched: all package changes occur upstream in Hackage.
## Adding a package
Anyone can add a package to Stackage, but it's highly encouraged that the actual package maintainer is also the Stackage maintainer. If that is not the case, you should drop the package maintainer a note first.
Anyone can add any package to Stackage but you may only add packages under your own name. It's highly encouraged that the actual package maintainer is also the Stackage maintainer, if that is not the case you should drop the package maintainer a note first.
To add your package, first fork this repository.
In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`.
To add a set of packages, you would add:
"My Name myemail@example.com @mygithubuser":
"My Name <myemail@example.com> @mygithubuser":
- package1
- package2
- package3
Note that the `master` branch is used for Stackage Nightly (not the `nightly` branch, which is used for the nightly docker builder imagine).
After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
```
$ cabal update
$ ghc --version # Should give v8.0.2
$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
$ cd PACKAGE-VERSION
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
$ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
$ cabal install --enable-tests --enable-benchmarks --allow-newer
$ cabal test
```
**NB** Please use commit messages like "add foo-bar" or "add johndev's packages"
(`build-constraints.yaml` is the most frequently changed file in this git repo
so commit messages like "update build-constraints.yaml" are not helpful).
**NB2** There can be a delay of up to an hour before package versions newly
uploaded to Hackage appear to our build server. If you just uploaded a package
to Hackage that you're trying to get included, we recommend waiting an hour
before opening the PR.
**NB2** There can be a delay of up to an hour before package versions
newly uploaded to Hackage appear to our build server. If you just
uploaded a package to Hackage that you're trying to get included, we
recommend waiting an hour before opening the PR. You can verify this
by making sure the latest version is listed at
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/.
## Uploading a new package version
## Uploading a new package
When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
When a new version of a package is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and raise GitHub issue tickets to resolve the issue.
If the new version doesn't compile then the package author should quickly (within 1 week) upload a fixed version.
If the new version doesn't compile then the package author should upload a fixed version.
If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite.
## Following dependency upgrades
If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is highly recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
**If restrictive version bounds are the only problem** then you must quickly (within 1 week) upload a new version with relaxed version bounds. Note that unlike the PVP, Stackage does not require upper bounds.
If a package is not modified in a timely manner, it may be temporarily
removed from Stackage by the curator team, at which point it is your
responsibility to add it back in via a new pull request. We typically
use fairly long windows on this, but at a minimum:
**If the new dependency causes breaking changes** then all package authors should quickly assess the likely impact on their package (within 1 week) and then produce a new compatible version. The expected timeline for new versions varies between 1 week and 1 month, depending on the significance of the change, and thus the work required to produce those new versions.
* If restrictive version bounds are the only problem, we will give at
least a week to respond.
* If there are real breaking changes, the curator team will retain
more discretion on how long a window to give before dropping
packages. Historically, this has usually waited until the cutting of
a new Long Term Support (LTS) major version.
**NOTE** Previously, this maintainer agreement put a time limit on
maintainers, requiring a certain level of responsiveness for
modifications to be made. We have explicitly removed that: anyone is
free to add a package to Stackage regardless of responsiveness
guarantees. However, as stated above, we may elect to temporarily
remove a package if it is not updated in a timely manner.
## Failing to meet the time limits
Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack.
The time limits are intended to stop people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for long.
The soft time limits are intended to prevent people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for longer periods.
## Upgrading to a new GHC version
@ -74,10 +105,11 @@ following the above steps, you can get your package into the next major LTS
Haskell release.
If you would like to get your package added to an existing LTS Haskell major
release (e.g., if `lts-3.21` is out, you would want your package to appear in
`lts-3.22`), please do the following in addition to the steps above:
release (e.g., if `lts-8.9` is out, you would want your package to appear in
`lts-8.10`), please do the following in addition to the steps above:
* Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
* Specify the LTS major version you would like your package to go into (e.g., lts-3)
* Specify the LTS major version you would like your package to go into (e.g., lts-8)
* Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
* Be patient! The LTS releases are by their nature more conservative than nightly, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue quickly, but there may be some delay.
* Be patient! The LTS releases are less frequent than Nightly and by their nature more conservative, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue as soon as possible, but it may take some time.
* To add a package to more than one LTS version please file separate tickets for each major LTS release, since they will typically be built and added at different times.

84
README.md
View File

@ -17,11 +17,11 @@ Add your package
We welcome all packages, provided:
* The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
* The package is compatible with the newest versions of all dependencies.
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
* The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
* The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
Full details on how to add a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package).
Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package).
__NOTE__: There is an approximate 30 minute delay between a package uploading
to Hackage and being available to the Travis build script to check upper
@ -57,34 +57,13 @@ flow](https://github.com/fpco/stackage/blob/master/DATA-FLOW.md).
Build the package set
---------------------
Generally, building the package set should be done only by the Stackage build
machine by the Stackage curation team. If you're interested in trying this
yourself, please check out [the curator
guide](https://github.com/fpco/stackage/blob/master/CURATORS.md), though be
aware that this is not a recommended practice and there likely will be problems
you will need to debug yourself.
### Docker
Note: This method has been disabled for now, but may be enabled again in the future.
If you'd like to check a build plan, or perform an entire build, without
specially configuring your system, Docker may be a good approach. To check if
some modifications to `build-constraints.yaml` are valid, try the following:
1. Create a local clone of the `stackage` repo
2. Make modifications to your local `build-constraints.yaml`
3. Inside the `stackage` working directory, run the following:
```
$ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage check'
```
Similarly, if you'd like to perform an entire build, you can replace the last step with:
```
$ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage nightly --skip-upload'
```
Generally only the stackage build server run by the stackage curator
team and people intrested in incorporating stackage snapshots into an
OS distribution need to build the entire package set. If you're
interested in trying this yourself, please check out
[the curator guide](https://github.com/fpco/stackage/blob/master/CURATORS.md),
though be aware that this is not a recommended practice and there
likely will be problems you will need to debug yourself.
## Processing
@ -105,3 +84,46 @@ The following describes at a high level the series of steps for processing
1. Load up most recent build plan
2. Convert build plan into constraints for next build
3. Continue from step (3) above
Frequently Asked Questions
--------------------------
__Why does Stackage have an older version of a package than Hackage?__
There are a number of answers to this question:
* Simplest reason: how old of a Stackage snapshot are you using? Once a
snapshot is created, it's frozen for all time. So if you use
nightly-2016-01-01, by the time you get to 2018, it will be pretty dated.
* If you're using an LTS snapshot: we lock down major versions when
first creating an LTS run, so subsequent minor versions will not get
new versions necessary. For example, if LTS 6.0 has `foo` version
1.2.3, and the author immediately thereafter releases a version
1.3.0 and never releases another 1.2.\* version, you'll never get
another update in the LTS 6 line
* Sometimes we have upper bounds in place because other packages have
problems with newer versions of dependencies. Open up the
[build-constraints file](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
and search for "Stackage upper bounds"
* Wired-in packages - those that ship with GHC and cannot be upgraded,
and packages depending on them - are fixed to GHC versions. Common
examples of this are containers and transformers. There's a lot more
information on this in
[an FP Complete blog post](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)
__How long do you maintain an LTS build?__
We only guarantee that we will maintain a single LTS major version at
a time, and that it will be maintained for at least three months. This
is the
[originally proposed support window](https://www.fpcomplete.com/blog/2014/12/backporting-bug-fixes),
and hasn't changed since then.
That said, we do maintain the capability to keep multiple LTS runs
operational in parallel, and with LTS 6 and 7 in fact did so. We
aren't changing our guarantees yet on longevity of a release, but are
trying to push out the bounds a bit farther.
__What time are Stackage snapshots published?__
Stackage Nightly and LTS are not released at a fixed time of day, they get pushed to stackage.org (and the metadata to the stackage-nightly and stackage-lts github repos) when their builds finish on the Stackage build server and the latest built haddocks have been synced over. This time varies greatly depending on build times for package updates, bounds breakage, problems with new packages being added and other build issues, etc. There are days when a release does not happen. LTS releases tend to happen over the weekend or early in the week.

2
automated/.gitignore vendored
View File

@ -8,3 +8,5 @@
/lts-*/
/stackage-curator/
/stackage-update/
/aws.sh
/crondir/

42
automated/build.sh
View File

@ -1,10 +1,12 @@
#!/usr/bin/env bash
set -eux
set -eu +x
ROOT=$(cd $(dirname $0) ; pwd)
TARGET=$1
source aws.sh
# For nightly-YYYY-MM-DD, tag should be nightly
# For lts-X.Y, tag should be ltsX
SHORTNAME=$(echo $TARGET | cut -d- -f 1)
@ -75,12 +77,13 @@ rm -f stackage-curator stackage-curator.bz2
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
bunzip2 stackage-curator.bz2
chmod +x stackage-curator
./stackage-curator --version
)
ARGS_COMMON="--rm -v $WORKDIR:/home/stackage/work -w /home/stackage/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR:/home/stackage/bin:ro"
ARGS_PREBUILD="$ARGS_COMMON -u $USER -v $CABAL_DIR:/home/stackage/.cabal -v $STACK_DIR:/home/stackage/.stack -v $GHC_DIR:/home/stackage/.ghc -v $DOT_STACKAGE_DIR:/home/stackage/.stackage"
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:/home/stackage/.cabal:ro -v $STACK_DIR:/home/stackage/.stack:ro -v $GHC_DIR:/home/stackage/.ghc:ro"
ARGS_UPLOAD="$ARGS_COMMON -u $USER -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY -e AWS_SECRET_KEY=$AWS_SECRET_KEY -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:/home/stackage/.stackage -v $SSH_DIR:/home/ubuntu/.ssh:ro -v $GITCONFIG:/home/stackage/.gitconfig:ro -v $CABAL_DIR:/home/stackage/.cabal:ro -v $STACK_DIR:/home/stackage/.stack:ro"
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
ARGS_PREBUILD="$ARGS_COMMON -u $USER -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
ARGS_UPLOAD="$ARGS_COMMON -u $USER -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
# Make sure we actually need this snapshot. We only check this for LTS releases
# since, for nightlies, we'd like to run builds even if they are unnecessary to
@ -88,32 +91,44 @@ ARGS_UPLOAD="$ARGS_COMMON -u $USER -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY -e AWS_ACCE
# below for why this is safe.)
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator check-target-available --target $TARGET"
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
fi
# Get latest stack
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
# Do all of the pre-build actions:
# Determine the new build plan unless NOPLAN is set
#
# * Update the package index
# * Update the package index (unless LTS)
# * Create a new plan
if [ "${NOPLAN:-}x" = "x" ]
then
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
fi
fi
# Do the rest of the pre-build actions:
#
# * Check that the plan is valid
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
# * Do a single unpack to create the package index cache (again due to directory perms)
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "/home/stackage/bin/stack update && stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-} && stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && /home/stackage/bin/stack unpack random"
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
# Now do the actual build. We need to first set the owner of the home directory
# correctly, so we run the command as root, change owner, and then use sudo to
# switch back to the current user
docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER /home/stackage && sudo -E -u $USER env \"PATH=\$PATH:/home/stackage/bin\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs 4 --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
# Make sure we actually need this snapshot. We used to perform this check
# exclusively before building. Now we perform it after as well for the case of
# nightly, where we don't perform this check beforehand. This is also slightly
# safer, in case someone else already uploaded a specific snapshot while we
# were building.
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator check-target-available --target $TARGET"
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
# Successful build, so we need to:
#
@ -121,4 +136,7 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator check-target-avail
# * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
# * Upload the new plan .yaml file to the appropriate Github repo
# * Register as a new Hackage distro
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
echo -n "Completed at "
date

24
automated/cron.sh Executable file
View File

@ -0,0 +1,24 @@
#!/usr/bin/env bash
set -eux
cd "$(dirname "${BASH_SOURCE[0]}")"
CRONDIR=$(pwd)/crondir
mkdir -p $CRONDIR
source aws.sh
IMAGE=fpco/stackage-server-prod:latest
docker pull $IMAGE
stack update
date
echo "Running stackage-server-cron..."
docker run --rm \
-v $CRONDIR:/home/ubuntu \
-v $HOME/.stack/indices:/home/ubuntu/.stack/indices:ro \
--workdir /home/ubuntu \
-p 17834:17834 \
$IMAGE \
bash -c "useradd $(whoami) -u $(id -u); sudo -u $(whoami) env HOME=/home/ubuntu AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY bash -c '/usr/local/bin/stackage-server-cron 2>&1 | tee -a /home/ubuntu/stackage-server-cron.log'"

13
automated/run-nightly.sh
View File

@ -1,15 +1,10 @@
#!/usr/bin/env bash
cd "$(dirname "${BASH_SOURCE[0]}")"
while true; do
/opt/stackage-build/stackage/automated/build.sh nightly-$(date -u +%F)
./build.sh nightly-$(date -u +%F)
date
echo
echo "Running stackage-server-cron..."
echo "('tail -f /home/ubuntu/stackage-server-cron.log' to watch)"
/home/ubuntu/stackage-server-cron.sh >> /home/ubuntu/stackage-server-cron.log 2>&1
echo "done."
date
sleep 30m
echo
done

44
become-a-curator.md Normal file
View File

@ -0,0 +1,44 @@
## We are no longer accepting applications, please wait until next time
We are looking to expand the stackage curator team!
As a stackage curator you will help the community by curating and
publishing the widely used nightly and LTS snapshots: Collections of
packages that are guaranteed to play well together. All curators are
volunteers.
This is a great opportunity to contribute to the haskell community and get
familiar with the tools of the trade! You will also be in regular
contact with experienced community members.
We don't have any prerequisites, but familiarity with any of Haskell,
Cabal, Stack, Linux, Git, and Docker is a plus.
We work in shifts of one week, which means you will be on duty every
6th week. Our suggestion is to pair you up with another curator to
help out in the beginning.
The workload is usually about 30 minutes per day, whenever you have
time. The work consists of keeping the builds running on the build
server, filing issues for packages needing updates when new releases
occur, and closing resolved issues. The LTS snapshots are prepared
once a week. All communication with package maintainers happen on
github.
We also have a curator slack channel where we help each other out.
The current curator team consists of:
* Adam Bergmark
* Dan Burton
* Jens Petersen
* Luke Murphy
* Michael Snoyman
We onboarded Luke as the newest member a few months ago and this
helped us iron out and document the process further.
You can read the curator documentation here: https://github.com/fpco/stackage/blob/master/CURATORS.md
To apply, please fill in this form: [REDACTED]
If you applied last time, we encourage you to do so again!

3854
build-constraints.yaml
View File

File diff suppressed because it is too large Load Diff

2
check
View File

@ -3,4 +3,4 @@
# Convenience script for checking constraints locally
cd `dirname $0`
exec stack exec --resolver ghc-8.0.1 stackage-curator check
exec stack exec --resolver ghc-8.2.1 stackage-curator check

138
debian-bootstrap.sh
View File

@ -13,7 +13,6 @@
set -exu
mkdir /home/stackage -p
locale-gen en_US.UTF-8
export DEBIAN_FRONTEND=noninteractive
apt-get update
@ -21,40 +20,46 @@ apt-get install -y software-properties-common
add-apt-repository ppa:hvr/ghc -y
add-apt-repository -y ppa:marutter/rrutter
# not sure what this was needed for
#add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy main'
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
# Set the GHC version
GHCVER=8.0.1
# Get Stack
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 575159689BEFB442
echo 'deb http://download.fpcomplete.com/ubuntu xenial main'|tee /etc/apt/sources.list.d/fpco.list
GHCVER=8.2.1
apt-get update
apt-get install -y \
build-essential \
ghc-$GHCVER \
ghc-$GHCVER-htmldocs \
hscolour \
sudo \
curl \
freeglut3-dev \
fsharp \
ghc-$GHCVER \
ghc-$GHCVER-dyn \
ghc-$GHCVER-htmldocs \
ghc-$GHCVER-prof \
git \
gradle \
hscolour \
libadns1-dev \
libaio1 \
libalut-dev \
libasound2-dev \
libblas-dev \
libbz2-dev \
libcairo2-dev \
libclang-3.9-dev \
libcurl4-openssl-dev \
libcwiid-dev \
libdevil-dev \
libedit-dev \
libedit2 \
libfftw3-dev \
libflac-dev \
libfreenect-dev \
libgd2-xpm-dev \
libgeoip-dev \
libgirepository1.0-dev \
libglfw3-dev \
libglib2.0-dev \
libglu1-mesa-dev \
libgmp3-dev \
@ -65,25 +70,39 @@ apt-get install -y \
libgtk2.0-dev \
libgtksourceview-3.0-dev \
libhidapi-dev \
libi2c-dev \
libicu-dev \
libimlib2-dev \
libjack-jackd2-dev \
libjudy-dev \
liblapack-dev \
libleveldb-dev \
liblmdb-dev \
liblzma-dev \
libmagic-dev \
libmagickcore-dev \
libmagickwand-dev \
libmarkdown2-dev \
libmono-2.0-dev \
libmp3lame-dev \
libmpfr-dev \
libmysqlclient-dev \
libncurses-dev \
libnfc-dev \
libnotify-dev \
libopenal-dev \
libpango1.0-dev \
libpcap0.8-dev \
libpq-dev \
libsdl1.2-dev \
libsdl2-dev \
libsdl2-gfx-dev \
libsdl2-image-dev \
libsdl2-mixer-dev \
libsdl2-ttf-dev \
libsnappy-dev \
libsndfile1-dev \
libsox-dev \
libsqlite3-dev \
libssl-dev \
libsystemd-dev \
@ -99,21 +118,33 @@ apt-get install -y \
libyaml-dev \
libzip-dev \
libzmq3-dev \
llvm \
llvm-3.9 \
locales \
m4 \
minisat \
mono-mcs \
nettle-dev \
nodejs \
npm \
openjdk-8-jdk \
python-mpltoolkits.basemap \
python3-matplotlib \
python3-numpy \
python3-pip \
r-base \
r-base-dev \
ruby-dev \
stack \
sudo \
wget \
xclip \
z3 \
zip \
zlib1g-dev
locale-gen en_US.UTF-8
curl -sSL https://get.haskellstack.org/ | sh
# Put documentation where we expect it
mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
@ -122,3 +153,80 @@ mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
# faster anyways and uses less RAM.
update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.gold" 20
update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
# GHC requires a specific LLVM version on the system PATH for its LLVM backend.
# This version is tracked here:
# https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
#
# GHC 8.2 requires LLVM 3.9 tools (specifically, llc-3.9 and opt-3.9).
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.9" 50
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.9" 50
# install ocilib dependencies then build and install ocilib
cd /tmp \
&& wget https://storage.googleapis.com/oracle.fpinsight.com/instantClient/oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
&& dpkg -i oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
&& rm -f oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
&& wget https://storage.googleapis.com/oracle.fpinsight.com/instantClient/oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
&& dpkg -i oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
&& rm -f oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
&& wget https://github.com/vrogier/ocilib/archive/v4.3.2.tar.gz \
&& tar xvf v4.3.2.tar.gz \
&& cd /tmp/ocilib-4.3.2 \
&& ./configure --with-oracle-import=linkage \
--with-oracle-charset=ansi \
--with-oracle-headers-path=/usr/include/oracle/12.1/client64 \
--with-oracle-lib-path=/usr/lib/oracle/12.1/client64/lib \
&& make \
&& make install \
&& cd \
&& rm -rf /tmp/ocilib-4.3.2 \
&& echo "/usr/local/lib" > /etc/ld.so.conf.d/usr-local.conf \
&& echo "/usr/lib/oracle/12.1/client64/lib" > /etc/ld.so.conf.d/oracle-client.conf \
&& ldconfig
# Add JDK to system paths.
echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
&& ldconfig
# llvm-4.0 for llvm-hs (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main" \
&& apt-get update \
&& apt-get install -y llvm-4.0
# Install version 3 of the protobuf compiler. (The `protobuf-compiler` package only
# supports version 2.)
curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip \
&& sudo unzip -o protoc-3.3.0-linux-x86_64.zip -d /usr bin/protoc \
&& rm -f protoc-3.3.0-linux-x84_64.zip
# Install the TensorFlow C API.
curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.1.0.tar.gz > libtensorflow.tar.gz \
&& sudo tar zxf libtensorflow.tar.gz -C /usr \
&& rm libtensorflow.tar.gz \
&& ldconfig
# NOTE: also update Dockerfile when cuda version changes
# Install CUDA toolkit
# The current version can be found at: https://developer.nvidia.com/cuda-downloads
CUDA_PKG=8.0.61-1 # update this on new version
CUDA_VER=${CUDA_PKG:0:3}
CUDA_APT=${CUDA_VER/./-}
pushd /tmp \
&& wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& dpkg -i cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& apt-get update -qq \
&& apt-get install -y cuda-drivers cuda-core-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} cuda-cufft-dev-${CUDA_APT} cuda-cublas-dev-${CUDA_APT} cuda-cusparse-dev-${CUDA_APT} cuda-cusolver-dev-${CUDA_APT} \
&& rm cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& export CUDA_PATH=/usr/local/cuda-${CUDA_VER} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH+x} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${LD_LIBRARY_PATH} \
&& export PATH=${CUDA_PATH}/bin:${PATH} \
&& popd
## non-free repo for mediabus-fdk-aac
#apt-add-repository multiverse \
# && apt-get update \
# && apt-get install -y libfdk-aac-dev