mirror of
https://github.com/commercialhaskell/stackage.git
synced 2026-01-12 23:38:29 +01:00
Merge branch 'master' into master
This commit is contained in:
Pascal Hartig
GitHub
commit
6f09c87d30
9
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
9
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
@ -0,0 +1,9 @@
|
||||
Checklist:
|
||||
- [ ] Meaningful commit message - please not `Update build-constraints.yml`
|
||||
- [ ] At least 30 minutes have passed since Hackage upload
|
||||
- [ ] On your own machine, in a new directory, you have successfully run the following set of commands (replace `$package` with the name of the package that is submitted, `$version` is the version of the package you want to get into Stackage):
|
||||
|
||||
stack unpack $package
|
||||
cd $package-$version
|
||||
stack init --resolver nightly
|
||||
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
|
||||
@ -2,7 +2,7 @@
|
||||
sudo: false
|
||||
|
||||
# Choose a lightweight base image; we provide our own build tools.
|
||||
language: c
|
||||
language: generic
|
||||
|
||||
# GHC depends on GMP. You can add other dependencies here as well.
|
||||
addons:
|
||||
@ -11,7 +11,7 @@ addons:
|
||||
- libgmp-dev
|
||||
|
||||
env:
|
||||
- GHCVER=8.0.1
|
||||
- GHCVER=8.4.1
|
||||
|
||||
install:
|
||||
# Download and unpack the stack executable
|
||||
|
||||
287
CURATORS.md
287
CURATORS.md
@ -1,10 +1,18 @@
|
||||
This is a collection of instructions covering the processes that the Stackage curators - the
|
||||
guys who maintain the Stackage project itself - should be doing on a regular basis.
|
||||
Originally this was handled largely by Michael Snoyman,
|
||||
but now we are a team of 4 people handling requests weekly in rotation.
|
||||
team who maintain the Stackage project itself - should be doing on a regular basis.
|
||||
Curation activities are mostly automated, and do not take up a significant amount of time.
|
||||
The following is the current list of curators, in alphabetical order:
|
||||
|
||||
## Workflow
|
||||
* Adam Bergmark (@bergmark)
|
||||
* Alexey Zabelin (@alexeyzab)
|
||||
* Chris Dornan (@cdornan)
|
||||
* Dan Burton (@danburton)
|
||||
* Jens Petersen (@juhp)
|
||||
* Joe Kachmar (@jkachmar)
|
||||
* Michael Snoyman (@snoyberg)
|
||||
* Mihai Maruseac (@mihaimaruseac)
|
||||
|
||||
## Workflow overview
|
||||
|
||||
This section sketches out at a high level how the entire Stackage build/curation
|
||||
process works:
|
||||
@ -23,10 +31,32 @@ process works:
|
||||
|
||||
The typical story on pull requests is: If Travis accepts it and the
|
||||
author only added packages under his/her own name, merge it. If the
|
||||
build later fails (see below), then block the package until it's
|
||||
fixed.
|
||||
build later fails (see [Adding Debian packages]), then block the
|
||||
package until it's fixed.
|
||||
|
||||
[Adding Debian packages]: https://github.com/fpco/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
|
||||
|
||||
If benchmarks, haddocks, or test suites fails at this point we
|
||||
typically also block the package until these issues are fixed. This in
|
||||
order to add packages with a clean slate.
|
||||
|
||||
Optionally we can check if packdeps says the package is up to date.
|
||||
Visit http://packdeps.haskellers.com/feed?needle=<package-name>
|
||||
|
||||
Builds may fail because of unrelated bounds changes. If this happens,
|
||||
first add any version bounds to get master into a passing state (see
|
||||
"Fixing bounds issues"), then re-run the travis build.
|
||||
|
||||
A common issue is that authors submit newly uploaded packages, it can
|
||||
take up to an hour before this has synced across the stack
|
||||
infrastructure. You can usually compare the versions of the package in
|
||||
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
|
||||
to what's on hackage to see if this is the case. Wait an hour and
|
||||
re-run the pull request.
|
||||
|
||||
Tests also commonly fail due to missing test files, and sometimes due
|
||||
to doctest limitations. You can point the maintainer to
|
||||
https://github.com/bergmark/blog/blob/master/2016/package-faq.md
|
||||
|
||||
## Fixing bounds issues
|
||||
|
||||
@ -36,20 +66,142 @@ issue on the Stackage repo about the problem, and modifying the
|
||||
build-constraints.yaml file to work around it in one of the ways below. Be sure
|
||||
to refer to the issue for workarounds added to that file.
|
||||
|
||||
* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
|
||||
* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
|
||||
* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
|
||||
### Temporary upper bounds
|
||||
|
||||
Most common technique, just prevent a new version of a library from
|
||||
being included immediately. This also applies to when only benchmarks
|
||||
and tests are affected.
|
||||
|
||||
* Copy the stackage-curator output and create a new issue, see e.g
|
||||
https://github.com/fpco/stackage/issues/2108
|
||||
|
||||
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
|
||||
|
||||
```yaml
|
||||
"Stackage upper bounds":
|
||||
# https://github.com/fpco/stackage/issues/2108
|
||||
- pipes < 4.3.0
|
||||
```
|
||||
|
||||
* Commit (message e.g. "Upper bound for #2108")
|
||||
* Optionally: Verify with `stackage-curator check` locally
|
||||
* Push
|
||||
* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
|
||||
|
||||
Sometimes releases for different packages are tightly coupled. Then it
|
||||
can make sense to combine them into one issue, as in
|
||||
https://github.com/fpco/stackage/issues/2143.
|
||||
|
||||
If a dependency that is not explicitly in stackage is causing test or
|
||||
benchmark failures you can skip or expect them to fail (see "Skipping
|
||||
tests and benchmarks" and "Expecting test/benchmark/haddock
|
||||
failures"). Bonus points for reporting this upstream to that packages'
|
||||
maintainer.
|
||||
|
||||
### Lifting upper bounds
|
||||
|
||||
You can try this when you notice that a package has been updated. You
|
||||
can also periodically try to lift bounds (I think it's good to do this
|
||||
at the start of your week /@bergmark)
|
||||
|
||||
If not all packages have been updated check if any of them are missing
|
||||
from the original issue and if so add a new comment mentioning them. A
|
||||
new package may appear if its dependencies were part of this issue but
|
||||
have been updated since the last time we checked. We want to give
|
||||
these new packages ample time to be upgraded.
|
||||
|
||||
If stackage-curator is happy commit the change ("Remove upper bounds
|
||||
and close #X"). After doing this the next nightly build may fail
|
||||
because some packages didn't have an upper bound in place, but
|
||||
compilation failed. In this case revert the previous commit so any
|
||||
disabled packages are enabled again, re-open the issue, and add a new
|
||||
comment with the failing packages. This is to give all maintainers
|
||||
enough time to upgrade for this case as well.
|
||||
|
||||
### Amending upper bounds
|
||||
|
||||
With the `pipes` example above there was later a new release of
|
||||
`pipes-safe` that required the **newer** version of `pipes`. You can
|
||||
add that package to the same upper bounds section,
|
||||
(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
|
||||
as we want to lift them simultaneously.
|
||||
|
||||
### Skipping tests and benchmarks
|
||||
|
||||
Sometimes tests and benchmark dependencies are forgotten or not cared
|
||||
for. To disable compilation for them add them to `skipped-tests` or
|
||||
`skipped-benchmarks`. If a package is added to these sections they
|
||||
won't be compiled, and their dependencies won't be taken into account.
|
||||
|
||||
There are sub sections under these headers that is used to group types
|
||||
of failures together, and also to document what type of failures
|
||||
exist.
|
||||
|
||||
### Expecting test/benchmark/haddock failures
|
||||
|
||||
The difference from the `skipped` sections is that items listed here
|
||||
are compiled and their dependencies are taken into account. These
|
||||
sections also have sub sections with groups and descriptions.
|
||||
|
||||
One big category of test suites in this section are those requiring
|
||||
running services. We don't want to run those, but we do want to check
|
||||
dependencies and compile them.
|
||||
|
||||
If there are no version bounds that would fix the issue or if you
|
||||
can't figure it out, file it
|
||||
(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
|
||||
maintainer for help.
|
||||
|
||||
### Waiting for new releases
|
||||
|
||||
Sometimes there is a failure reported on a (now possibly closed) issue
|
||||
on an external tracker. If an issue gets resolved but there is no
|
||||
hackage release yet we'd like to get notified when it's uploaded.
|
||||
|
||||
Add the package with its current version to the
|
||||
`tell-me-when-its-released` section. This will cause the build to stop
|
||||
when the new version is out.
|
||||
|
||||
### Excluding packages
|
||||
|
||||
In an extreme case of a non-responsive maintainer, you can remove the
|
||||
package entirely from Stackage. We try to avoid that whenever
|
||||
possible.
|
||||
|
||||
This typically happens when we move to a new major GHC release or when
|
||||
there are only a few packages waiting for updates on an upper bounds
|
||||
issue.
|
||||
|
||||
Comment out the offending packages from the "packages" section and add
|
||||
a comment saying why it was disabled:
|
||||
|
||||
```
|
||||
# - swagger # bounds: aeson 1.0
|
||||
```
|
||||
|
||||
If this causes reverse dependencies to be disabled we should notify
|
||||
the maintainers of those packages.
|
||||
|
||||
|
||||
## Updating the content of the Docker image used for building
|
||||
|
||||
### Adding Debian packages for required system tools or libraries
|
||||
Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
|
||||
Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
|
||||
since new packages are not added to the current lts release.
|
||||
After you've committed those changes, merging them into the `nightly` branch should
|
||||
trigger a DockerHub build. Simply run:
|
||||
|
||||
```bash
|
||||
$ git checkout nightly
|
||||
$ git merge master
|
||||
$ git push
|
||||
```
|
||||
|
||||
Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
|
||||
|
||||
Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
|
||||
Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
|
||||
It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
|
||||
|
||||
If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
|
||||
|
||||
### Upgrading GHC version
|
||||
The Dockerfile contains information on which GHC versions should be used. You
|
||||
@ -60,6 +212,9 @@ major version number (e.g., lts3 for lts-3.\*).
|
||||
Note that when starting a new LTS major release, you'll need to modify Docker
|
||||
Hub to create a new Docker tag for the relevant branch name.
|
||||
|
||||
You'll need to update both the `PATH` in `Dockerfile` and the `GHCVER` variable
|
||||
in `debian-bootstrap.sh`.
|
||||
|
||||
### Getting the new image to the build server
|
||||
Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
|
||||
below). Instead of pulling an unbounded number of images, I typically just
|
||||
@ -79,6 +234,26 @@ rm -r nightly/work/builds/nightly/
|
||||
```
|
||||
This should also be done when moving the Nightly docker image to a new version of Ubuntu.
|
||||
|
||||
If you're impatient and would like to build the Docker image on the
|
||||
build server instead of waiting for Docker Hub, you can run the
|
||||
following command:
|
||||
|
||||
```
|
||||
DIR=$(mktemp -d)
|
||||
(cd $DIR \
|
||||
&& git clone https://github.com/fpco/stackage \
|
||||
&& cd stackage \
|
||||
&& docker build --tag snoyberg/stackage:nightly .)
|
||||
rm -rf $DIR
|
||||
```
|
||||
|
||||
Note that we do a clean clone of the `stackage` repo instead of using
|
||||
the existing checkout because of how `docker build` works: it will
|
||||
send the entire local directory contents as context to the Docker
|
||||
daemon, which in the case of the build tree is a _lot_ of content. (We
|
||||
can discuss the wisdom—or lack thereof—of Docker's
|
||||
approach separately.)
|
||||
|
||||
## stackage-build server
|
||||
|
||||
You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
|
||||
@ -86,7 +261,7 @@ You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
|
||||
```
|
||||
Host stackage-build
|
||||
User curators
|
||||
Hostname ec2-52-5-20-252.compute-1.amazonaws.com
|
||||
Hostname build.stackage.org
|
||||
```
|
||||
|
||||
### Running the build script
|
||||
@ -106,7 +281,7 @@ we're just not there yet.
|
||||
/opt/stackage-build/stackage/automated/build.sh lts-3.0
|
||||
```
|
||||
|
||||
Recommended: run these from inside a `screen` session. If you get version bound
|
||||
Recommended: run these from inside a `tmux` session. If you get version bound
|
||||
problems on nightly or LTS major, you need to fix build-constraints.yaml (see
|
||||
info above). For an LTS minor bump, you'll typically want to use the
|
||||
`CONSTRAINTS` environment variable, e.g.:
|
||||
@ -126,14 +301,15 @@ If a build fails for bounds reasons, see all of the advice above. If the code
|
||||
itself doesn't build, or tests fail, open up an issue and then either put in a
|
||||
version bound to avoid that version or something else. It's difficult to give
|
||||
universal advice on how to solve things, since each situation is unique. Let's
|
||||
develop this advice over time. For now: if you're not sure, ask Michael for
|
||||
guidance.
|
||||
develop this advice over time. For now: if you're not sure, ask for guidance.
|
||||
|
||||
__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
|
||||
build plan, you can set the environment variable `NOPLAN=1`. This is
|
||||
useful for such cases as an intermittent test failure, out of memory
|
||||
condition, or manually tweaking the plan file.
|
||||
|
||||
Note LTS builds inherit the current Hackage data (stack updated for Nigthly) to avoid excess extra rebuilding.
|
||||
|
||||
### Timing
|
||||
|
||||
A looping script on the build server keeps trying to build nightly
|
||||
@ -146,9 +322,82 @@ LTS minor bumps typically are run on Sundays.
|
||||
### Website sync debugging (and other out of disk space errors)
|
||||
|
||||
* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
|
||||
* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
|
||||
* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
|
||||
* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
|
||||
* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
|
||||
|
||||
### Wiping the cache
|
||||
|
||||
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
|
||||
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
|
||||
You can wipe the nightly cache and rebuild everything by doing
|
||||
`rm -rf /var/stackage/stackage/automated/nightly`.
|
||||
Replace nightly with `lts7` to wipe the LTS 7 cache.
|
||||
|
||||
### Force a single package rebuild
|
||||
|
||||
You can force a single package to rebuild by deleting its "previous result"
|
||||
file, e.g.:
|
||||
|
||||
```
|
||||
$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
|
||||
```
|
||||
|
||||
## Local curator setup
|
||||
|
||||
We do not run the full stackage build locally as that might take too
|
||||
much time. However, some steps on the other hand are much faster to do
|
||||
yourself, e.g. verifying constraints without building anything.
|
||||
|
||||
To get started, install `stackage-curator` via Git, or [the Linux binary]:
|
||||
|
||||
```
|
||||
$ git clone git@github.com:fpco/stackage-curator.git
|
||||
$ cd stackage-curator && stack install
|
||||
```
|
||||
|
||||
It is a good idea to upgrade `stackage-curator` at the start of your week.
|
||||
Then, clone the stackage repo, get the latest packages and run dependency
|
||||
resolution:
|
||||
|
||||
```
|
||||
$ git clone git@github.com:fpco/stackage.git
|
||||
$ stack update && stackage-curator check
|
||||
```
|
||||
|
||||
This can be used to make sure all version bounds are in place, including for
|
||||
test suites and benchmarks, to check whether bounds can be lifted, and to get
|
||||
[tell-me-when-its-released] notifications.
|
||||
|
||||
`stackage-curator` does not build anything, so you wont see any compilation
|
||||
errors for builds, tests and benchmarks.
|
||||
|
||||
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
|
||||
|
||||
## Adding new curators
|
||||
|
||||
1. Add public ssh key to `~/.ssh/authorized_keys` on build server
|
||||
2. Add to fpco/stackage project.
|
||||
|
||||
## Dealing with a new GHC release
|
||||
|
||||
As mentioned in the [GHC upgrade note], the major impact of a new GHC release
|
||||
is on the packages that are causing upper bounds to be put in place. In order
|
||||
to minimise out-of-date breakage and allow maintainers to have a solid chance
|
||||
of getting their packages into the newest LTS, we try to do the following:
|
||||
|
||||
Make an early announcement (in the form of a blog post, typically) of the new
|
||||
GHC release on the nightly build and the planned deadline for the new LTS release.
|
||||
Make it clear, that in the time coming up to this, we hope package maintainers
|
||||
will upgrade their packages to allow for the new GHC release.
|
||||
|
||||
We prefer to prune packages causing upper bounds constraints **after** the LTS
|
||||
release to allow the maximum amount of packages to get into the newest LTS.
|
||||
|
||||
You will almost always need to update the Win32 package version listed in the
|
||||
build-constraints.yaml file.
|
||||
|
||||
After the first LTS release, the package pruning process may begin in the
|
||||
nightly build in order to move forward with getting the latest versions of
|
||||
packages compatible with the new GHC release.
|
||||
|
||||
[GHC upgrade note]: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
|
||||
|
||||
@ -1,8 +1,12 @@
|
||||
FROM ubuntu:16.04
|
||||
FROM fpco/pid1:16.04
|
||||
|
||||
ENV HOME /home/stackage
|
||||
ENV LANG en_US.UTF-8
|
||||
ENV PATH /opt/ghc/8.0.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
# NOTE: also update debian-bootstrap.sh when cuda version changes
|
||||
ENV PATH /usr/local/cuda-8.0/bin:/opt/ghc/8.4.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
ENV CUDA_PATH /usr/local/cuda-8.0
|
||||
ENV LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/nvvm/lib64
|
||||
|
||||
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
|
||||
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh
|
||||
|
||||
164
MAINTAINERS.md
164
MAINTAINERS.md
@ -2,81 +2,161 @@ This project is built around the concept of maintainers taking responsibility fo
|
||||
|
||||
The idea behind Stackage is that, if all packages work with the newest versions of dependencies, we avoid dependency hell. Specifically, we aim for:
|
||||
|
||||
* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
|
||||
* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
|
||||
* All packages are compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
|
||||
* All packages in a snapshot are compatible with the versions of libraries that ship with the GHC used in the snapshot ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
|
||||
|
||||
Packages in Stackage are not patched: all package changes occur upstream in Hackage.
|
||||
|
||||
## Adding a package
|
||||
|
||||
Anyone can add any package to Stackage but you may only add packages under your own name. It's highly encouraged that the actual package maintainer is also the Stackage maintainer, if that is not the case you should drop the package maintainer a note first.
|
||||
|
||||
To add your package, first fork this repository.
|
||||
In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`.
|
||||
To add a set of packages, you would add:
|
||||
To add your package you can edit [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) directly on github or fork the project. There's a section called `packages` where you would add yourself and your packages:
|
||||
|
||||
"My Name myemail@example.com @mygithubuser":
|
||||
"My Name <myemail@example.com> @mygithubuser":
|
||||
- package1
|
||||
- package2
|
||||
- package3
|
||||
|
||||
After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
|
||||
If your library depends on a C library, please add it to the `debian-bootstrap.sh` script.
|
||||
|
||||
After doing that commit with a message like "add foo-bar" and send a pull request.
|
||||
|
||||
The continuous integration job will do some checks to see if your package's dependencies are up-to-date.
|
||||
|
||||
The CI job notably doesn't compile packages, run tests, build documentation, or find missing C libraries.
|
||||
If you want to be proactive or if CI fails, you can make sure that your package builds against the latest nightly:
|
||||
|
||||
If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
|
||||
```
|
||||
# Build from the tarball on Hackage to check for missing files
|
||||
$ stack unpack yourpackage && cd yourpackage-*
|
||||
# Generate a pristine stack.yaml, adding any missing extra-deps
|
||||
$ rm -f stack.yaml && stack init --resolver nightly --solver
|
||||
# Build, generate docs, test, and build benchmarks
|
||||
$ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
|
||||
```
|
||||
|
||||
This approach works well, but has two limitations you should be aware
|
||||
of:
|
||||
|
||||
* It won't notify you of restrictive upper bounds in your package if
|
||||
Stackage has the same upper bounds. For that reason, we recommend
|
||||
using [Packdeps](http://packdeps.haskellers.com/) (see "Following
|
||||
dependency upgrades" below).
|
||||
* If the latest Stackage Nightly is missing some of the latest
|
||||
packages, your build above may succeed whereas the Travis job may
|
||||
fail. Again: Packdeps will help you detect this situation.
|
||||
|
||||
Alternatively, you can build with `cabal`. Note that this may end up
|
||||
using older dependency versions:
|
||||
|
||||
```
|
||||
$ ghc --version # Should be the same as the latest nightly, it's in the title of https://www.stackage.org/nightly
|
||||
$ cabal update
|
||||
$ ghc --version # Should give v8.0.1
|
||||
$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
|
||||
$ cd PACKAGE-VERSION
|
||||
$ cabal get PACKAGE
|
||||
$ cd PACKAGE-*
|
||||
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
|
||||
$ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
|
||||
$ cabal install --enable-tests --enable-benchmarks --allow-newer
|
||||
$ cabal test
|
||||
$ cabal haddock
|
||||
```
|
||||
|
||||
**NB** Please use commit messages like "add foo-bar" or "add johndev's packages"
|
||||
(`build-constraints.yaml` is the most frequently changed file in this git repo
|
||||
so commit messages like "update build-constraints.yaml" are not helpful).
|
||||
## Github and Notifications
|
||||
|
||||
**NB2** There can be a delay of up to an hour before package versions
|
||||
newly uploaded to Hackage appear to our build server. If you just
|
||||
uploaded a package to Hackage that you're trying to get included, we
|
||||
recommend waiting an hour before opening the PR. You can verify this
|
||||
by making sure the latest version is listed at
|
||||
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/.
|
||||
Stackage uses Github notifications to reduce overhead of contacting individual
|
||||
package maintainers through various channels. As a package maintainer, you will
|
||||
receive notifications for a number of reasons, package build failures of
|
||||
different sorts, blockages and bounds issues, etc.
|
||||
|
||||
## Uploading a new package
|
||||
Please note, Github does some throttling on the number people that can be
|
||||
notified within one issue. This means that on issues created with a large
|
||||
number of packages affected, maintainers may not receive a notification. This
|
||||
is not ideal, but Stackage is largely a manual process and done on a best
|
||||
effort basis.
|
||||
|
||||
When a new version of a package is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and raise GitHub issue tickets to resolve the issue.
|
||||
## Uploading a new package version
|
||||
|
||||
If the new version doesn't compile then the package author should quickly (within 1 week) upload a fixed version.
|
||||
When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage. That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
|
||||
|
||||
If the new version doesn't compile then the package author should upload a fixed version.
|
||||
|
||||
If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite.
|
||||
|
||||
|
||||
## Following dependency upgrades
|
||||
|
||||
If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is highly recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
|
||||
If a new version of a dependency is released your package may fail to
|
||||
build or tests may fail. In most cases we will add an upper bound on
|
||||
the dependency so that your package remains in nightly. We'll also
|
||||
create a github issue pinging you with the bounds issues or give build
|
||||
logs showing failures. It's then up to you to modify your package.
|
||||
|
||||
**If restrictive version bounds are the only problem** then you must quickly (within 1 week) upload a new version with relaxed version bounds. Note that unlike the PVP, Stackage does not require upper bounds.
|
||||
We recommend that you also follow the dependencies of your packages on
|
||||
[Packdeps](http://packdeps.haskellers.com/) (typically using the RSS
|
||||
feeds) as well as that often gives you notice ahead of stackage
|
||||
issues. There are cases where we will not notice a new release of a
|
||||
package because of other upper bounds that are in place.
|
||||
|
||||
**If the new dependency causes breaking changes** then all package authors should quickly assess the likely impact on their package (within 1 week) and then produce a new compatible version. The expected timeline for new versions varies between 1 week and 1 month, depending on the significance of the change, and thus the work required to produce those new versions.
|
||||
If a package is not updated in time, it may be temporarily removed
|
||||
from Stackage by the curator team. We strive to notify you when this
|
||||
happens. If it does you are always welcome to file another pull
|
||||
request to add it back.
|
||||
|
||||
We typically use fairly long windows before disabling packages, but it
|
||||
is decided on a case-by-case basis.
|
||||
|
||||
## Failing to meet the time limits
|
||||
* If restrictive version bounds are the only problem, we will give
|
||||
maintainers at least a week to respond.
|
||||
* If there are real breaking changes, the curator team will retain
|
||||
more discretion on how long a window to give before dropping
|
||||
packages.
|
||||
* We usually drop all upper bounds and disable packages when we create
|
||||
a new Long Term Support (LTS) major version.
|
||||
* There are rare cases where an upper bound or build failure are hard
|
||||
to deal with so then we may disable
|
||||
|
||||
Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack.
|
||||
**NOTE** Previously we had stricter time limits, but we decided to
|
||||
remove that: Anyone is free to add a package to Stackage regardless of
|
||||
responsiveness guarantees. However, as stated above, we may elect to
|
||||
temporarily remove a package if it is not updated in a timely manner.
|
||||
|
||||
The time limits are intended to stop people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for long.
|
||||
## Delays
|
||||
|
||||
Maintainers are humans, humans get sick/have babies/go on
|
||||
holiday. Sometimes a dependency upgrade is extra time consuming.
|
||||
|
||||
Consider finding a co-maintainer with access to SCM and Hackage to
|
||||
help you out.
|
||||
|
||||
We appreciate if you notify us of any expected delays in the Github
|
||||
issues, some kind soul might decide to help out.
|
||||
|
||||
## Upgrading to a new GHC version
|
||||
|
||||
The Stackage curation team tries to move Stackage Nightly to new versions of GHC quickly as they become available, while keeping LTS Haskell on a regular release schedule. For package maintainers, the most important impacts of a new GHC release are:
|
||||
The Stackage curation team tries to move Stackage Nightly to new
|
||||
versions of GHC quickly as they become available, while keeping LTS
|
||||
Haskell on a regular release schedule. For package maintainers, the
|
||||
most important impacts of a new GHC release are:
|
||||
|
||||
* We will typically do a sweep through the Stackage upper bounds and aggressively remove packages that block them. This is because, in most cases, we will need to move to the newest versions of a package to get support for the latest GHC, and asking package maintainers to backport their fixes is an undue burden
|
||||
* We will definitely do this at a GHC major version release, and may do so at a minor version release
|
||||
* Packages that are incompatible with the newest GHC version will be temporarily blocked
|
||||
* We will typically do a sweep through the Stackage upper bounds and
|
||||
remove blocking packages. We prefer to do it this way rather than
|
||||
ask other maintainers to backport fixes.
|
||||
* We will definitely do this at a GHC major version release, and
|
||||
may do so at a minor version release
|
||||
* Packages that are incompatible with the newest GHC version will be
|
||||
temporarily disabled
|
||||
|
||||
If your package ends up being temporarily removed from Stackage Nightly, please simply send a pull request to add it back once it and its dependencies are compatible with the newest GHC version.
|
||||
If your package ends up being temporarily disabled from Stackage
|
||||
Nightly, please simply send a pull request to add it back once it and
|
||||
its dependencies are compatible with the newest GHC version.
|
||||
|
||||
Note that it is _not_ a goal of LTS Haskell to track the latest
|
||||
version of GHC. If you want the latest and greatest, Stackage Nightly
|
||||
is your best bet. In particular, LTS Haskell will often—but not
|
||||
always—avoid upgrading to the first point release of GHC
|
||||
releases (e.g., 8.2.1) to allow further testing and to get the
|
||||
benefits of the first bugfix release (e.g., 8.2.2).
|
||||
|
||||
## Adding a package to an LTS snapshot
|
||||
|
||||
@ -87,10 +167,16 @@ following the above steps, you can get your package into the next major LTS
|
||||
Haskell release.
|
||||
|
||||
If you would like to get your package added to an existing LTS Haskell major
|
||||
release (e.g., if `lts-3.21` is out, you would want your package to appear in
|
||||
`lts-3.22`), please do the following in addition to the steps above:
|
||||
release (e.g., if `lts-8.9` is out, you would want your package to appear in
|
||||
`lts-8.10`), please do the following in addition to the steps above:
|
||||
|
||||
* Check that your package can be built with that LTS major version (e.g. `stack build --test --bench --haddock --resolver lts-8.10`)
|
||||
* Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
|
||||
* Specify the LTS major version you would like your package to go into (e.g., lts-3)
|
||||
* Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
|
||||
* Be patient! The LTS releases are by their nature more conservative than nightly, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue quickly, but there may be some delay.
|
||||
* Specify the LTS major versions you would like your packages to go into (e.g. lts-8)
|
||||
* Provide a list of packages you would like added
|
||||
* If relevant, mention any upper bounds that are needed on those packages
|
||||
* Be patient! The LTS releases are less frequent than Nightly. The
|
||||
Stackage curators will try to get to your issue as soon as possible,
|
||||
but it may take some time.
|
||||
* We gradually stop maintainng old LTS major versions, so your
|
||||
request may take longer or be declined if it's for an old LTS.
|
||||
|
||||
48
README.md
48
README.md
@ -17,7 +17,7 @@ Add your package
|
||||
We welcome all packages, provided:
|
||||
|
||||
* The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
|
||||
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
|
||||
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
|
||||
* The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
|
||||
* The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
|
||||
|
||||
@ -37,7 +37,6 @@ project information. In addition, we have the following repositories:
|
||||
|
||||
* [stackage-server](https://github.com/fpco/stackage-server) [](https://travis-ci.org/fpco/stackage-server)
|
||||
* [stackage-curator](https://github.com/fpco/stackage-curator) [](https://travis-ci.org/fpco/stackage-curator)
|
||||
* [stackage-types](https://github.com/fpco/stackage-types) [](https://travis-ci.org/fpco/stackage-types)
|
||||
* [lts-haskell](https://github.com/fpco/lts-haskell)
|
||||
* [stackage-nightly](https://github.com/fpco/stackage-nightly)
|
||||
|
||||
@ -58,7 +57,7 @@ Build the package set
|
||||
---------------------
|
||||
|
||||
Generally only the stackage build server run by the stackage curator
|
||||
team and people intrested in incorporating stackage snapshots into an
|
||||
team and people interested in incorporating stackage snapshots into an
|
||||
OS distribution need to build the entire package set. If you're
|
||||
interested in trying this yourself, please check out
|
||||
[the curator guide](https://github.com/fpco/stackage/blob/master/CURATORS.md),
|
||||
@ -84,3 +83,46 @@ The following describes at a high level the series of steps for processing
|
||||
1. Load up most recent build plan
|
||||
2. Convert build plan into constraints for next build
|
||||
3. Continue from step (3) above
|
||||
|
||||
Frequently Asked Questions
|
||||
--------------------------
|
||||
|
||||
__Why does Stackage have an older version of a package than Hackage?__
|
||||
|
||||
There are a number of answers to this question:
|
||||
|
||||
* Simplest reason: how old of a Stackage snapshot are you using? Once a
|
||||
snapshot is created, it's frozen for all time. So if you use
|
||||
nightly-2016-01-01, by the time you get to 2018, it will be pretty dated.
|
||||
* If you're using an LTS snapshot: we lock down major versions when
|
||||
first creating an LTS run, so subsequent minor versions will not get
|
||||
new versions necessary. For example, if LTS 6.0 has `foo` version
|
||||
1.2.3, and the author immediately thereafter releases a version
|
||||
1.3.0 and never releases another 1.2.\* version, you'll never get
|
||||
another update in the LTS 6 line
|
||||
* Sometimes we have upper bounds in place because other packages have
|
||||
problems with newer versions of dependencies. Open up the
|
||||
[build-constraints file](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
|
||||
and search for "Stackage upper bounds"
|
||||
* Wired-in packages - those that ship with GHC and cannot be upgraded,
|
||||
and packages depending on them - are fixed to GHC versions. Common
|
||||
examples of this are containers and transformers. There's a lot more
|
||||
information on this in
|
||||
[an FP Complete blog post](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)
|
||||
|
||||
__How long do you maintain an LTS build?__
|
||||
|
||||
We only guarantee that we will maintain a single LTS major version at
|
||||
a time, and that it will be maintained for at least three months. This
|
||||
is the
|
||||
[originally proposed support window](https://www.fpcomplete.com/blog/2014/12/backporting-bug-fixes),
|
||||
and hasn't changed since then.
|
||||
|
||||
That said, we do maintain the capability to keep multiple LTS runs
|
||||
operational in parallel, and with LTS 6 and 7 in fact did so. We
|
||||
aren't changing our guarantees yet on longevity of a release, but are
|
||||
trying to push out the bounds a bit farther.
|
||||
|
||||
__What time are Stackage snapshots published?__
|
||||
|
||||
Stackage Nightly and LTS are not released at a fixed time of day, they get pushed to stackage.org (and the metadata to the stackage-nightly and stackage-lts github repos) when their builds finish on the Stackage build server and the latest built haddocks have been synced over. This time varies greatly depending on build times for package updates, bounds breakage, problems with new packages being added and other build issues, etc. There are days when a release does not happen. LTS releases tend to happen over the weekend or early in the week.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -eux
|
||||
set -eu +x
|
||||
|
||||
ROOT=$(cd $(dirname $0) ; pwd)
|
||||
TARGET=$1
|
||||
@ -30,6 +30,7 @@ DOT_STACKAGE_DIR=$ROOT/dot-stackage
|
||||
WORKDIR=$ROOT/$TAG/work
|
||||
EXTRA_BIN_DIR=$ROOT/extra-bin
|
||||
SSH_DIR=$ROOT/ssh-$SHORTNAME
|
||||
USERID=$(id -u)
|
||||
|
||||
mkdir -p \
|
||||
"$CABAL_DIR" \
|
||||
@ -77,12 +78,13 @@ rm -f stackage-curator stackage-curator.bz2
|
||||
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
bunzip2 stackage-curator.bz2
|
||||
chmod +x stackage-curator
|
||||
./stackage-curator --version
|
||||
)
|
||||
|
||||
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
|
||||
ARGS_PREBUILD="$ARGS_COMMON -u $USER -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
|
||||
ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
|
||||
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
|
||||
ARGS_UPLOAD="$ARGS_COMMON -u $USER -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
|
||||
ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
|
||||
|
||||
# Make sure we actually need this snapshot. We only check this for LTS releases
|
||||
# since, for nightlies, we'd like to run builds even if they are unnecessary to
|
||||
@ -96,22 +98,31 @@ fi
|
||||
# Get latest stack
|
||||
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
|
||||
|
||||
# Do all of the pre-build actions:
|
||||
# Determine the new build plan unless NOPLAN is set
|
||||
#
|
||||
# * Update the package index
|
||||
# * Update the package index (unless LTS)
|
||||
# * Create a new plan
|
||||
if [ "${NOPLAN:-}x" = "x" ]
|
||||
then
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
|
||||
else
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Do the rest of the pre-build actions:
|
||||
#
|
||||
# * Check that the plan is valid
|
||||
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
|
||||
# * Do a single unpack to create the package index cache (again due to directory perms)
|
||||
if [ "${NOPLAN:-}x" = "x" ]
|
||||
then
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-} && stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
|
||||
fi
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
|
||||
|
||||
# Now do the actual build. We need to first set the owner of the home directory
|
||||
# correctly, so we run the command as root, change owner, and then use sudo to
|
||||
# switch back to the current user
|
||||
docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
|
||||
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs 4 --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
|
||||
|
||||
# Make sure we actually need this snapshot. We used to perform this check
|
||||
# exclusively before building. Now we perform it after as well for the case of
|
||||
@ -128,11 +139,5 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-
|
||||
# * Register as a new Hackage distro
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
|
||||
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
echo "Running cron.sh (hiding verbose output)"
|
||||
./cron.sh | grep -v '^Skipping'
|
||||
echo "done."
|
||||
fi
|
||||
|
||||
echo -n "Completed at "
|
||||
date
|
||||
|
||||
@ -2,14 +2,23 @@
|
||||
|
||||
set -eux
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
CRONDIR=$(pwd)/crondir
|
||||
mkdir -p $CRONDIR
|
||||
source aws.sh
|
||||
|
||||
IMAGE=fpco/stackage-server-prod:latest
|
||||
docker pull $IMAGE
|
||||
|
||||
stack update
|
||||
|
||||
date
|
||||
echo "Running stackage-server-cron..."
|
||||
echo "('tail -f $CRONDIR/stackage-server-cron.log' to watch)"
|
||||
docker run --rm \
|
||||
-v $CRONDIR:/home/ubuntu \
|
||||
-w /home/ubuntu \
|
||||
fpco/stackage-server-prod:latest \
|
||||
-v $HOME/.stack/indices:/home/ubuntu/.stack/indices:ro \
|
||||
--workdir /home/ubuntu \
|
||||
-p 17834:17834 \
|
||||
$IMAGE \
|
||||
bash -c "useradd $(whoami) -u $(id -u); sudo -u $(whoami) env HOME=/home/ubuntu AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY bash -c '/usr/local/bin/stackage-server-cron 2>&1 | tee -a /home/ubuntu/stackage-server-cron.log'"
|
||||
|
||||
@ -5,12 +5,6 @@ cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
while true; do
|
||||
./build.sh nightly-$(date -u +%F)
|
||||
date
|
||||
echo
|
||||
|
||||
echo "Running cron.sh (hiding verbose output)"
|
||||
./cron.sh > cron.log 2>&1 # | grep -v '^Skipping'
|
||||
echo "done."
|
||||
date
|
||||
|
||||
sleep 30m
|
||||
echo
|
||||
done
|
||||
|
||||
44
become-a-curator.md
Normal file
44
become-a-curator.md
Normal file
@ -0,0 +1,44 @@
|
||||
## We are no longer accepting applications, please wait until next time
|
||||
|
||||
We are looking to expand the stackage curator team!
|
||||
|
||||
As a stackage curator you will help the community by curating and
|
||||
publishing the widely used nightly and LTS snapshots: Collections of
|
||||
packages that are guaranteed to play well together. All curators are
|
||||
volunteers.
|
||||
|
||||
This is a great opportunity to contribute to the haskell community and get
|
||||
familiar with the tools of the trade! You will also be in regular
|
||||
contact with experienced community members.
|
||||
|
||||
We don't have any prerequisites, but familiarity with any of Haskell,
|
||||
Cabal, Stack, Linux, Git, and Docker is a plus.
|
||||
|
||||
We work in shifts of one week, which means you will be on duty every
|
||||
6th week. Our suggestion is to pair you up with another curator to
|
||||
help out in the beginning.
|
||||
|
||||
The workload is usually about 30 minutes per day, whenever you have
|
||||
time. The work consists of keeping the builds running on the build
|
||||
server, filing issues for packages needing updates when new releases
|
||||
occur, and closing resolved issues. The LTS snapshots are prepared
|
||||
once a week. All communication with package maintainers happen on
|
||||
github.
|
||||
|
||||
We also have a curator slack channel where we help each other out.
|
||||
|
||||
The current curator team consists of:
|
||||
* Adam Bergmark
|
||||
* Dan Burton
|
||||
* Jens Petersen
|
||||
* Luke Murphy
|
||||
* Michael Snoyman
|
||||
|
||||
We onboarded Luke as the newest member a few months ago and this
|
||||
helped us iron out and document the process further.
|
||||
|
||||
You can read the curator documentation here: https://github.com/fpco/stackage/blob/master/CURATORS.md
|
||||
|
||||
To apply, please fill in this form: [REDACTED]
|
||||
|
||||
If you applied last time, we encourage you to do so again!
|
||||
File diff suppressed because it is too large
Load Diff
2
check
2
check
@ -3,4 +3,4 @@
|
||||
# Convenience script for checking constraints locally
|
||||
|
||||
cd `dirname $0`
|
||||
exec stack exec --resolver ghc-8.0.1 stackage-curator check
|
||||
exec stack exec --resolver ghc-8.4.1 stackage-curator check
|
||||
|
||||
@ -13,7 +13,6 @@
|
||||
set -exu
|
||||
|
||||
mkdir /home/stackage -p
|
||||
locale-gen en_US.UTF-8
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
@ -21,40 +20,47 @@ apt-get install -y software-properties-common
|
||||
|
||||
add-apt-repository ppa:hvr/ghc -y
|
||||
add-apt-repository -y ppa:marutter/rrutter
|
||||
# not sure what this was needed for
|
||||
#add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse
|
||||
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
|
||||
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy main'
|
||||
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
|
||||
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
|
||||
|
||||
# Set the GHC version
|
||||
GHCVER=8.0.1
|
||||
|
||||
# Get Stack
|
||||
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 575159689BEFB442
|
||||
echo 'deb http://download.fpcomplete.com/ubuntu xenial main'|tee /etc/apt/sources.list.d/fpco.list
|
||||
GHCVER=8.4.1
|
||||
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
apt-transport-https \
|
||||
build-essential \
|
||||
ghc-$GHCVER \
|
||||
ghc-$GHCVER-dyn \
|
||||
ghc-$GHCVER-prof \
|
||||
ghc-$GHCVER-htmldocs \
|
||||
hscolour \
|
||||
sudo \
|
||||
cmake \
|
||||
curl \
|
||||
freeglut3-dev \
|
||||
freetds-dev \
|
||||
fsharp \
|
||||
g++ \
|
||||
gawk \
|
||||
ghc-$GHCVER \
|
||||
ghc-$GHCVER-dyn \
|
||||
ghc-$GHCVER-htmldocs \
|
||||
ghc-$GHCVER-prof \
|
||||
git \
|
||||
gnupg \
|
||||
gradle \
|
||||
hscolour \
|
||||
libadns1-dev \
|
||||
libaio1 \
|
||||
libalut-dev \
|
||||
libasound2-dev \
|
||||
libblas-dev \
|
||||
libbz2-dev \
|
||||
libcairo2-dev \
|
||||
libclang-3.7-dev \
|
||||
libclang-3.9-dev \
|
||||
libcurl4-openssl-dev \
|
||||
libcwiid-dev \
|
||||
libdevil-dev \
|
||||
libedit-dev \
|
||||
libedit2 \
|
||||
libfftw3-dev \
|
||||
libflac-dev \
|
||||
libfreenect-dev \
|
||||
libgd2-xpm-dev \
|
||||
libgeoip-dev \
|
||||
@ -70,7 +76,11 @@ apt-get install -y \
|
||||
libgtk2.0-dev \
|
||||
libgtksourceview-3.0-dev \
|
||||
libhidapi-dev \
|
||||
libi2c-dev \
|
||||
libicu-dev \
|
||||
libimlib2-dev \
|
||||
libjack-jackd2-dev \
|
||||
libjavascriptcoregtk-4.0-dev \
|
||||
libjudy-dev \
|
||||
liblapack-dev \
|
||||
libleveldb-dev \
|
||||
@ -80,17 +90,28 @@ apt-get install -y \
|
||||
libmagickcore-dev \
|
||||
libmagickwand-dev \
|
||||
libmarkdown2-dev \
|
||||
libmono-2.0-dev \
|
||||
libmp3lame-dev \
|
||||
libmpfr-dev \
|
||||
libmysqlclient-dev \
|
||||
libncurses-dev \
|
||||
libnfc-dev \
|
||||
liboath-dev \
|
||||
libnotify-dev \
|
||||
libopenal-dev \
|
||||
libpango1.0-dev \
|
||||
libpcap0.8-dev \
|
||||
libpq-dev \
|
||||
libre2-dev \
|
||||
libsdl1.2-dev \
|
||||
libsdl2-dev \
|
||||
libsdl2-gfx-dev \
|
||||
libsdl2-image-dev \
|
||||
libsdl2-mixer-dev \
|
||||
libsdl2-ttf-dev \
|
||||
libsnappy-dev \
|
||||
libsndfile1-dev \
|
||||
libsox-dev \
|
||||
libsqlite3-dev \
|
||||
libssl-dev \
|
||||
libsystemd-dev \
|
||||
@ -105,26 +126,58 @@ apt-get install -y \
|
||||
libxss-dev \
|
||||
libyaml-dev \
|
||||
libzip-dev \
|
||||
libzstd-dev \
|
||||
libzmq3-dev \
|
||||
llvm-3.7 \
|
||||
llvm-3.9 \
|
||||
locales \
|
||||
m4 \
|
||||
minisat \
|
||||
mono-mcs \
|
||||
nettle-dev \
|
||||
ninja-build \
|
||||
nodejs \
|
||||
npm \
|
||||
openjdk-8-jdk \
|
||||
python-mpltoolkits.basemap \
|
||||
python3-matplotlib \
|
||||
python3-numpy \
|
||||
python3-pip \
|
||||
r-base \
|
||||
r-base-dev \
|
||||
ruby-dev \
|
||||
stack \
|
||||
sudo \
|
||||
unixodbc-dev \
|
||||
wget \
|
||||
xclip \
|
||||
z3 \
|
||||
zip \
|
||||
zlib1g-dev
|
||||
|
||||
# odbc
|
||||
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
|
||||
curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
|
||||
apt-get update
|
||||
ACCEPT_EULA=Y apt-get install msodbcsql17 -y
|
||||
|
||||
locale-gen en_US.UTF-8
|
||||
|
||||
curl -sSL https://get.haskellstack.org/ | sh
|
||||
|
||||
# Put documentation where we expect it
|
||||
mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
|
||||
|
||||
# llvm-5.0 for GHC (separate since it needs wget)
|
||||
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
|
||||
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-5.0 main" \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y llvm-5.0
|
||||
|
||||
# llvm-6.0 for llvm-hs (separate since it needs wget)
|
||||
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
|
||||
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y llvm-6.0
|
||||
|
||||
# Buggy versions of ld.bfd fail to link some Haskell packages:
|
||||
# https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
|
||||
# faster anyways and uses less RAM.
|
||||
@ -135,9 +188,14 @@ update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
|
||||
# This version is tracked here:
|
||||
# https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
|
||||
#
|
||||
# GHC 8.0 requires LLVM 3.7 tools (specifically, llc-3.7 and opt-3.7).
|
||||
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.7" 50
|
||||
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.7" 50
|
||||
# GHC 8.4 requires LLVM 5.0 tools (specifically, llc-5.0 and opt-5.0).
|
||||
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-5.0" 50
|
||||
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-5.0" 50
|
||||
|
||||
# Made sure a "node" binary is in the path, as well as "nodejs".
|
||||
# A historical naming collision on Debian means that the binary is called "nodejs",
|
||||
# but some tools like tsc still expect "node" to exist.
|
||||
ln -s /usr/bin/nodejs /usr/bin/node
|
||||
|
||||
# install ocilib dependencies then build and install ocilib
|
||||
cd /tmp \
|
||||
@ -147,9 +205,9 @@ cd /tmp \
|
||||
&& wget https://storage.googleapis.com/oracle.fpinsight.com/instantClient/oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& dpkg -i oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& rm -f oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& wget https://github.com/vrogier/ocilib/archive/v4.2.1.tar.gz \
|
||||
&& tar xvf v4.2.1.tar.gz \
|
||||
&& cd /tmp/ocilib-4.2.1 \
|
||||
&& wget https://github.com/vrogier/ocilib/archive/v4.3.2.tar.gz \
|
||||
&& tar xvf v4.3.2.tar.gz \
|
||||
&& cd /tmp/ocilib-4.3.2 \
|
||||
&& ./configure --with-oracle-import=linkage \
|
||||
--with-oracle-charset=ansi \
|
||||
--with-oracle-headers-path=/usr/include/oracle/12.1/client64 \
|
||||
@ -157,7 +215,47 @@ cd /tmp \
|
||||
&& make \
|
||||
&& make install \
|
||||
&& cd \
|
||||
&& rm -rf /tmp/ocilib-4.2.1 \
|
||||
&& rm -rf /tmp/ocilib-4.3.2 \
|
||||
&& echo "/usr/local/lib" > /etc/ld.so.conf.d/usr-local.conf \
|
||||
&& echo "/usr/lib/oracle/12.1/client64/lib" > /etc/ld.so.conf.d/oracle-client.conf \
|
||||
&& ldconfig
|
||||
|
||||
# Add JDK to system paths.
|
||||
echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
|
||||
&& ldconfig
|
||||
|
||||
# Install version 3 of the protobuf compiler. (The `protobuf-compiler` package only
|
||||
# supports version 2.)
|
||||
curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip \
|
||||
&& sudo unzip -o protoc-3.3.0-linux-x86_64.zip -d /usr bin/protoc \
|
||||
&& rm -f protoc-3.3.0-linux-x84_64.zip
|
||||
|
||||
# Install the TensorFlow C API.
|
||||
curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.1.0.tar.gz > libtensorflow.tar.gz \
|
||||
&& sudo tar zxf libtensorflow.tar.gz -C /usr \
|
||||
&& rm libtensorflow.tar.gz \
|
||||
&& ldconfig
|
||||
|
||||
# NOTE: also update Dockerfile when cuda version changes
|
||||
# Install CUDA toolkit
|
||||
# The current version can be found at: https://developer.nvidia.com/cuda-downloads
|
||||
CUDA_PKG=8.0.61-1 # update this on new version
|
||||
CUDA_VER=${CUDA_PKG:0:3}
|
||||
CUDA_APT=${CUDA_VER/./-}
|
||||
|
||||
pushd /tmp \
|
||||
&& wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
|
||||
&& dpkg -i cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
|
||||
&& apt-get update -qq \
|
||||
&& apt-get install -y cuda-drivers cuda-core-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} cuda-cufft-dev-${CUDA_APT} cuda-cublas-dev-${CUDA_APT} cuda-cusparse-dev-${CUDA_APT} cuda-cusolver-dev-${CUDA_APT} \
|
||||
&& rm cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
|
||||
&& export CUDA_PATH=/usr/local/cuda-${CUDA_VER} \
|
||||
&& export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH+x} \
|
||||
&& export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${LD_LIBRARY_PATH} \
|
||||
&& export PATH=${CUDA_PATH}/bin:${PATH} \
|
||||
&& popd
|
||||
|
||||
# non-free repo for mediabus-fdk-aac
|
||||
apt-add-repository multiverse \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y nvidia-cuda-dev
|
||||
|
||||
Loading…
Reference in New Issue
Block a user