Merge branch 'master' into nightly

.travis.yml

@@ -2,7 +2,7 @@
 sudo: false
 
 # Choose a lightweight base image; we provide our own build tools.
-language: c
+language: generic
 
 # GHC depends on GMP. You can add other dependencies here as well.
 addons:

CURATORS.md

@@ -4,7 +4,7 @@ Originally this was handled largely by Michael Snoyman,
 but now we are a team of 4 people handling requests weekly in rotation.
 Curation activities are mostly automated, and do not take up a significant amount of time.
 
-## Workflow
+## Workflow overview
 
 This section sketches out at a high level how the entire Stackage build/curation
 process works:
@@ -23,10 +23,30 @@ process works:
 
 The typical story on pull requests is: If Travis accepts it and the
 author only added packages under his/her own name, merge it.  If the
-build later fails (see below), then block the package until it's
+build later fails (see "Adding Debian packages for required system tools or libraries"),
-fixed.
+then block the package until it's fixed.
+
+If benchmarks, haddocks, or test suites fails at this point we
+typically also block the package until these issues are fixed. This in
+order to add packages with a clean slate.
 
 Optionally we can check if packdeps says the package is up to date.
+Visit http://packdeps.haskellers.com/feed?needle=<package-name>
+
+Builds may fail because of unrelated bounds changes. If this happens,
+first add any version bounds to get master into a passing state (see
+"Fixing bounds issues"), then re-run the travis build.
+
+A common issue is that authors submit newly uploaded packages, it can
+take up to an hour before this has synced across the stack
+infrastructure. You can usually compare the versions of the package in
+https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
+to what's on hackage to see if this is the case. Wait an hour and
+re-run the pull request.
+
+Tests also commonly fail due to missing test files, and sometimes due
+to doctest limitations. You can point the maintainer to
+https://github.com/bergmark/blog/blob/master/2016/package-faq.md
 
 ## Fixing bounds issues
 
@@ -36,20 +56,136 @@ issue on the Stackage repo about the problem, and modifying the
 build-constraints.yaml file to work around it in one of the ways below. Be sure
 to refer to the issue for workarounds added to that file.
 
-* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
+### Temporary upper bounds
-* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
+
-* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
+Most common technique, just prevent a new version of a library from
+being included immediately. This also applies to when only benchmarks
+and tests are affected.
+
+* Copy the stackage-curator output and create a new issue, see e.g
+https://github.com/fpco/stackage/issues/2108
+
+* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
+
+```yaml
+    "Stackage upper bounds":
+        # https://github.com/fpco/stackage/issues/2108
+        - pipes < 4.3.0
+```
+
+* Commit (message e.g. "Upper bound for #2108")
+* Optionally: Verify with `stackage-curator check` locally
+* Push
+* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
+
+Sometimes releases for different packages are tightly coupled. Then it
+can make sense to combine them into one issue, as in
+https://github.com/fpco/stackage/issues/2143.
+
+If a dependency that is not explicitly in stackage is causing test or
+benchmark failures you can skip or expect them to fail (see "Skipping
+tests and benchmarks" and "Expecting test/benchmark/haddock
+failures"). Bonus points for reporting this upstream to that packages'
+maintainer.
+
+### Lifting upper bounds
+
+You can try this when you notice that a package has been updated. You
+can also periodically try to lift bounds (I think it's good to do this
+at the start of your week /@bergmark)
+
+If not all packages have been updated check if any of them are missing
+from the original issue and if so add a new comment mentioning them. A
+new package may appear if its dependencies were part of this issue but
+have been updated since the last time we checked. We want to give
+these new packages ample time to be upgraded.
+
+If stackage-curator is happy commit the change ("Remove upper bounds
+and close #X"). After doing this the next nightly build may fail
+because some packages didn't have an upper bound in place, but
+compilation failed. In this case revert the previous commit so any
+disabled packages are enabled again, re-open the issue, and add a new
+comment with the failing packages. This is to give all maintainers
+enough time to upgrade for this case as well.
+
+### Amending upper bounds
+
+With the `pipes` example above there was later a new release of
+`pipes-safe` that required the **newer** version of `pipes`. You can
+add that package to the same upper bounds section,
+(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
+as we want to lift them simultaneously.
+
+### Skipping tests and benchmarks
+
+Sometimes tests and benchmark dependencies are forgotten or not cared
+for. To disable compilation for them add them to `skipped-tests` or
+`skipped-benchmarks`. If a package is added to these sections they
+won't be compiled, and their dependencies won't be taken into account.
+
+There are sub sections under these headers that is used to group types
+of failures together, and also to document what type of failures
+exist.
+
+### Expecting test/benchmark/haddock failures
+
+The difference from the `skipped` sections is that items listed here
+are compiled and their dependencies are taken into account. These
+sections also have sub sections with groups and descriptions.
+
+One big category of test suites in this section are those requiring
+running services. We don't want to run those, but we do want to check
+dependencies and compile them.
+
+If there are no version bounds that would fix the issue or if you
+can't figure it out, file it
+(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
+maintainer for help.
+
+### Waiting for new releases
+
+Sometimes there is a failure reported on a (now possibly closed) issue
+on an external tracker. If an issue gets resolved but there is no
+hackage release yet we'd like to get notified when it's uploaded.
+
+Add the package with its current version to the
+`tell-me-when-its-released` section. This will cause the build to stop
+when the new version is out.
+
+### Excluding packages
+
+In an extreme case of a non-responsive maintainer, you can remove the
+package entirely from Stackage. We try to avoid that whenever
+possible.
+
+This typically happens when we move to a new major GHC release or when
+there are only a few packages waiting for updates on an upper bounds
+issue.
+
+Comment out the offending packages from the "packages" section and add
+a comment saying why it was disabled:
+
+```
+        # - swagger # bounds: aeson 1.0
+```
+
+If this causes reverse dependencies to be disabled we should notify
+the maintainers of those packages.
+
 
 ## Updating the content of the Docker image used for building
 
 ### Adding Debian packages for required system tools or libraries
 Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
-Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
+Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
 since new packages are not added to the current lts release.
 
 Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
 
-Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
+Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
+It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
+
+If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
 
 ### Upgrading GHC version
 The Dockerfile contains information on which GHC versions should be used. You
@@ -86,7 +222,7 @@ You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
 ```
 Host stackage-build
     User curators
-    Hostname ec2-52-5-20-252.compute-1.amazonaws.com
+    Hostname build.stackage.org
 ```
 
 ### Running the build script
@@ -106,7 +242,7 @@ we're just not there yet.
 /opt/stackage-build/stackage/automated/build.sh lts-3.0
 ```
 
-Recommended: run these from inside a `screen` session. If you get version bound
+Recommended: run these from inside a `tmux` session. If you get version bound
 problems on nightly or LTS major, you need to fix build-constraints.yaml (see
 info above). For an LTS minor bump, you'll typically want to use the
 `CONSTRAINTS` environment variable, e.g.:
@@ -126,13 +262,13 @@ If a build fails for bounds reasons, see all of the advice above. If the code
 itself doesn't build, or tests fail, open up an issue and then either put in a
 version bound to avoid that version or something else. It's difficult to give
 universal advice on how to solve things, since each situation is unique. Let's
-develop this advice over time. For now: if you're not sure, ask Michael for
+develop this advice over time. For now: if you're not sure, ask for guidance.
-guidance.
 
 __`NOPLAN=1`__ If you wish to rerun a build without recalculating a
 build plan, you can set the environment variable `NOPLAN=1`. This is
 useful for such cases as an intermittent test failure, out of memory
-condition, or manually tweaking the plan file.
+condition, or manually tweaking the plan file. This is the default for
+LTS builds.
 
 ### Timing
 
@@ -146,9 +282,58 @@ LTS minor bumps typically are run on Sundays.
 ### Website sync debugging (and other out of disk space errors)
 
 * You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
-* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
+* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
-* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
+* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
 
 ### Wiping the cache
 
-Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
+Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
+You can wipe the nightly cache and rebuild everything by doing
+`rm -rf /var/stackage/stackage/automated/nightly`.
+Replace nightly with `lts7` to wipe the LTS 7 cache.
+
+### Force a single package rebuild
+
+You can force a single package to rebuild by deleting its "previous result"
+file, e.g.:
+
+```
+$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
+```
+
+## Local curator setup
+
+We do not run the full stackage build locally as that might take too
+much time. However, some steps on the other hand are much faster to do
+yourself, e.g. verifying constraints without building anything.
+
+To get started, install `stackage-curator` via Git, or [the Linux binary]:
+
+```
+$ git clone git@github.com:fpco/stackage-curator.git
+$ cd stackage-curator && stack install
+```
+
+It is a good idea to upgrade `stackage-curator` at the start of your week.
+Then, clone the stackage repo, get the latest packages and run dependency
+resolution:
+
+```
+$ git clone git@github.com:fpco/stackage.git
+$ stack update && stackage-curator check
+```
+
+This can be used to make sure all version bounds are in place, including for
+test suites and benchmarks, to check whether bounds can be lifted, and to get
+[tell-me-when-its-released] notifications.
+
+`stackage-curator` does not build anything, so you wont see any compilation
+errors for builds, tests and benchmarks.
+
+[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
+[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
+
+## Adding new curators
+
+1. Add public ssh key to `~/.ssh/authorized_keys` on build server
+2. Add to fpco/stackage project.

automated/build.sh

@@ -77,7 +77,7 @@ rm -f stackage-curator stackage-curator.bz2
 wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
 bunzip2 stackage-curator.bz2
 chmod +x stackage-curator
-stackage-curator --version
+./stackage-curator --version
 )
 
 ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
@@ -121,7 +121,7 @@ docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-fil
 # Now do the actual build. We need to first set the owner of the home directory
 # correctly, so we run the command as root, change owner, and then use sudo to
 # switch back to the current user
-docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
+docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
 
 # Make sure we actually need this snapshot. We used to perform this check
 # exclusively before building. Now we perform it after as well for the case of

automated/cron.sh

@@ -13,8 +13,8 @@ docker pull $IMAGE
 
 stack update
 
+date
 echo "Running stackage-server-cron..."
-echo "('tail -f $CRONDIR/stackage-server-cron.log' to watch)"
 docker run --rm \
     -v $CRONDIR:/home/ubuntu \
     -v $HOME/.stack/indices:/home/ubuntu/.stack/indices:ro \

debian-bootstrap.sh

@@ -56,6 +56,7 @@ apt-get install -y \
     libedit-dev \
     libedit2 \
     libfftw3-dev \
+    libflac-dev \
     libfreenect-dev \
     libgd2-xpm-dev \
     libgeoip-dev \