haskell/stackage
1
0
Fork 0
mirror of https://github.com/commercialhaskell/stackage.git synced 2026-01-11 23:08:30 +01:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Merge branch 'master' into master

Browse Source
This commit is contained in:
John MacFarlane 2017-01-25 17:04:50 +01:00 committed by GitHub
commit a271647215
10 changed files with 1073 additions and 599 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.travis.yml
View File

@ -2,7 +2,7 @@
sudo: false
# Choose a lightweight base image; we provide our own build tools.
language: c
language: generic
# GHC depends on GMP. You can add other dependencies here as well.
addons:
@ -11,7 +11,7 @@ addons:
- libgmp-dev
env:
- GHCVER=8.0.1
- GHCVER=8.0.2
install:
# Download and unpack the stack executable

238
CURATORS.md
View File

@ -4,7 +4,7 @@ Originally this was handled largely by Michael Snoyman,
but now we are a team of 4 people handling requests weekly in rotation.
Curation activities are mostly automated, and do not take up a significant amount of time.
## Workflow
## Workflow overview
This section sketches out at a high level how the entire Stackage build/curation
process works:
@ -23,10 +23,30 @@ process works:
The typical story on pull requests is: If Travis accepts it and the
author only added packages under his/her own name, merge it. If the
build later fails (see below), then block the package until it's
fixed.
build later fails (see "Adding Debian packages for required system tools or libraries"),
then block the package until it's fixed.
If benchmarks, haddocks, or test suites fails at this point we
typically also block the package until these issues are fixed. This in
order to add packages with a clean slate.
Optionally we can check if packdeps says the package is up to date.
Visit http://packdeps.haskellers.com/feed?needle=<package-name>
Builds may fail because of unrelated bounds changes. If this happens,
first add any version bounds to get master into a passing state (see
"Fixing bounds issues"), then re-run the travis build.
A common issue is that authors submit newly uploaded packages, it can
take up to an hour before this has synced across the stack
infrastructure. You can usually compare the versions of the package in
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
to what's on hackage to see if this is the case. Wait an hour and
re-run the pull request.
Tests also commonly fail due to missing test files, and sometimes due
to doctest limitations. You can point the maintainer to
https://github.com/bergmark/blog/blob/master/2016/package-faq.md
## Fixing bounds issues
@ -36,20 +56,136 @@ issue on the Stackage repo about the problem, and modifying the
build-constraints.yaml file to work around it in one of the ways below. Be sure
to refer to the issue for workarounds added to that file.
* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
### Temporary upper bounds
Most common technique, just prevent a new version of a library from
being included immediately. This also applies to when only benchmarks
and tests are affected.
* Copy the stackage-curator output and create a new issue, see e.g
https://github.com/fpco/stackage/issues/2108
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
```yaml
"Stackage upper bounds":
# https://github.com/fpco/stackage/issues/2108
- pipes < 4.3.0
```
* Commit (message e.g. "Upper bound for #2108")
* Optionally: Verify with `stackage-curator check` locally
* Push
* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
Sometimes releases for different packages are tightly coupled. Then it
can make sense to combine them into one issue, as in
https://github.com/fpco/stackage/issues/2143.
If a dependency that is not explicitly in stackage is causing test or
benchmark failures you can skip or expect them to fail (see "Skipping
tests and benchmarks" and "Expecting test/benchmark/haddock
failures"). Bonus points for reporting this upstream to that packages'
maintainer.
### Lifting upper bounds
You can try this when you notice that a package has been updated. You
can also periodically try to lift bounds (I think it's good to do this
at the start of your week /@bergmark)
If not all packages have been updated check if any of them are missing
from the original issue and if so add a new comment mentioning them. A
new package may appear if its dependencies were part of this issue but
have been updated since the last time we checked. We want to give
these new packages ample time to be upgraded.
If stackage-curator is happy commit the change ("Remove upper bounds
and close #X"). After doing this the next nightly build may fail
because some packages didn't have an upper bound in place, but
compilation failed. In this case revert the previous commit so any
disabled packages are enabled again, re-open the issue, and add a new
comment with the failing packages. This is to give all maintainers
enough time to upgrade for this case as well.
### Amending upper bounds
With the `pipes` example above there was later a new release of
`pipes-safe` that required the **newer** version of `pipes`. You can
add that package to the same upper bounds section,
(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
as we want to lift them simultaneously.
### Skipping tests and benchmarks
Sometimes tests and benchmark dependencies are forgotten or not cared
for. To disable compilation for them add them to `skipped-tests` or
`skipped-benchmarks`. If a package is added to these sections they
won't be compiled, and their dependencies won't be taken into account.
There are sub sections under these headers that is used to group types
of failures together, and also to document what type of failures
exist.
### Expecting test/benchmark/haddock failures
The difference from the `skipped` sections is that items listed here
are compiled and their dependencies are taken into account. These
sections also have sub sections with groups and descriptions.
One big category of test suites in this section are those requiring
running services. We don't want to run those, but we do want to check
dependencies and compile them.
If there are no version bounds that would fix the issue or if you
can't figure it out, file it
(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
maintainer for help.
### Waiting for new releases
Sometimes there is a failure reported on a (now possibly closed) issue
on an external tracker. If an issue gets resolved but there is no
hackage release yet we'd like to get notified when it's uploaded.
Add the package with its current version to the
`tell-me-when-its-released` section. This will cause the build to stop
when the new version is out.
### Excluding packages
In an extreme case of a non-responsive maintainer, you can remove the
package entirely from Stackage. We try to avoid that whenever
possible.
This typically happens when we move to a new major GHC release or when
there are only a few packages waiting for updates on an upper bounds
issue.
Comment out the offending packages from the "packages" section and add
a comment saying why it was disabled:
```
# - swagger # bounds: aeson 1.0
```
If this causes reverse dependencies to be disabled we should notify
the maintainers of those packages.
## Updating the content of the Docker image used for building
### Adding Debian packages for required system tools or libraries
Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
since new packages are not added to the current lts release.
Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
### Upgrading GHC version
The Dockerfile contains information on which GHC versions should be used. You
@ -86,7 +222,7 @@ You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
```
Host stackage-build
User curators
Hostname ec2-52-5-20-252.compute-1.amazonaws.com
Hostname build.stackage.org
```
### Running the build script
@ -106,7 +242,7 @@ we're just not there yet.
/opt/stackage-build/stackage/automated/build.sh lts-3.0
```
Recommended: run these from inside a `screen` session. If you get version bound
Recommended: run these from inside a `tmux` session. If you get version bound
problems on nightly or LTS major, you need to fix build-constraints.yaml (see
info above). For an LTS minor bump, you'll typically want to use the
`CONSTRAINTS` environment variable, e.g.:
@ -126,13 +262,13 @@ If a build fails for bounds reasons, see all of the advice above. If the code
itself doesn't build, or tests fail, open up an issue and then either put in a
version bound to avoid that version or something else. It's difficult to give
universal advice on how to solve things, since each situation is unique. Let's
develop this advice over time. For now: if you're not sure, ask Michael for
guidance.
develop this advice over time. For now: if you're not sure, ask for guidance.
__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
build plan, you can set the environment variable `NOPLAN=1`. This is
useful for such cases as an intermittent test failure, out of memory
condition, or manually tweaking the plan file.
condition, or manually tweaking the plan file. This is the default for
LTS builds.
### Timing
@ -146,9 +282,79 @@ LTS minor bumps typically are run on Sundays.
### Website sync debugging (and other out of disk space errors)
* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
### Wiping the cache
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
You can wipe the nightly cache and rebuild everything by doing
`rm -rf /var/stackage/stackage/automated/nightly`.
Replace nightly with `lts7` to wipe the LTS 7 cache.
### Force a single package rebuild
You can force a single package to rebuild by deleting its "previous result"
file, e.g.:
```
$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
```
## Local curator setup
We do not run the full stackage build locally as that might take too
much time. However, some steps on the other hand are much faster to do
yourself, e.g. verifying constraints without building anything.
To get started, install `stackage-curator` via Git, or [the Linux binary]:
```
$ git clone git@github.com:fpco/stackage-curator.git
$ cd stackage-curator && stack install
```
It is a good idea to upgrade `stackage-curator` at the start of your week.
Then, clone the stackage repo, get the latest packages and run dependency
resolution:
```
$ git clone git@github.com:fpco/stackage.git
$ stack update && stackage-curator check
```
This can be used to make sure all version bounds are in place, including for
test suites and benchmarks, to check whether bounds can be lifted, and to get
[tell-me-when-its-released] notifications.
`stackage-curator` does not build anything, so you wont see any compilation
errors for builds, tests and benchmarks.
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
## Adding new curators
1. Add public ssh key to `~/.ssh/authorized_keys` on build server
2. Add to fpco/stackage project.
## Dealing with a new GHC release
As mentioned in the [GHC upgrade note], the major impact of a new GHC release
is on the packages that are causing upper bounds to be put in place. In order
to minimise out-of-date breakage and allow maintainers to have a solid chance
of getting their packages into the newest LTS, we try to do the following:
Make an early announcement (in the form of a blog post, typically) of the new
GHC release on the nightly build and the planned deadline for the new LTS release.
Make it clear, that in the time coming up to this, we hope package maintainers
will upgrade their packages to allow for the new GHC release.
We prefer to prune packages causing upper bounds constraints **after** the LTS
release to allow the maximum amount of packages to get into the newest LTS.
After the first LTS release, the package pruning process may begin in order to
move forward with getting the latest versions of packages compatible with the
new GHC release.
[GHC upgrade note]: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version

4
Dockerfile
View File

@ -1,8 +1,8 @@
FROM ubuntu:16.04
FROM fpco/pid1:16.04
ENV HOME /home/stackage
ENV LANG en_US.UTF-8
ENV PATH /opt/ghc/8.0.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV PATH /opt/ghc/8.0.2/bin:/usr/sbin:/usr/bin:/sbin:/bin
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh

2
MAINTAINERS.md
View File

@ -24,7 +24,7 @@ After doing that, send a pull request (with a commit message like "add foo-bar")
If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
```
$ cabal update
$ ghc --version # Should give v8.0.1
$ ghc --version # Should give v8.0.2
$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
$ cd PACKAGE-VERSION
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".

33
automated/build.sh
View File

@ -1,6 +1,6 @@
#!/usr/bin/env bash
set -eux
set -eu +x
ROOT=$(cd $(dirname $0) ; pwd)
TARGET=$1
@ -77,6 +77,7 @@ rm -f stackage-curator stackage-curator.bz2
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
bunzip2 stackage-curator.bz2
chmod +x stackage-curator
./stackage-curator --version
)
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
@ -96,22 +97,31 @@ fi
# Get latest stack
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
# Do all of the pre-build actions:
# Determine the new build plan unless NOPLAN is set
#
# * Update the package index
# * Update the package index (unless LTS)
# * Create a new plan
if [ "${NOPLAN:-}x" = "x" ]
then
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
fi
fi
# Do the rest of the pre-build actions:
#
# * Check that the plan is valid
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
# * Do a single unpack to create the package index cache (again due to directory perms)
if [ "${NOPLAN:-}x" = "x" ]
then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-} && stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
fi
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
# Now do the actual build. We need to first set the owner of the home directory
# correctly, so we run the command as root, change owner, and then use sudo to
# switch back to the current user
docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs 4 --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
# Make sure we actually need this snapshot. We used to perform this check
# exclusively before building. Now we perform it after as well for the case of
@ -128,11 +138,4 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-
# * Register as a new Hackage distro
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
if [ $SHORTNAME = "lts" ]
then
echo "Running cron.sh (hiding verbose output)"
./cron.sh | grep -v '^Skipping'
echo "done."
fi
date

15
automated/cron.sh
View File

@ -2,14 +2,23 @@
set -eux
cd "$(dirname "${BASH_SOURCE[0]}")"
CRONDIR=$(pwd)/crondir
mkdir -p $CRONDIR
source aws.sh
IMAGE=fpco/stackage-server-prod:latest
docker pull $IMAGE
stack update
date
echo "Running stackage-server-cron..."
echo "('tail -f $CRONDIR/stackage-server-cron.log' to watch)"
docker run --rm \
-v $CRONDIR:/home/ubuntu \
-w /home/ubuntu \
fpco/stackage-server-prod:latest \
-v $HOME/.stack/indices:/home/ubuntu/.stack/indices:ro \
--workdir /home/ubuntu \
-p 17834:17834 \
$IMAGE \
bash -c "useradd $(whoami) -u $(id -u); sudo -u $(whoami) env HOME=/home/ubuntu AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY bash -c '/usr/local/bin/stackage-server-cron 2>&1 | tee -a /home/ubuntu/stackage-server-cron.log'"

7
automated/run-nightly.sh
View File

@ -4,12 +4,7 @@ cd "$(dirname "${BASH_SOURCE[0]}")"
while true; do
./build.sh nightly-$(date -u +%F)
echo "Running cron.sh (hiding verbose output)"
./cron.sh > cron.log 2>&1 # | grep -v '^Skipping'
echo "done."
echo
date
sleep 30m
echo
done

1358
build-constraints.yaml
View File

File diff suppressed because it is too large Load Diff

2
check
View File

@ -3,4 +3,4 @@
# Convenience script for checking constraints locally
cd `dirname $0`
exec stack exec --resolver ghc-8.0.1 stackage-curator check
exec stack exec --resolver ghc-8.0.2 stackage-curator check

9
debian-bootstrap.sh
View File

@ -25,7 +25,7 @@ add-apt-repository -y ppa:marutter/rrutter
#add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse
# Set the GHC version
GHCVER=8.0.1
GHCVER=8.0.2
# Get Stack
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 575159689BEFB442
@ -45,6 +45,7 @@ apt-get install -y \
git \
libadns1-dev \
libaio1 \
libalut-dev \
libasound2-dev \
libblas-dev \
libbz2-dev \
@ -55,6 +56,7 @@ apt-get install -y \
libedit-dev \
libedit2 \
libfftw3-dev \
libflac-dev \
libfreenect-dev \
libgd2-xpm-dev \
libgeoip-dev \
@ -83,6 +85,7 @@ apt-get install -y \
libmpfr-dev \
libmysqlclient-dev \
libncurses-dev \
libnfc-dev \
libnotify-dev \
libopenal-dev \
libpango1.0-dev \
@ -161,3 +164,7 @@ cd /tmp \
&& echo "/usr/local/lib" > /etc/ld.so.conf.d/usr-local.conf \
&& echo "/usr/lib/oracle/12.1/client64/lib" > /etc/ld.so.conf.d/oracle-client.conf \
&& ldconfig
# Add JDK to system paths.
echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
&& ldconfig