Merge branch 'master' into master

This commit is contained in:
Jens Petersen 2019-10-04 17:51:51 +08:00 committed by GitHub
commit 614efd8442
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 3828 additions and 1102 deletions

View File

@ -6,3 +6,5 @@ cabal.sandbox.config
tarballs
*.yaml
.git
automated
.github

View File

@ -1,9 +1,8 @@
Checklist:
- [ ] Meaningful commit message - please not `Update build-constraints.yml`
- [ ] At least 30 minutes have passed since Hackage upload
- [ ] On your own machine, in a new directory, you have successfully run the following set of commands (replace `$package` with the name of the package that is submitted, `$version` is the version of the package you want to get into Stackage):
- [ ] Meaningful commit message, eg `add my-cool-package` (please not mention `build-constraints.yml`)
- [ ] At least 30 minutes have passed since uploading to Hackage
- [ ] On your own machine, in a _new directory_, you have successfully run the following set of commands (replace `$package` with the name of the package that is submitted, and `$version` with the version of the package you want to get into Stackage):
stack unpack $package-$version
cd $package-$version
stack unpack $package-$version # $version is optional
stack init --resolver nightly
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks

4
.gitignore vendored
View File

@ -4,3 +4,7 @@ nightly-*.yaml
lts-*.yaml
*.swp
check-plan.yaml
*~
/constraints.yaml
/snapshot.yaml
/snapshot-incomplete.yaml

View File

@ -1,5 +1,5 @@
# Use new container infrastructure to enable caching
sudo: false
# Disable Docker stuff to try to make it faster
sudo: true
# Choose a lightweight base image; we provide our own build tools.
language: generic
@ -10,31 +10,17 @@ addons:
packages:
- libgmp-dev
env:
- GHCVER=8.4.3
install:
# Download and unpack the stack executable
- mkdir -p ~/.local/bin
- export PATH=$HOME/.local/bin:$PATH
- travis_retry curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'
# Get stackage-curator
- wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
- bunzip2 stackage-curator.bz2
- chmod +x stackage-curator
- mv stackage-curator ~/.local/bin
# Install GHC and cabal-install
- stack setup $GHCVER
# Update the index
- travis_retry stack update
matrix:
include:
#- env: CHECK_SCRIPT="./etc/ci-script.sh"
- env: CHECK_SCRIPT="./etc/curator-2-check.sh"
# Edit etc/ci-script.sh to change GHC version
script:
- stack --resolver ghc-$GHCVER exec stackage-curator check
- $CHECK_SCRIPT
cache:
directories:
- $HOME/.stack
- $HOME/.stackage/curator/cache
# Caching doesn't work with sudo: true
#cache:
# directories:
# - $HOME/.stack
# - $HOME/.stackage/curator/cache

View File

@ -18,13 +18,13 @@ This section sketches out at a high level how the entire Stackage build/curation
process works:
* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
* stackage-curator can check that build plan to ensure all version bounds are consistent
* [curator](https://github.com/commercialhaskell/stack/tree/master/subs/curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
* `curator` can check that build plan to ensure all version bounds are consistent
* The [Travis job](https://github.com/commercialhaskell/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
* Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
* Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://hub.docker.com/r/commercialhaskell/stackage/) for running builds
* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh)
* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/commercialhaskell/stackage-nightly)
* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/commercialhaskell/lts-haskell)
* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
## Pull requests
@ -72,7 +72,7 @@ Most common technique, just prevent a new version of a library from
being included immediately. This also applies to when only benchmarks
and tests are affected.
* Copy the stackage-curator output and create a new issue, see e.g
* Copy the `curator` output and create a new issue, see e.g
https://github.com/commercialhaskell/stackage/issues/2108
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
@ -84,7 +84,7 @@ https://github.com/commercialhaskell/stackage/issues/2108
```
* Commit (message e.g. "Upper bound for #2108")
* Optionally: Verify with `stackage-curator check` locally
* Optionally: Verify with `./check` locally
* Push
* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
@ -110,7 +110,7 @@ new package may appear if its dependencies were part of this issue but
have been updated since the last time we checked. We want to give
these new packages ample time to be upgraded.
If stackage-curator is happy commit the change ("Remove upper bounds
If `curator` is happy commit the change ("Remove upper bounds
and close #X"). After doing this the next nightly build may fail
because some packages didn't have an upper bound in place, but
compilation failed. In this case revert the previous commit so any
@ -140,8 +140,9 @@ exist.
### Expecting test/benchmark/haddock failures
The difference from the `skipped` sections is that items listed here
are compiled and their dependencies are taken into account. These
sections also have sub sections with groups and descriptions.
are compiled and their dependencies are taken into account
(but they are allowed to fail to build).
These sections also have subsections with groups and descriptions.
One big category of test suites in this section are those requiring
running services. We don't want to run those, but we do want to check
@ -215,6 +216,11 @@ Hub to create a new Docker tag for the relevant branch name.
You'll need to update both the `PATH` in `Dockerfile` and the `GHCVER` variable
in `debian-bootstrap.sh`.
Ensure that the [global-hints.yaml
file](https://github.com/fpco/stackage-content/blob/master/stack/global-hints.yaml)
is updated with information on the latest GHC release by cloning that
repo and running `./update-global-hints.yaml ghc-X.Y.Z`.
### Getting the new image to the build server
Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
below). Instead of pulling an unbounded number of images, I typically just
@ -225,7 +231,7 @@ docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
```
but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
but `docker pull commercialhaskell/stackage:nightly` can also be run instead just to update the nightly image say.
For a new GHC version you should also delete the cache directories on the stackage-build server to
force all packages to be rebuilt. See: [issue#746](https://github.com/commercialhaskell/stackage/issues/746). Eg:
@ -236,14 +242,16 @@ This should also be done when moving the Nightly docker image to a new version o
If you're impatient and would like to build the Docker image on the
build server instead of waiting for Docker Hub, you can run the
following command:
following command (replacing `BRANCH=nightly` if the image for a different branch is desired):
```
BRANCH=nightly
DIR=$(mktemp -d)
(cd $DIR \
&& git clone https://github.com/commercialhaskell/stackage \
&& cd stackage \
&& docker build --tag snoyberg/stackage:nightly .)
&& git checkout $BRANCH
&& docker build --tag commercialhaskell/stackage:$BRANCH .)
rm -rf $DIR
```
@ -307,7 +315,10 @@ develop this advice over time. For now: if you're not sure, ask for guidance.
__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
build plan, you can set the environment variable `NOPLAN=1`. This is
useful for such cases as an intermittent test failure, out of memory
condition, or manually tweaking the plan file.
condition, or manually tweaking the plan file. (When using `NOPLAN=1`,
if one needs to revert one package, say due to a build or test regression,
one can edit `current-plan.yaml` and updated the SHA256 hash of the .cabal file,
to avoid having to rebuild everything again.)
Note LTS builds inherit the current Hackage data (stack updated for Nigthly) to avoid excess extra rebuilding.
@ -342,36 +353,44 @@ file, e.g.:
$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
```
### Restarting docker
If docker hangs with e.g. `docker: Error response from daemon:
connection error: desc = "transport: dial unix
/var/run/docker/containerd/docker-containerd .sock: connect:
connection refused".` you can restart it with `sudo systemctl restart
docker.service`.
## Local curator setup
We do not run the full stackage build locally as that might take too
much time. However, some steps on the other hand are much faster to do
yourself, e.g. verifying constraints without building anything.
To get started, install `stackage-curator` via Git, or [the Linux binary]:
To get started, install `curator` via Git:
```
$ git clone git@github.com:fpco/stackage-curator.git
$ cd stackage-curator && stack install
$ git clone git@github.com:commercialhaskell/curator.git
$ cd curator && stack install curator
```
It is a good idea to upgrade `stackage-curator` at the start of your week.
It is a good idea to upgrade `curator` at the start of your week.
Then, clone the stackage repo, get the latest packages and run dependency
resolution:
```
$ git clone git@github.com:commercialhaskell/stackage.git
$ stack update && stackage-curator check
$ cd stackage
$ ./check
```
This can be used to make sure all version bounds are in place, including for
test suites and benchmarks, to check whether bounds can be lifted, and to get
[tell-me-when-its-released] notifications.
`stackage-curator` does not build anything, so you wont see any compilation
`curator` does not build anything, so you wont see any compilation
errors for builds, tests and benchmarks.
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
[tell-me-when-its-released]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#waiting-for-new-releases
## Adding new curators

View File

@ -1,12 +1,15 @@
FROM fpco/pid1:16.04
FROM fpco/pid1:18.04
ENV HOME /home/stackage
ENV LANG en_US.UTF-8
ENV GHCVER 8.8.1
# NOTE: also update debian-bootstrap.sh when cuda version changes
ENV PATH /usr/local/cuda-8.0/bin:/opt/ghc/8.4.3/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV CUDA_PATH /usr/local/cuda-8.0
ENV LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/nvvm/lib64
ENV PATH /home/stackage/.stack/programs/x86_64-linux/ghc-$GHCVER/bin:/usr/local/cuda-10.0/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV CUDA_PATH /usr/local/cuda-10.0
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh
# Include file path
ENV CPATH /usr/lib/jvm/java-8-openjdk-amd64/include:/usr/lib/jvm/java-8-openjdk-amd64/include/linux:/usr/lib/llvm-3.7/include

View File

@ -21,6 +21,9 @@ To add your package you can edit [`build-constraints.yaml`](https://github.com/f
If your library depends on a C library, please add it to the `debian-bootstrap.sh` script.
Any dependencies of your packages that are not already part of stackage are added implictly, but it is prefered
to add all packages explicitly. It is planned to remove this behaviour in the future.
After doing that commit with a message like "add foo-bar" and send a pull request.
The continuous integration job will do some checks to see if your package's dependencies are up-to-date.
@ -32,7 +35,7 @@ If you want to be proactive or if CI fails, you can make sure that your package
# Build from the tarball on Hackage to check for missing files
$ stack unpack yourpackage && cd yourpackage-*
# Generate a pristine stack.yaml, adding any missing extra-deps
$ rm -f stack.yaml && stack init --resolver nightly --solver
$ rm -f stack.yaml && stack init --resolver nightly
# Build, generate docs, test, and build benchmarks
$ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
```
@ -178,7 +181,7 @@ release (e.g., if `lts-8.9` is out, you would want your package to appear in
* Be patient! The LTS releases are less frequent than Nightly. The
Stackage curators will try to get to your issue as soon as possible,
but it may take some time.
* We gradually stop maintainng old LTS major versions, so your
* We gradually stop maintaining old LTS major versions, so your
request may take longer or be declined if it's for an old LTS.
## LTS package guarantees and exceptions
@ -201,7 +204,9 @@ purely on Stackage Curator discretion. The most common examples are:
considered a major version bump, and disallowed in an LTS minor
version bump. However, if a package is following SemVer, this
would not be a breaking change, and curators may elect to include
it.
it. Note though that curators and their tooling will not know your
package is following SemVer, so in this case you would have to open
an issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new).
* If a package has overly restrictive version bounds on a
dependency, in particular constraining a minor version

View File

@ -2,13 +2,14 @@ stackage
========
[![Build Status](https://travis-ci.org/commercialhaskell/stackage.svg?branch=master)](https://travis-ci.org/commercialhaskell/stackage)
[![Join the chat at https://gitter.im/commercialhaskell/stackage](https://badges.gitter.im/commercialhaskell/stackage.svg)](https://gitter.im/commercialhaskell/stackage?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
"Stable Hackage": creating a vetted set of packages from Hackage.
This repository is for package authors and maintainers to get their packages into Stackage.
If you simply want to use Stackage as an end user, please follow the instructions on [https://www.stackage.org/](https://www.stackage.org).
We strongly recommend using the Haskell tool stack for doing builds, which
includes built-in Stackage support: [stack](https://github.com/commercialhaskell/stack) [![Build Status](https://travis-ci.org/commercialhaskell/stack.svg?branch=master)](https://travis-ci.org/commercialhaskell/stack).
includes built-in Stackage support: [stack](https://github.com/commercialhaskell/stack) [![Build Status](https://dev.azure.com/commercialhaskell/stack/_apis/build/status/commercialhaskell.stack?branchName=master)](https://dev.azure.com/commercialhaskell/stack/_build/latest?definitionId=1?branchName=master).
Add your package

View File

@ -1,8 +0,0 @@
{-
NOTE: This module is no longer used for tracking packages included in Stackage.
To simplify the codebase, that information is now stored in the
build-constraints.yaml configuration file. The file should be self-explanatory.
Sorry for the inconvenience.
-}

171
automated/build-next.sh Executable file
View File

@ -0,0 +1,171 @@
#!/usr/bin/env bash
set -eu +x -o pipefail
ROOT=$(cd $(dirname $0) ; pwd)
TARGET=$1
source aws.sh
# For nightly-YYYY-MM-DD, tag should be nightly
# For lts-X.Y, tag should be ltsX
SHORTNAME=$(echo $TARGET | cut -d- -f 1)
if [ $SHORTNAME = "lts" ]
then
TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@')
else
TAG=$SHORTNAME
fi
IMAGE=commercialhaskell/stackage:$TAG
CABAL_DIR=$ROOT/cabal
PANTRY_DIR=$ROOT/pantry
STACK_DIR=$ROOT/stack-$TAG
GHC_DIR=$ROOT/ghc
DOT_STACKAGE_DIR=$ROOT/dot-stackage
WORKDIR=$ROOT/$TAG/work
# ssh key is used for committing snapshots (and their constraints) to Github
SSH_DIR=$ROOT/ssh
USERID=$(id -u)
mkdir -p \
"$CABAL_DIR" \
"$PANTRY_DIR" \
"$STACK_DIR" \
"$GHC_DIR" \
"$DOT_STACKAGE_DIR" \
"$WORKDIR" \
"$SSH_DIR"
GITCONFIG=$ROOT/gitconfig
cat >$GITCONFIG <<EOF
[user]
email = michael+stackage-build@fpcomplete.com
name = Stackage Build host
EOF
HACKAGE_CREDS=$ROOT/hackage-creds
function require_400_file {
if [ ! -f "$1" ]
then
echo File not found: "$1"
exit 1
fi
chmod 400 "$1"
}
require_400_file "$SSH_DIR/id_rsa"
require_400_file "$HACKAGE_CREDS"
mkdir -p $ROOT/bin
BINDIR=$(cd $ROOT/bin ; pwd)
(
# See etc/curator-exes/README.md
CURATOR_EXES=95a207fb4d5bd2e2a255350ce18f55976344eeb71e6f27a25f63e8dba28a4cd1
cd $BINDIR
rm -f curator stack *.bz2
wget "https://s3.amazonaws.com/download.fpcomplete.com/curator-exes/curator-exes-$CURATOR_EXES.tar.bz2"
tar xf "curator-exes-$CURATOR_EXES.tar.bz2"
echo curator: $(./curator --version)
echo stack: $(./stack --version)
)
# We share pantry directory between snapshots while the other content in .stack
# is stored separately (because e.g. Ubuntu releases between LTS and nightly
# could differ). Also the order of binds is important.
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/curator:/usr/bin/curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $BINDIR/stack:/usr/bin/stack:ro -v $STACK_DIR:$HOME/.stack -v $PANTRY_DIR:$HOME/.stack/pantry"
ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $GHC_DIR:$HOME/.ghc:ro"
# instance-data is an undocumented feature of S3 used by amazonka,
# see https://github.com/brendanhay/amazonka/issues/271
ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY"
# Make sure we actually need this snapshot. We only check this for LTS releases
# since, for nightlies, we'd like to run builds even if they are unnecessary to
# get early warning information of upcoming failures. (See the duplicate check
# below for why this is safe.)
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator check-target-available --target $TARGET"
fi
# Determine the new build plan unless NOPLAN is set
#
# * Update the package index (unless LTS)
# * Create a new plan
if [ "${NOPLAN:-}x" = "x" ]
then
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator constraints --target $TARGET && curator snapshot-incomplete --target $TARGET && curator snapshot"
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator update && curator constraints --target $TARGET && curator snapshot-incomplete --target $TARGET && curator snapshot"
fi
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator snapshot"
fi
# Do the rest of the pre-build actions:
#
# * Check that the snapshot is valid
# * Fetch and unpack all needed tarballs (the build step does not have write access to the tarball directory)
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c 'GHCVER=$(sed -n "s/^ghc-version: \(.*\)/\1/p" constraints.yaml) && stack setup ghc-$GHCVER --verbosity=error && stack exec --resolver=ghc-$GHCVER curator check-snapshot && curator unpack'
case $SHORTNAME in
lts) JOBS=1 ;;
nightly) JOBS=2 ;;
esac
# Now do the actual build. We need to first set the owner of the home directory
# correctly, so we run the command as root, change owner, and then use sudo to
# switch back to the current user
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" curator build --jobs $JOBS" 2>&1 | tee "$SHORTNAME-build.log"
# Make sure we actually need this snapshot. We used to perform this check
# exclusively before building. Now we perform it after as well for the case of
# nightly, where we don't perform this check beforehand. This is also slightly
# safer, in case someone else already uploaded a specific snapshot while we
# were building.
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator check-target-available --target $TARGET"
# Successful build, so we need to:
#
# * Upload the docs to S3
# * Upload the new snapshot .yaml file to the appropriate Github repo, also upload its constraints
# * Register as a new Hackage distro (currently disabled)
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "curator upload-docs --target $TARGET && curator upload-github --target $TARGET && exec curator hackage-distro --target $TARGET"
# information about the new snapshots on Hackage
$BINDIR/curator legacy-bulk --stackage-snapshots dot-stackage/curator/stackage-snapshots/ --lts-haskell dot-stackage/curator/lts-haskell/ --stackage-nightly dot-stackage/curator/stackage-nightly/
(
if [ $SHORTNAME = "lts" ]
then
cd dot-stackage/curator/lts-haskell
else
cd dot-stackage/curator/stackage-nightly
fi
git add *.yaml
git diff-index --quiet HEAD && echo No changes && exit 0
git config user.name "Stackage build server"
git config user.email "michael@snoyman.com"
git commit -a -m "More conversions $(date)"
if [ $SHORTNAME = "lts" ]
then
GIT_SSH_COMMAND="ssh -i $ROOT/ssh-lts/id_rsa" git push origin master
else
GIT_SSH_COMMAND="ssh -i $ROOT/ssh-nightly/id_rsa" git push origin master
fi
)
echo -n "Completed at "
date

View File

@ -1,147 +0,0 @@
#!/usr/bin/env bash
set -eu +x
ROOT=$(cd $(dirname $0) ; pwd)
TARGET=$1
source aws.sh
# For nightly-YYYY-MM-DD, tag should be nightly
# For lts-X.Y, tag should be ltsX
SHORTNAME=$(echo $TARGET | cut -d- -f 1)
if [ $SHORTNAME = "lts" ]
then
TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@')
else
TAG=$SHORTNAME
fi
IMAGE=snoyberg/stackage:$TAG
PLAN_FILE=current-plan.yaml
DOCMAP_FILE=current-docmap.yaml
CABAL_DIR=$ROOT/cabal
STACK_DIR=$ROOT/stack
GHC_DIR=$ROOT/ghc
DOT_STACKAGE_DIR=$ROOT/dot-stackage
WORKDIR=$ROOT/$TAG/work
EXTRA_BIN_DIR=$ROOT/extra-bin
SSH_DIR=$ROOT/ssh-$SHORTNAME
USERID=$(id -u)
mkdir -p \
"$CABAL_DIR" \
"$STACK_DIR" \
"$GHC_DIR" \
"$DOT_STACKAGE_DIR" \
"$WORKDIR" \
"$EXTRA_BIN_DIR" \
"$SSH_DIR"
GITCONFIG=$ROOT/gitconfig
cat >$GITCONFIG <<EOF
[user]
email = michael+stackage-build@fpcomplete.com
name = Stackage Build host
EOF
cat >$SSH_DIR/known_hosts <<EOF
|1|Qn0iij8BnxGZXbyFSozS9zWkH+Q=|YrKKNp2KHO3/oc4UBFIe1zOvhDc= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
|1|RxBEt2ljiEppr019szMIhbY12m0=|0FZ2Oji1LphRbPLLEQhFzTmL69I= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
EOF
AUTH_TOKEN=$ROOT/auth-token
HACKAGE_CREDS=$ROOT/hackage-creds
function require_400_file {
if [ ! -f "$1" ]
then
echo File not found: "$1"
exit 1
fi
chmod 400 "$1"
}
require_400_file "$SSH_DIR/id_rsa"
require_400_file "$AUTH_TOKEN"
require_400_file "$HACKAGE_CREDS"
mkdir -p $ROOT/bin
BINDIR=$(cd $ROOT/bin ; pwd)
(
cd $BINDIR
rm -f stackage-curator stackage-curator.bz2
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
bunzip2 stackage-curator.bz2
chmod +x stackage-curator
./stackage-curator --version
)
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
# Make sure we actually need this snapshot. We only check this for LTS releases
# since, for nightlies, we'd like to run builds even if they are unnecessary to
# get early warning information of upcoming failures. (See the duplicate check
# below for why this is safe.)
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
fi
# Get latest stack
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
# Determine the new build plan unless NOPLAN is set
#
# * Update the package index (unless LTS)
# * Create a new plan
if [ "${NOPLAN:-}x" = "x" ]
then
if [ $SHORTNAME = "lts" ]
then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
fi
fi
# Do the rest of the pre-build actions:
#
# * Check that the plan is valid
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
# * Do a single unpack to create the package index cache (again due to directory perms)
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
case $SHORTNAME in
lts) JOBS=1 ;;
nightly) JOBS=2 ;;
esac
# Now do the actual build. We need to first set the owner of the home directory
# correctly, so we run the command as root, change owner, and then use sudo to
# switch back to the current user
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs $JOBS --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET"
# Make sure we actually need this snapshot. We used to perform this check
# exclusively before building. Now we perform it after as well for the case of
# nightly, where we don't perform this check beforehand. This is also slightly
# safer, in case someone else already uploaded a specific snapshot while we
# were building.
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
# Successful build, so we need to:
#
# * Upload the docs to S3
# * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
# * Upload the new plan .yaml file to the appropriate Github repo
# * Register as a new Hackage distro
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
echo -n "Completed at "
date

View File

@ -0,0 +1,35 @@
#!/usr/bin/env bash
set -eux
if [[ ! -f convert ]]
then
curl https://s3.amazonaws.com/www.snoyman.com/convert-old-stackage-c9c4d06b31cb7aafedd23aa316b8008c45e4d4dd.bz2 > convert.bz2
chmod +x convert.bz2
bunzip2 convert.bz2
fi
cd $(dirname ${BASH_SOURCE[0]})
for d in lts-haskell stackage-nightly stackage-snapshots
do
if [[ ! -d "$d" ]]
then
git clone https://github.com/commercialhaskell/$d
else
(cd "$d" && git pull || echo "Git pull failed, ignoring")
fi
done
(
export HOME=$(pwd)
./convert
)
cd stackage-snapshots
git add lts nightly
git diff-index --quiet HEAD && echo No changes && exit 0
git config user.name "Stackage build server"
git config user.email "michael@snoyman.com"
git commit -m "More conversions $(date)"
GIT_SSH_COMMAND='ssh -i ../../ssh-lts/id_rsa' git push git@github.com:commercialhaskell/stackage-snapshots master

View File

@ -0,0 +1,5 @@
# Building Stackage with Nix
The script `build.sh` is supposed to be run in a directory where `curator unpack`
was run - it assumes existence of `unpack-dir` directory with a Stackage
snapshot `stack.yaml` and unpacked packages in it.

View File

@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -eu +x
NIX_VERSION="2.0.4"
curl https://nixos.org/releases/nix/nix-$NIX_VERSION/install | sh
# no code to install stack2nix yet
stack2nix --cabal2nix-args="--no-hpack" unpack-dir > snapshot.nix
nix build -f snapshot-fixed.nix

View File

@ -0,0 +1,23 @@
let
spec = builtins.fromJSON (builtins.readFile ./nixpkgs-src.json);
src = import <nix/fetchurl.nix> {
url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
inherit (spec) sha256;
};
nixcfg = import <nix/config.nix>;
in builtins.derivation {
system = builtins.currentSystem;
name = "${src.name}-unpacked";
builder = builtins.storePath nixcfg.shell;
inherit src;
args = [
(builtins.toFile "builder" ''
$coreutils/mkdir $out
cd $out
$gzip -d < $src | $tar -x --strip-components=1
'')
];
coreutils = builtins.storePath nixcfg.coreutils;
tar = builtins.storePath nixcfg.tar;
gzip = builtins.storePath nixcfg.gzip;
}

View File

@ -0,0 +1,6 @@
{
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "c2950341d038995bf46a7b72db961bb3d3e9ac12",
"sha256": "0ghirjm6xvsihci3fladrn4nrnh2yr529v7c406hmw0yp25kv0i4"
}

View File

@ -0,0 +1,41 @@
{ pkgs ? import (import ./fetch-nixpkgs.nix) {}
, compiler ? pkgs.haskell.packages.ghc863
}:
let
snapshot = import ./snapshot.nix { pkgs = pkgs; compiler = compiler; };
in
snapshot.override (with pkgs.haskell.lib; {
overrides = self: super: {
# disabling packages from non-hackage-packages.nix
futhark = null;
multi-ghc-travis = null;
vaultenv = null;
# native deps
check-email = super.check-email.override { resolv = null; };
flac = super.flac.override { FLAC = pkgs.flac; };
freenect = super.freenect.override { freenect_sync = null; libfreenect = null; };
haskell-gi = super.haskell-gi.override { gobject-introspection = pkgs.gnome3.gobjectIntrospection; };
HDBC-mysql = super.HDBC-mysql.override { mysqlclient = pkgs.mysql;};
libffi = super.libffi.override { ffi = null; };
# fixes
blas-ffi = overrideCabal super.blas-ffi (
old: { libraryPkgconfigDepends = old.libraryPkgconfigDepends ++ [ pkgs.openblasCompat ]; }
);
odbc = overrideCabal super.odbc (
old: { librarySystemDepends = old.librarySystemDepends ++ [ pkgs.freetds ]; }
);
html-entities = null; # https://github.com/nikita-volkov/html-entities/issues/8
category = null; # conflict in Prelude.hs
constraint = null; # because of category
lame = null; # fails with 'Total ticks: 451489'
proto-lens-protobuf-types = overrideCabal super.proto-lens-protobuf-types (
old: { libraryToolDepends = old.libraryToolDepends ++ [ pkgs.protobuf ]; }
);
# libraft = null; # xxx/postgres/entries.sql: openFile: does not exist (No such file or directory) in libraft-0.2.0.0
xmlbf-xeno = null; # because of html-entities
xmonad = null; # xmonad requires extra patches for Nix
xmonad-contrib = null;
xmonad-extras = null;
hlibgit2 = disableHardening super.hlibgit2 [ "format" ];
};
})

View File

@ -3,8 +3,12 @@
cd "$(dirname "${BASH_SOURCE[0]}")"
while true; do
./build.sh nightly-$(date -u +%F)
./build-next.sh nightly-$(date -u +%F)
date
#./new-stackage-format/convert.sh > /dev/null 2> /dev/null
#date
sleep 30m
echo
done

File diff suppressed because it is too large Load Diff

12
check
View File

@ -1,6 +1,14 @@
#!/bin/sh
#!/bin/bash
# Convenience script for checking constraints locally
cd `dirname $0`
exec stack exec --resolver ghc-8.4.3 stackage-curator check
export GHCVER=$(sed -n "s/^ghc-version: \"\(.*\)\"/\1/p" "build-constraints.yaml")
NIGHTLY="nightly-$(date +%Y-%m-%d)"
curator update &&
curator constraints --target=$NIGHTLY &&
curator snapshot-incomplete --target=$NIGHTLY &&
curator snapshot &&
stack --resolver ghc-$GHCVER exec curator check-snapshot

View File

@ -10,38 +10,34 @@
# instructions, see:
# http://www.stackage.org/install
set -exu
set -exuo pipefail
mkdir /home/stackage -p
mkdir -p /home/stackage
export LANG=C.UTF-8
export DEBIAN_FRONTEND=noninteractive
# Get curl
apt-get update
apt-get install -y software-properties-common
apt-get install -y curl
add-apt-repository ppa:hvr/ghc -y
add-apt-repository -y ppa:marutter/rrutter
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy main'
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
GHCVER=8.4.3
# Get Stack and GHC
curl -sSL https://get.haskellstack.org/ | sh -s - -d /usr/bin
stack setup --resolver ghc-$GHCVER
apt-get update
apt-get install -y \
apt-transport-https \
build-essential \
cmake \
curl \
dvipng \
freeglut3-dev \
freetds-dev \
fsharp \
g++ \
gawk \
ghc-$GHCVER \
ghc-$GHCVER-dyn \
ghc-$GHCVER-htmldocs \
ghc-$GHCVER-prof \
git \
gnupg \
gradle \
@ -62,16 +58,16 @@ apt-get install -y \
libfftw3-dev \
libflac-dev \
libfreenect-dev \
libgd2-xpm-dev \
libgd-dev \
libgeoip-dev \
libgirepository1.0-dev \
libglfw3-dev \
libglib2.0-dev \
libglu1-mesa-dev \
libgmp3-dev \
libgnutls-dev \
libgnutls28-dev \
libgsasl7-dev \
libgsl0-dev \
libgsl-dev \
libgtk-3-dev \
libgtk2.0-dev \
libgtksourceview-3.0-dev \
@ -94,23 +90,29 @@ apt-get install -y \
libmp3lame-dev \
libmpfr-dev \
libmysqlclient-dev \
libncurses-dev \
libncurses5-dev \
libnfc-dev \
liboath-dev \
libnotify-dev \
libopenal-dev \
libopenmpi-dev \
libpango1.0-dev \
libpcap0.8-dev \
libpcre2-dev \
libpq-dev \
libprotobuf-dev \
libre2-dev \
librocksdb-dev \
libsdl1.2-dev \
libsdl2-dev \
libsdl2-gfx-dev \
libsdl2-image-dev \
libsdl2-mixer-dev \
libsdl2-ttf-dev \
libsecp256k1-dev \
libsnappy-dev \
libsndfile1-dev \
libsodium-dev \
libsox-dev \
libsqlite3-dev \
libssl-dev \
@ -119,6 +121,7 @@ apt-get install -y \
libtre-dev \
libudev-dev \
libusb-1.0-0-dev \
libvte-2.91-dev \
libwebkitgtk-3.0-dev \
libxau-dev \
libxml2-dev \
@ -128,30 +131,33 @@ apt-get install -y \
libzip-dev \
libzstd-dev \
libzmq3-dev \
llvm-3.9 \
llvm-7 \
llvm-8 \
locales \
m4 \
minisat \
mono-mcs \
nettle-dev \
ninja-build \
nodejs \
npm \
openjdk-8-jdk \
python-mpltoolkits.basemap \
python3-matplotlib \
python3-numpy \
python3-pip \
python3-scipy \
r-base \
r-base-dev \
ruby-dev \
software-properties-common \
sudo \
texlive \
unixodbc-dev \
wget \
xclip \
z3 \
zip \
zlib1g-dev
zlib1g-dev \
zsh
# odbc
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
@ -159,25 +165,14 @@ curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17 -y
# llvm for llvm-hs
curl https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
add-apt-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main"
apt-get update
apt-get install llvm-9-dev -y
locale-gen en_US.UTF-8
curl -sSL https://get.haskellstack.org/ | sh
# Put documentation where we expect it
mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
# llvm-5.0 for GHC (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-5.0 main" \
&& apt-get update \
&& apt-get install -y llvm-5.0
# llvm-6.0 for llvm-hs (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" \
&& apt-get update \
&& apt-get install -y llvm-6.0
# Buggy versions of ld.bfd fail to link some Haskell packages:
# https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
# faster anyways and uses less RAM.
@ -188,14 +183,13 @@ update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
# This version is tracked here:
# https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
#
# GHC 8.4 requires LLVM 5.0 tools (specifically, llc-5.0 and opt-5.0).
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-5.0" 50
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-5.0" 50
# GHC 8.8 requires LLVM 7 tools (?) (specifically, llc-7 and opt-7).
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-7" 50
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-7" 50
# Made sure a "node" binary is in the path, as well as "nodejs".
# A historical naming collision on Debian means that the binary is called "nodejs",
# but some tools like tsc still expect "node" to exist.
ln -s /usr/bin/nodejs /usr/bin/node
# nodejs 10 (nodejs8 in bionic needs conflicting libssl10-dev)
curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs
# install ocilib dependencies then build and install ocilib
cd /tmp \
@ -225,47 +219,33 @@ echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf
&& ldconfig
# Install erlang/otp platform and its dependencies
ERLANG_VERSION="20.2.2"
ERLANG_DEB_FILE="esl-erlang_${ERLANG_VERSION}-1~debian~jessie_amd64.deb"
ERLANG_DEB_FILE="esl-erlang_21.1-1~ubuntu~bionic_amd64.deb"
pushd /tmp \
&& wget https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/${ERLANG_DEB_FILE} \
&& wget https://packages.erlang-solutions.com/erlang/debian/pool/${ERLANG_DEB_FILE} \
&& (dpkg -i ${ERLANG_DEB_FILE}; apt-get install -yf) \
&& rm ${ERLANG_DEB_FILE} \
&& popd
# Install version 3 of the protobuf compiler. (The `protobuf-compiler` package only
# supports version 2.)
curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip \
&& sudo unzip -o protoc-3.3.0-linux-x86_64.zip -d /usr bin/protoc \
&& rm -f protoc-3.3.0-linux-x84_64.zip
# Install the TensorFlow C API.
curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.1.0.tar.gz > libtensorflow.tar.gz \
&& sudo tar zxf libtensorflow.tar.gz -C /usr \
&& rm libtensorflow.tar.gz \
&& ldconfig
# Install libsodium
curl https://download.libsodium.org/libsodium/releases/LATEST.tar.gz > libsodium.tar.gz \
&& sudo tar xfz libsodium.tar.gz -C /tmp \
&& rm libsodium.tar.gz \
&& cd /tmp/libsodium-stable \
&& ./configure \
&& make install
# NOTE: also update Dockerfile when cuda version changes
# Install CUDA toolkit
# The current version can be found at: https://developer.nvidia.com/cuda-downloads
CUDA_PKG=8.0.61-1 # update this on new version
CUDA_VER=${CUDA_PKG:0:3}
CUDA_APT=${CUDA_VER/./-}
CUDA_PKG=10.0.130-1
CUDA_VER=10.0
CUDA_APT=10-0
pushd /tmp \
&& wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& dpkg -i cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_${CUDA_PKG}_amd64.deb \
&& apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub \
&& dpkg -i cuda-repo-ubuntu1804_${CUDA_PKG}_amd64.deb \
&& apt-get update -qq \
&& apt-get install -y cuda-drivers cuda-core-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} cuda-cufft-dev-${CUDA_APT} cuda-cublas-dev-${CUDA_APT} cuda-cusparse-dev-${CUDA_APT} cuda-cusolver-dev-${CUDA_APT} \
&& rm cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& rm cuda-repo-ubuntu1804_${CUDA_PKG}_amd64.deb \
&& export CUDA_PATH=/usr/local/cuda-${CUDA_VER} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH+x} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${LD_LIBRARY_PATH} \
@ -276,3 +256,34 @@ pushd /tmp \
apt-add-repository multiverse \
&& apt-get update \
&& apt-get install -y nvidia-cuda-dev
export CLANG_PURE_LLVM_LIB_DIR=/usr/lib/llvm-7/lib;
export CLANG_PURE_LLVM_INCLUDE_DIR=/usr/lib/llvm-7/include;
# protoc, for proto-lens-combinators test suite
# Instructions from: https://google.github.io/proto-lens/installing-protoc.html
PROTOC_ZIP=protoc-3.3.0-linux-x86_64.zip
curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr bin/protoc
rm -f $PROTOC_ZIP
# Update library search paths
echo /usr/local/cuda-10.0/lib64 > /etc/ld.so.conf.d/cuda.conf
echo /usr/local/cuda-10.0/nvvm/lib64 >> /etc/ld.so.conf.d/cuda.conf
echo /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server > /etc/ld.so.conf.d/java.conf
echo /usr/lib/llvm-3.7/lib > /etc/ld.so.conf.d/llvm.conf
ldconfig
# Install librdkafka (Apache Kafka C/C++ library)
wget -qO - https://packages.confluent.io/deb/5.2/archive.key | apt-key add -
add-apt-repository "deb https://packages.confluent.io/deb/5.2 stable main"
apt-get update && apt install -y librdkafka-dev
# EOF: don't build anything below this line
# Cleanup
apt-get clean
rm -rf /var/lib/apt/lists/*

25
etc/ci-script.sh Executable file
View File

@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euxo pipefail
export GHCVER=8.8.1
# Download and unpack the stack executable
mkdir -p ~/.local/bin
export PATH=$HOME/.local/bin:$PATH
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'
# Get stackage-curator
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
bunzip2 stackage-curator.bz2
chmod +x stackage-curator
mv stackage-curator ~/.local/bin
# Install GHC
stack setup $GHCVER
# Update the index
stack update
# Check
exec stack --resolver ghc-$GHCVER exec stackage-curator check

31
etc/curator-2-check.sh Executable file
View File

@ -0,0 +1,31 @@
#!/usr/bin/env bash
set -euxo pipefail
ETC=$(cd $(dirname $0) ; pwd)
export GHCVER=$(sed -n "s/^ghc-version: \"\(.*\)\"/\1/p" "$ETC/../build-constraints.yaml")
# Download and unpack the stack executable
mkdir -p ~/.local/bin
export PATH=$HOME/.local/bin:$PATH
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'
# Get new Stackage curator
CURATOR2=stackage-curator-2-f6258124cff9a7e92bcb5704164a70e149080e88
wget "https://download.fpcomplete.com/stackage-curator-2/$CURATOR2.bz2"
bunzip2 "$CURATOR2.bz2"
chmod +x $CURATOR2
mv $CURATOR2 ~/.local/bin/stackage-curator-2
# Install GHC
stack setup $GHCVER
# curator's constraint command has target as a required parameter
# because of a different constraints handling in minor LTS version bumps
NIGHTLY="nightly-$(date +%Y-%m-%d)"
# New curator check
stackage-curator-2 update &&
stackage-curator-2 constraints --target=$NIGHTLY &&
stackage-curator-2 snapshot-incomplete &&
stackage-curator-2 snapshot &&
stack --resolver ghc-$GHCVER exec stackage-curator-2 check-snapshot

1
etc/curator-exes/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
/bin/

View File

@ -0,0 +1,13 @@
FROM fpco/stack-build:lts-14
# Prime the snapshot so rebuilds are faster
RUN mkdir -p /artifacts
RUN stack update
RUN stack build --no-install-ghc --resolver lts-13.28 --only-dependencies stack amazonka-s3
RUN git clone --depth 1 https://github.com/commercialhaskell/stack && \
git clone --depth 1 https://github.com/commercialhaskell/curator
COPY stack.yaml .
RUN stack --no-install-ghc install --local-bin-path /artifacts

View File

@ -0,0 +1,5 @@
Scripts to create and upload the curator-exes.tar.bz2 files. Make sure to
update the hash in automated/build-next.sh.
NOTE: Depending on what's exactly on the master branches when running the
`./run.sh` script, the `stack.yaml` file may need some tweaks.

11
etc/curator-exes/run.sh Executable file
View File

@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -eux
rm -rf bin
mkdir -p bin
docker build . --tag curator-exes
docker run --rm -v $(pwd)/bin:/output curator-exes cp /artifacts/* output
aws s3 cp curators-exes/* s3://download.fpcomplete.com/curator-exes/

View File

@ -0,0 +1,22 @@
resolver: lts-13.28
packages:
- stack
- curator
extra-deps:
- hi-file-parser-0.1.0.0@sha256:3fc95852035c8e0fa6d5f9d22e5108067c8f4816ac84675bddc971b20144a9ae,2317
- http-download-0.1.0.0@sha256:b02d76d28364b74fb90d3ec3c46fd81f9d77bcf71fa40c1d341a7251a0fc40e8,2023
- pantry-0.1.1.2@sha256:6f06556f8e934afd517559d6f3dae947db91ef3801ee4e5cee9916c83a0c4d52,4628
- rio-prettyprint-0.1.0.0@sha256:764e1a357275c3c62d57c494d038feae01cf745788e7d0e97367ba793c9670c3,1359
- ansi-terminal-0.9.1@sha256:48f53532d0f365ffa568c8cf0adc84c66f800a7d80d3329e4f04fa75392f4af1,3225
- rio-0.1.12.0@sha256:a0b42682a455ffde6a4d894b135117e379aea7ed1810c2cfed471b8f5879968c,3633
- unliftio-0.2.12@sha256:b089fbc2ff2628a963c2c4b12143f2020874e3e5144ffd6c62b25639a0ca1483,3325
- github: fpco/casa
commit: a0cd05c1a9ce33acbb1bdfb9324370f884916e88
# sha256: 812e05de8f9abdbd1101fc94895dcfd5c35962c4d67a43cc20cbc06eb59ea1f0
# size: 9508
subdirs:
- casa-client
- casa-types

View File

@ -1,9 +0,0 @@
## 0.7.3.0
* Added the executables split off from stackage-cli.
## 0.7.0.0
* First release of this incarnation of the stackage package. Previously, this
package provided completely different functionality. That functionality has
since moved to stackage-curator.

View File

@ -1,20 +0,0 @@
Copyright (c) 2015 FP Complete
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -1,2 +0,0 @@
This is a dummy wrapper package, forcing installation of other packages which
provide real functionality.

View File

@ -1,2 +0,0 @@
import Distribution.Simple
main = defaultMain

View File

@ -1,3 +0,0 @@
-- | This module does absolutely nothing. It's present so that cabal can more
-- easily track whether the stackage package is installed.
module Stackage.Dummy () where

View File

@ -1,29 +0,0 @@
name: stackage
version: 0.7.3.2
synopsis: Dummy package forcing installation of other Stackage packages
homepage: https://www.stackage.org/
license: MIT
license-file: LICENSE
author: Michael Snoyman
maintainer: michael@snoyman.com
category: Development
build-type: Simple
extra-source-files: README.md ChangeLog.md
cabal-version: >=1.10
library
exposed-modules: Stackage.Dummy
build-depends: base < 10
, stackage-cli >= 0.1.0
, stackage-update
, stackage-upload
, stackage-install
, stackage-build-plan
, stackage-cabal >= 0.1.1
, stackage-sandbox >= 0.1.1
, stackage-setup >= 0.0.1
default-language: Haskell2010
source-repository head
type: git
location: git://github.com/commercialhaskell/stackage.git