Merge branch 'master' into nightly

This commit is contained in:
Jens Petersen 2021-06-11 01:48:20 +08:00
commit e16554cce0
13 changed files with 688 additions and 368 deletions

View File

@ -1,13 +1,13 @@
Checklist:
- [ ] Meaningful commit message, eg `add my-cool-package` (please not mention `build-constraints.yml`)
- [ ] At least 30 minutes have passed since uploading to Hackage
- [ ] On your own machine, you have successfully run the following command (replace `$package` with the name of the package that is submitted, and `$version` with the version of the package you want to get into Stackage):
- [ ] On your own machine, you have successfully run the following command (find verify-package in the root of this repo):
./verify-package $package # or $package-$version
The script runs virtually the following commands in a clean directory:
stack unpack $package-$version # $version is optional
stack unpack $package-$version # `-$version` is optional
cd $package-$version
rm -f stack.yaml && stack init --resolver nightly
rm -f stack.yaml && stack init --resolver nightly --ignore-subdirs
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks

21
.github/workflows/check.yml vendored Normal file
View File

@ -0,0 +1,21 @@
name: check
on: [push,pull_request]
jobs:
build:
runs-on: ubuntu-latest
name: Stackage check
steps:
- uses: actions/checkout@v2
- uses: haskell/actions/setup@v1
with:
ghc-version: '8.10'
stack-version: '2.5'
- uses: actions/cache@v2
with:
path: |
~/.stack
~/.stackage/curator/cache
key: ${{ runner.os }}-stackage-nightly-${{ hashFiles('**/build-constraints.yaml') }}
restore-keys: |
${{ runner.os }}-
- run: etc/check.sh

View File

@ -1,21 +0,0 @@
# Disable Docker stuff to try to make it faster
sudo: true
dist: bionic
# Choose a lightweight base image; we provide our own build tools.
language: generic
# GHC depends on GMP. You can add other dependencies here as well.
addons:
apt:
packages:
- libgmp-dev
# Edit etc/ci-script.sh to change GHC version
script:
- ./etc/check.sh
cache:
directories:
- $HOME/.stack
- $HOME/.stackage/curator/cache

View File

@ -220,7 +220,7 @@ file](https://github.com/fpco/stackage-content/blob/master/stack/global-hints.ya
is updated with information on the latest GHC release by cloning that
repo and running `./update-global-hints.yaml ghc-X.Y.Z`.
Also required to build an LTS minor bump with a ghc version change: On the build server, modify `/var/stackage/stackage/automated/wrk/lts-$THIS_LTS_MAJOR_VER/constraints.yaml` and update the ghc-version. Then run `NOPLAN=1 /var/stackage/stackage/automated/build lts-$THIS_LTS_MINOR_BUMP` to build the LTS.
Also required to build an LTS minor bump with a ghc version change: On the build server, modify `/var/stackage/stackage/automated/work/lts-$THIS_LTS_MAJOR_VER/constraints.yaml` and update the ghc-version. (You may need to update sibling files as well.) Then run `NOPLAN=1 /var/stackage/stackage/automated/build.sh lts-$THIS_LTS_MINOR_BUMP` to build the LTS.
### Getting the new image to the build server
Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
@ -307,14 +307,12 @@ First run `build.sh` to regenerate updated `ltsXX/work/constraints.yaml` and `lt
For an LTS minor bump, you'll typically want to:
* Add constraints to package `range:` fields _under_ the `source:` field in that `constraints.yaml`.
* Add constraints to package `range:` fields _under_ the `source:` field in that `constraints.yaml`, and edit `snapshot-incomplete.yaml` to change the version used for that package, if necessary.
* Add new packages to the `constraints.yaml` file
* Test, benchmark, haddock failures can also be added to package fields in the `constraints.yaml` if necessary, though it should be avoided if possible for LTS.
Then run `NOPLAN=1 build.sh` to build the generate an updated snapshot.
This replaces `CONSTRAINTS=...' /var/stackage/stackage/automated/build.sh lts-x.y` for the old curator-1.
If a build fails for bounds reasons, see all of the advice above. If the code
itself doesn't build, or tests fail, open up an issue and then either put in a
version bound to avoid that version or something else. It's difficult to give
@ -329,7 +327,7 @@ if one needs to revert one package, say due to a build or test regression,
one can edit `current-plan.yaml` and updated the SHA256 hash of the .cabal file,
to avoid having to rebuild everything again.)
_Sadly no longer true currently_: ~~Note LTS builds inherit the current Hackage data (stack updated for Nightly) to avoid excess extra rebuilding.~~
Note LTS builds without NOPLAN will use the latest Hackage data.
### Timing

View File

@ -2,7 +2,7 @@ FROM fpco/pid1:18.04
ENV HOME /home/stackage
ENV LANG en_US.UTF-8
ENV GHCVER 9.0.1
ENV GHCVER 8.10.5
# NOTE: also update debian-bootstrap.sh when cuda version changes
ENV PATH /home/stackage/.stack/programs/x86_64-linux/ghc-$GHCVER/bin:/usr/local/cuda-10.0/bin:/usr/sbin:/usr/bin:/sbin:/bin

View File

@ -26,23 +26,15 @@ don't maintain this package yourself it is preferable that the actual
maintainer is also the stackage maintainer, but you are allowed to add
it under your own name.
If your library depends on a C library, please add it to the `debian-bootstrap.sh` script.
If your package depends on a C library, please add it to the `debian-bootstrap.sh` script.
After doing that commit with a message like "add foo-bar" and send a pull request.
After doing that, commit with a message like "add foo-bar" and send a pull request.
The continuous integration job will do some checks to see if your package's dependencies are up-to-date.
The CI job notably doesn't compile packages, run tests, build documentation, or find missing C libraries.
If you want to be proactive or if CI fails, you can make sure that your package builds against the latest nightly:
```
# Build from the tarball on Hackage to check for missing files
$ stack unpack yourpackage && cd yourpackage-*
# Generate a pristine stack.yaml, adding any missing extra-deps
$ rm -f stack.yaml && stack init --resolver nightly
# Build, generate docs, test, and build benchmarks
$ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
```
If you want to be proactive or if CI fails, you can make sure that your package builds against the latest nightly.
See the [verify-package](https://github.com/commercialhaskell/stackage/blob/master/verify-package) script in this repository.
This approach works well, but has two limitations you should be aware
of:
@ -50,26 +42,11 @@ of:
* It won't notify you of restrictive upper bounds in your package if
Stackage has the same upper bounds. For that reason, we recommend
using [Packdeps](http://packdeps.haskellers.com/) (see "Following
dependency upgrades" below).
dependency upgrades" below). You can also run `cabal outdated`.
* If the latest Stackage Nightly is missing some of the latest
packages, your build above may succeed whereas the Travis job may
fail. Again: Packdeps will help you detect this situation.
Alternatively, you can build with `cabal`. Note that this may end up
using older dependency versions:
```
$ ghc --version # Should be the same as the latest nightly, it's in the title of https://www.stackage.org/nightly
$ cabal update
$ cabal get PACKAGE
$ cd PACKAGE-*
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
$ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
$ cabal install --enable-tests --enable-benchmarks --allow-newer
$ cabal test
$ cabal haddock
```
## Github and Notifications
Stackage uses Github notifications to reduce overhead of contacting individual
@ -173,7 +150,7 @@ ending in `.0`), the package set is taken from Stackage Nightly. Therefore, by
following the above steps, you can get your package into the next major LTS
Haskell release.
If you would like to get your package added to the current LTS Haskell
If you would like to get your package added to the current LTS Haskell
major release, please do the following in addition to the steps for Nightly described earlier:
* Check that your package can be built with the current LTS version (e.g. `stack build --test --bench --haddock --resolver lts`)

View File

@ -1,16 +1,16 @@
stackage
========
[![Build Status](https://travis-ci.org/commercialhaskell/stackage.svg?branch=master)](https://travis-ci.org/commercialhaskell/stackage)
[![check](https://github.com/commercialhaskell/stackage/actions/workflows/check.yml/badge.svg)](https://github.com/commercialhaskell/stackage/actions/workflows/check.yml)
[![Join the chat at https://gitter.im/commercialhaskell/stackage](https://badges.gitter.im/commercialhaskell/stackage.svg)](https://gitter.im/commercialhaskell/stackage?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
"Stable Hackage": creating a vetted set of packages from Hackage.
This repository is for package authors and maintainers to get their packages into Stackage.
Stable sets of Haskell Packages from Hackage
_This repository is for package authors and maintainers to get their packages into Stackage._
If you simply want to use Stackage as an end user, please follow the instructions on [https://www.stackage.org/](https://www.stackage.org).
We strongly recommend using the Haskell tool stack for doing builds, which
includes built-in Stackage support: [stack](https://github.com/commercialhaskell/stack) [![Build Status](https://dev.azure.com/commercialhaskell/stack/_apis/build/status/commercialhaskell.stack?branchName=master)](https://dev.azure.com/commercialhaskell/stack/_build/latest?definitionId=1?branchName=master).
We strongly recommend using the Haskell [stack](https://github.com/commercialhaskell/stack) tool for doing builds, which
includes built-in Stackage support.
Add your package
----------------
@ -25,7 +25,7 @@ We welcome all packages, provided:
Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#adding-a-package).
__NOTE__: There is an approximate 30 minute delay between a package uploading
to Hackage and being available to the Travis build script to check upper
to Hackage and being available to the Github workflow action to check upper
bounds. If a pull request is marked as failed due to using an older version,
please close and reopen the PR to retrigger a Travis build.
@ -36,24 +36,14 @@ The Stackage project consists of multiple repositories. This repository
contains the metadata on packages to be included in future builds and some
project information. In addition, we have the following repositories:
* [stackage-server](https://github.com/fpco/stackage-server) [![Build Status](https://travis-ci.org/fpco/stackage-server.svg?branch=master)](https://travis-ci.org/fpco/stackage-server)
* [stackage-curator](https://github.com/fpco/stackage-curator) [![Build Status](https://travis-ci.org/fpco/stackage-curator.svg?branch=master)](https://travis-ci.org/fpco/stackage-curator)
* [lts-haskell](https://github.com/fpco/lts-haskell)
* [stackage-nightly](https://github.com/fpco/stackage-nightly)
We also support some add-on tools to cabal-install to make its usage with
Stackage both easier and more secure:
* [stackage-cli](https://github.com/fpco/stackage-cli) [![Build Status](https://travis-ci.org/fpco/stackage-cli.svg?branch=master)](https://travis-ci.org/fpco/stackage-cli)
* [stackage-update](https://github.com/fpco/stackage-update) [![Build Status](https://travis-ci.org/fpco/stackage-update.svg?branch=master)](https://travis-ci.org/fpco/stackage-update)
* [stackage-upload](https://github.com/fpco/stackage-upload) [![Build Status](https://travis-ci.org/fpco/stackage-upload.svg?branch=master)](https://travis-ci.org/fpco/stackage-upload)
* [stackage-install](https://github.com/fpco/stackage-install) [![Build Status](https://travis-ci.org/fpco/stackage-install.svg?branch=master)](https://travis-ci.org/fpco/stackage-install)
* [stackage-build-plan](https://github.com/fpco/stackage-build-plan) [![Build Status](https://travis-ci.org/fpco/stackage-build-plan.svg?branch=master)](https://travis-ci.org/fpco/stackage-build-plan)
* [stackage-server](https://github.com/fpco/stackage-server) [![Base image](https://github.com/fpco/stackage-server/actions/workflows/base.yml/badge.svg)](https://github.com/fpco/stackage-server/actions/workflows/base.yml)
* [curator](https://github.com/commercialhaskell/curator) [![Build Status](https://travis-ci.org/fpco/stackage-curator.svg?branch=master)](https://travis-ci.org/fpco/stackage-curator)
* [lts-haskell](https://github.com/commercialhaskell/lts-haskell)
* [stackage-snapshots](https://github.com/commercialhaskell/stackage-snapshots/)
Curious how it all fits together? See the [Stackage data
flow](https://github.com/commercialhaskell/stackage/blob/master/DATA-FLOW.md).
Build the package set
---------------------

View File

@ -63,7 +63,7 @@ BINDIR=$(cd $ROOT/work/bin ; pwd)
cd $BINDIR
rm -f curator stack *.bz2
curl -L "https://download.fpcomplete.com/stackage-curator-2/curator-7c719d6d48839c94a79dc2ad2ace89074e3dd997.bz2" | bunzip2 > curator
curl -L "https://github.com/commercialhaskell/curator/releases/download/commit-1ef870178f6a0544fe5a3fd23a83fc075aabd695/curator.bz2" | bunzip2 > curator
chmod +x curator
echo -n "curator version: "
docker run --rm -v $(pwd)/curator:/exe $IMAGE /exe --version
@ -108,7 +108,7 @@ then
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator update && curator constraints --target $TARGET && curator snapshot-incomplete --target $TARGET && curator snapshot"
fi
else
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator snapshot"
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator snapshot-incomplete --target $TARGET && curator snapshot"
fi
@ -141,8 +141,8 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator check-target-available
# * Upload the new snapshot .yaml file to the appropriate Github repo, also upload its constraints
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "curator upload-docs --target $TARGET && curator upload-github --target $TARGET"
# For some reason, registering on Hackage fails with inscrutable error messages. Disabling.
# docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator hackage-distro --target $TARGET"
# fixed in https://github.com/commercialhaskell/curator/pull/24
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator hackage-distro --target $TARGET"
# Build and push docker image fpco/stack-build & fpco/stack-build-small for current release

View File

@ -0,0 +1,90 @@
FROM ubuntu:18.04
LABEL maintainer="manny@fpcomplete.com"
ARG GHC_VERSION=8.8.3
ARG LTS_SLUG=lts-17.0
ARG PID1_VERSION=0.1.2.0
ARG STACK_VERSION=2.5.1
ARG CUDA_VERSION=10.0
ARG JVM_PATH=/usr/lib/jvm/java-8-openjdk-amd64
ARG LLVM_PATH=/usr/lib/llvm-7
ARG BOOTSTRAP_COMMIT=1afd08ff0a3c3627f17eb1b928d35f337db40e08
ARG DEBIAN_FRONTEND=noninteractive
ARG VARIANT=build
ARG STACK_ROOT=/home/stackage/.stack
#
# Set encoding to UTF-8 and PATH to find GHC and cabal/stack-installed binaries.
#
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PATH=/root/.local/bin:/usr/local/cuda-$CUDA_VERSION/bin:$STACK_ROOT/programs/x86_64-linux/ghc-$GHC_VERSION/bin:$PATH \
CUDA_PATH=/usr/local/cuda-$CUDA_VERSION \
CPATH=$JVM_PATH/include:$JVM_PATH/include/linux:$LLVM_PATH/include
#
# Install pre-requisites
#
RUN apt-get update && \
apt-get install -y --no-install-recommends \
wget netbase ca-certificates g++ gcc libc6-dev libffi-dev libgmp-dev \
make xz-utils zlib1g-dev git gnupg libtinfo-dev jq && \
rm -rf /var/lib/apt/lists/*
#
# Use Stackage's debian-bootstrap.sh script to install system libraries and
# tools required to build any Stackage package.
# Re-installs 'stack' *after* running debian-bootstrap.sh since that may have
# installed a different version.
# In the case of 'small' image, just install Stack and GHC.
#
RUN if [ "$VARIANT" != "small" ]; then \
wget -qO- https://raw.githubusercontent.com/fpco/stackage/$BOOTSTRAP_COMMIT/debian-bootstrap.sh | sed "s/^GHCVER=8.6.5$/GHCVER=$GHC_VERSION/" | GHCVER=$GHC_VERSION bash; \
fi && \
wget -qO- https://github.com/commercialhaskell/stack/releases/download/v$STACK_VERSION/stack-$STACK_VERSION-linux-x86_64.tar.gz | tar xz --wildcards --strip-components=1 -C /usr/bin '*/stack' && \
if [ "$VARIANT" = "small" ]; then \
stack setup --resolver ghc-$GHC_VERSION; \
fi && \
rm -rf /var/lib/apt/lists/* && \
cd $STACK_ROOT && \
find . -type f -not -path "./programs/x86_64-linux/ghc-$GHC_VERSION/*" -exec rm '{}' \; && \
find . -type d -print0 |sort -rz |xargs -0 rmdir 2>/dev/null || true
#
# Configure Stack to use the GHC installed in the Docker image rather than installing its own
#
RUN mkdir /etc/stack/ && \
echo "system-ghc: true" >/etc/stack/config.yaml
#
# Use 'stack' to install basic Haskell tools like alex, happy, and cpphs. We
# remove most of the STACK_ROOT afterward to save space, but keep the 'share'
# files that some of these tools require.
#
RUN stack --resolver=$LTS_SLUG --local-bin-path=/usr/bin install \
happy alex cpphs gtk2hs-buildtools hscolour hlint hindent && \
cd $STACK_ROOT && \
find . -type f -not -path './snapshots/*/share/*' -and -not -path "./programs/x86_64-linux/ghc-$GHC_VERSION/*" -exec rm '{}' \; && \
find . -type d -print0 |sort -rz |xargs -0 rmdir 2>/dev/null || true
#
# Install 'pid1' init daemon
#
RUN wget -O- "https://github.com/fpco/pid1/releases/download/v$PID1_VERSION/pid1-$PID1_VERSION-linux-x86_64.tar.gz" | tar xzf - -C /usr/local && \
chown root:root /usr/local/sbin && \
chown root:root /usr/local/sbin/pid1
#
# Set up pid1 entrypoint and default command
#
ENTRYPOINT ["/usr/local/sbin/pid1"]
CMD ["bash"]

View File

@ -0,0 +1,3 @@
FROM $DOCKER_REPO:lts-17.9
ARG STACK_VERSION=2.7.1
RUN wget -qO- https://github.com/commercialhaskell/stack/releases/download/v$STACK_VERSION/stack-$STACK_VERSION-linux-x86_64.tar.gz | tar xz --wildcards --strip-components=1 -C /usr/local/bin '*/stack'

View File

@ -0,0 +1,90 @@
FROM ubuntu:18.04
LABEL maintainer="manny@fpcomplete.com"
ARG GHC_VERSION=8.8.4
ARG LTS_SLUG=lts-17.3
ARG PID1_VERSION=0.1.2.0
ARG STACK_VERSION=2.5.1
ARG CUDA_VERSION=10.0
ARG JVM_PATH=/usr/lib/jvm/java-8-openjdk-amd64
ARG LLVM_PATH=/usr/lib/llvm-7
ARG BOOTSTRAP_COMMIT=1afd08ff0a3c3627f17eb1b928d35f337db40e08
ARG DEBIAN_FRONTEND=noninteractive
ARG VARIANT=build
ARG STACK_ROOT=/home/stackage/.stack
#
# Set encoding to UTF-8 and PATH to find GHC and cabal/stack-installed binaries.
#
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PATH=/root/.local/bin:/usr/local/cuda-$CUDA_VERSION/bin:$STACK_ROOT/programs/x86_64-linux/ghc-$GHC_VERSION/bin:$PATH \
CUDA_PATH=/usr/local/cuda-$CUDA_VERSION \
CPATH=$JVM_PATH/include:$JVM_PATH/include/linux:$LLVM_PATH/include
#
# Install pre-requisites
#
RUN apt-get update && \
apt-get install -y --no-install-recommends \
wget netbase ca-certificates g++ gcc libc6-dev libffi-dev libgmp-dev \
make xz-utils zlib1g-dev git gnupg libtinfo-dev jq && \
rm -rf /var/lib/apt/lists/*
#
# Use Stackage's debian-bootstrap.sh script to install system libraries and
# tools required to build any Stackage package.
# Re-installs 'stack' *after* running debian-bootstrap.sh since that may have
# installed a different version.
# In the case of 'small' image, just install Stack and GHC.
#
RUN if [ "$VARIANT" != "small" ]; then \
wget -qO- https://raw.githubusercontent.com/fpco/stackage/$BOOTSTRAP_COMMIT/debian-bootstrap.sh | sed "s/^GHCVER=8.6.5$/GHCVER=$GHC_VERSION/" | GHCVER=$GHC_VERSION bash; \
fi && \
wget -qO- https://github.com/commercialhaskell/stack/releases/download/v$STACK_VERSION/stack-$STACK_VERSION-linux-x86_64.tar.gz | tar xz --wildcards --strip-components=1 -C /usr/bin '*/stack' && \
if [ "$VARIANT" = "small" ]; then \
stack setup --resolver ghc-$GHC_VERSION; \
fi && \
rm -rf /var/lib/apt/lists/* && \
cd $STACK_ROOT && \
find . -type f -not -path "./programs/x86_64-linux/ghc-$GHC_VERSION/*" -exec rm '{}' \; && \
find . -type d -print0 |sort -rz |xargs -0 rmdir 2>/dev/null || true
#
# Configure Stack to use the GHC installed in the Docker image rather than installing its own
#
RUN mkdir /etc/stack/ && \
echo "system-ghc: true" >/etc/stack/config.yaml
#
# Use 'stack' to install basic Haskell tools like alex, happy, and cpphs. We
# remove most of the STACK_ROOT afterward to save space, but keep the 'share'
# files that some of these tools require.
#
RUN stack --resolver=$LTS_SLUG --local-bin-path=/usr/bin install \
happy alex cpphs gtk2hs-buildtools hscolour hlint hindent && \
cd $STACK_ROOT && \
find . -type f -not -path './snapshots/*/share/*' -and -not -path "./programs/x86_64-linux/ghc-$GHC_VERSION/*" -exec rm '{}' \; && \
find . -type d -print0 |sort -rz |xargs -0 rmdir 2>/dev/null || true
#
# Install 'pid1' init daemon
#
RUN wget -O- "https://github.com/fpco/pid1/releases/download/v$PID1_VERSION/pid1-$PID1_VERSION-linux-x86_64.tar.gz" | tar xzf - -C /usr/local && \
chown root:root /usr/local/sbin && \
chown root:root /usr/local/sbin/pid1
#
# Set up pid1 entrypoint and default command
#
ENTRYPOINT ["/usr/local/sbin/pid1"]
CMD ["bash"]

File diff suppressed because it is too large Load Diff

View File

@ -30,11 +30,12 @@ exit() {
}
trap exit EXIT
# When updating these commands, make sure to update .github/PULL_REQUEST_TEMPLATE.md as well.
cd "$dir"
stack unpack "$package"
cd "$(ls | head -n 1)"
rm -f stack.yaml
stack init --resolver nightly
stack init --resolver nightly --ignore-subdirs
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks