haskell/stackage
1
0
Fork 0
mirror of https://github.com/commercialhaskell/stackage.git synced 2026-01-13 07:48:31 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #1 from commercialhaskell/master

Browse Source 
update fork to current master
This commit is contained in:
Eitan Chatav 2018-06-26 12:17:34 -07:00 committed by GitHub
commit 0839e43b1c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 1953 additions and 1387 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View File

@ -0,0 +1,9 @@
Checklist:
- [ ] Meaningful commit message - please not `Update build-constraints.yml`
- [ ] At least 30 minutes have passed since Hackage upload
- [ ] On your own machine, in a new directory, you have successfully run the following set of commands (replace `$package` with the name of the package that is submitted, `$version` is the version of the package you want to get into Stackage):
stack unpack $package-$version
cd $package-$version
stack init --resolver nightly
stack build --resolver nightly --haddock --test --bench --no-run-benchmarks

2
.travis.yml
View File

@ -11,7 +11,7 @@ addons:
- libgmp-dev
env:
- GHCVER=8.2.1
- GHCVER=8.4.3
install:
# Download and unpack the stack executable

72
CURATORS.md
View File

@ -1,20 +1,28 @@
This is a collection of instructions covering the processes that the Stackage curators - the
guys who maintain the Stackage project itself - should be doing on a regular basis.
Originally this was handled largely by Michael Snoyman,
but now we are a team of 5 people handling requests weekly in rotation.
team who maintain the Stackage project itself - should be doing on a regular basis.
Curation activities are mostly automated, and do not take up a significant amount of time.
The following is the current list of curators, in alphabetical order:
* Adam Bergmark (@bergmark)
* Alexey Zabelin (@alexeyzab)
* Chris Dornan (@cdornan)
* Dan Burton (@danburton)
* Jens Petersen (@juhp)
* Joe Kachmar (@jkachmar)
* Michael Snoyman (@snoyberg)
* Mihai Maruseac (@mihaimaruseac)
## Workflow overview
This section sketches out at a high level how the entire Stackage build/curation
process works:
* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
* stackage-curator can check that build plan to ensure all version bounds are consistent
* The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
* Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh)
* The [Travis job](https://github.com/commercialhaskell/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
* Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh)
* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
@ -26,7 +34,7 @@ author only added packages under his/her own name, merge it. If the
build later fails (see [Adding Debian packages]), then block the
package until it's fixed.
[Adding Debian packages]: https://github.com/fpco/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
[Adding Debian packages]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
If benchmarks, haddocks, or test suites fails at this point we
typically also block the package until these issues are fixed. This in
@ -65,13 +73,13 @@ being included immediately. This also applies to when only benchmarks
and tests are affected.
* Copy the stackage-curator output and create a new issue, see e.g
https://github.com/fpco/stackage/issues/2108
https://github.com/commercialhaskell/stackage/issues/2108
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
```yaml
"Stackage upper bounds":
# https://github.com/fpco/stackage/issues/2108
# https://github.com/commercialhaskell/stackage/issues/2108
- pipes < 4.3.0
```
@ -82,7 +90,7 @@ https://github.com/fpco/stackage/issues/2108
Sometimes releases for different packages are tightly coupled. Then it
can make sense to combine them into one issue, as in
https://github.com/fpco/stackage/issues/2143.
https://github.com/commercialhaskell/stackage/issues/2143.
If a dependency that is not explicitly in stackage is causing test or
benchmark failures you can skip or expect them to fail (see "Skipping
@ -115,7 +123,7 @@ enough time to upgrade for this case as well.
With the `pipes` example above there was later a new release of
`pipes-safe` that required the **newer** version of `pipes`. You can
add that package to the same upper bounds section,
(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
(e.g. https://github.com/commercialhaskell/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
as we want to lift them simultaneously.
### Skipping tests and benchmarks
@ -141,7 +149,7 @@ dependencies and compile them.
If there are no version bounds that would fix the issue or if you
can't figure it out, file it
(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
(e.g. https://github.com/commercialhaskell/stackage/issues/2133) to ask the
maintainer for help.
### Waiting for new releases
@ -204,6 +212,9 @@ major version number (e.g., lts3 for lts-3.\*).
Note that when starting a new LTS major release, you'll need to modify Docker
Hub to create a new Docker tag for the relevant branch name.
You'll need to update both the `PATH` in `Dockerfile` and the `GHCVER` variable
in `debian-bootstrap.sh`.
### Getting the new image to the build server
Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
below). Instead of pulling an unbounded number of images, I typically just
@ -217,7 +228,7 @@ docker rmi $(docker images -q)
but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
For a new GHC version you should also delete the cache directories on the stackage-build server to
force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg:
force all packages to be rebuilt. See: [issue#746](https://github.com/commercialhaskell/stackage/issues/746). Eg:
```
rm -r nightly/work/builds/nightly/
```
@ -230,7 +241,7 @@ following command:
```
DIR=$(mktemp -d)
(cd $DIR \
&& git clone https://github.com/fpco/stackage \
&& git clone https://github.com/commercialhaskell/stackage \
&& cd stackage \
&& docker build --tag snoyberg/stackage:nightly .)
rm -rf $DIR
@ -285,6 +296,7 @@ Valid arguments to include in this environment variable:
* `--add-package` to add a brand new package
* `--expect-test-failure` to expect tests to fail
* `--expect-haddock-failure` to expect haddocks to fail
* `--expect-bench-failure` to expect benchmarks to fail
If a build fails for bounds reasons, see all of the advice above. If the code
itself doesn't build, or tests fail, open up an issue and then either put in a
@ -348,7 +360,7 @@ Then, clone the stackage repo, get the latest packages and run dependency
resolution:
```
$ git clone git@github.com:fpco/stackage.git
$ git clone git@github.com:commercialhaskell/stackage.git
$ stack update && stackage-curator check
```
@ -360,12 +372,12 @@ test suites and benchmarks, to check whether bounds can be lifted, and to get
errors for builds, tests and benchmarks.
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
[tell-me-when-its-released]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#waiting-for-new-releases
## Adding new curators
1. Add public ssh key to `~/.ssh/authorized_keys` on build server
2. Add to fpco/stackage project.
2. Add to commercialhaskell/stackage project.
## Dealing with a new GHC release
@ -389,4 +401,26 @@ After the first LTS release, the package pruning process may begin in the
nightly build in order to move forward with getting the latest versions of
packages compatible with the new GHC release.
[GHC upgrade note]: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
[GHC upgrade note]: https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
## New LTS major bump
Every 3-6 months, we make a new major release of LTS. The procedure we follow for this is:
1. Write a blog post on stackage.org announcing the intent to cut a major
release. Give an estimated date two weeks in the future from the publication
date of the post.
2. Spread the blog post on social media and mailing lists as much as possible.
3. Expect maintainers to send significant requests for added packages and
relaxed upper bounds. There will likely be some hard decisions to be made
regarding relaxing a bound versus keeping more packages. All of these changes
occur on master and affect nightly.
4. Once the estimated date hits, push a new `lts-XX` branch and trigger Docker
Hub to build a Docker image for the new release.
5. Run the build procedure for the new LTS release.
6. After the LTS build completes, more aggressively prune upper bounds from
`build-constraints.yaml`.
7. Once both (5) and (6) are done, publish a new blog post on stackage.org
announcing the new LTS and Nightly, with links to the change pages on
stackage.org. Include a reminder that requests for packages to be added to LTS
may be made on commercialhaskell/lts-haskell.

6
DATA-FLOW.md
View File

@ -15,7 +15,7 @@ There are three inputs into the data flow:
Hackage provides both cabal file metadata (via the 00-index.tar file) and
tarballs of the individual packages.
* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml)
is the primary Stackage input file. This is where package maintainers can add
packages to the Stackage package set. This also defines upper bounds, skipped
tests, and a few other pieces of metadata.
@ -60,7 +60,7 @@ The heart of running Stackage builds is the
[stackage-curator](https://github.com/fpco/stackage-curator) tool. We run this
on a daily basis on the Stackage build server for Stackage Nightly, and on a
weekly basis for LTS Haskell. The build process is [highly
automated](https://github.com/fpco/stackage/blob/master/automated/build.sh) and
automated](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh) and
leverages Docker quite a bit.
stackage-curator needs to know about the most recent versions of all packages,
@ -80,7 +80,7 @@ By combining these constraints with the current package data, stackage-curator
can generate a build plan and check it. (As an aside, this build plan
generation and checking also occurs every time you make a pull request to the
stackage repo.) If there are version bounds problems, one of the [Stackage
curators](https://github.com/fpco/stackage/blob/master/CURATORS.md) will open
curators](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md) will open
up a Github issue and will add upper bounds, temporarily block a package, or
some other corrective action.

6
Dockerfile
View File

@ -2,7 +2,11 @@ FROM fpco/pid1:16.04
ENV HOME /home/stackage
ENV LANG en_US.UTF-8
ENV PATH /opt/ghc/8.2.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
# NOTE: also update debian-bootstrap.sh when cuda version changes
ENV PATH /usr/local/cuda-8.0/bin:/opt/ghc/8.4.3/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV CUDA_PATH /usr/local/cuda-8.0
ENV LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/nvvm/lib64
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh

199
MAINTAINERS.md
View File

@ -12,89 +12,151 @@ Packages in Stackage are not patched: all package changes occur upstream in Hack
Anyone can add any package to Stackage but you may only add packages under your own name. It's highly encouraged that the actual package maintainer is also the Stackage maintainer, if that is not the case you should drop the package maintainer a note first.
To add your package, first fork this repository.
In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`.
To add a set of packages, you would add:
To add your package you can edit [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) directly on github or fork the project. There's a section called `packages` where you would add yourself and your packages:
"My Name <myemail@example.com> @mygithubuser":
- package1
- package2
- package3
Note that the `master` branch is used for Stackage Nightly (not the `nightly` branch, which is used for the nightly docker builder imagine).
If your library depends on a C library, please add it to the `debian-bootstrap.sh` script.
After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
After doing that commit with a message like "add foo-bar" and send a pull request.
The continuous integration job will do some checks to see if your package's dependencies are up-to-date.
The CI job notably doesn't compile packages, run tests, build documentation, or find missing C libraries.
If you want to be proactive or if CI fails, you can make sure that your package builds against the latest nightly:
If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
```
# Build from the tarball on Hackage to check for missing files
$ stack unpack yourpackage && cd yourpackage-*
# Generate a pristine stack.yaml, adding any missing extra-deps
$ rm -f stack.yaml && stack init --resolver nightly --solver
# Build, generate docs, test, and build benchmarks
$ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
```
This approach works well, but has two limitations you should be aware
of:
* It won't notify you of restrictive upper bounds in your package if
Stackage has the same upper bounds. For that reason, we recommend
using [Packdeps](http://packdeps.haskellers.com/) (see "Following
dependency upgrades" below).
* If the latest Stackage Nightly is missing some of the latest
packages, your build above may succeed whereas the Travis job may
fail. Again: Packdeps will help you detect this situation.
Alternatively, you can build with `cabal`. Note that this may end up
using older dependency versions:
```
$ ghc --version # Should be the same as the latest nightly, it's in the title of https://www.stackage.org/nightly
$ cabal update
$ ghc --version # Should give v8.0.2
$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
$ cd PACKAGE-VERSION
$ cabal get PACKAGE
$ cd PACKAGE-*
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
$ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
$ cabal install --enable-tests --enable-benchmarks --allow-newer
$ cabal test
$ cabal haddock
```
**NB** Please use commit messages like "add foo-bar" or "add johndev's packages"
(`build-constraints.yaml` is the most frequently changed file in this git repo
so commit messages like "update build-constraints.yaml" are not helpful).
## Github and Notifications
**NB2** There can be a delay of up to an hour before package versions
newly uploaded to Hackage appear to our build server. If you just
uploaded a package to Hackage that you're trying to get included, we
recommend waiting an hour before opening the PR. You can verify this
by making sure the latest version is listed at
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/.
Stackage uses Github notifications to reduce overhead of contacting individual
package maintainers through various channels. As a package maintainer, you will
receive notifications for a number of reasons, package build failures of
different sorts, blockages and bounds issues, etc.
Please note, Github does some throttling on the number people that can be
notified within one issue. This means that on issues created with a large
number of packages affected, maintainers may not receive a notification. This
is not ideal, but Stackage is largely a manual process and done on a best
effort basis.
## Uploading a new package version
When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage. That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
If the new version doesn't compile then the package author should upload a fixed version.
If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite.
## Following dependency upgrades
If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
If a new version of a dependency is released your package may fail to
build or tests may fail. In most cases we will add an upper bound on
the dependency so that your package remains in nightly. We'll also
create a github issue pinging you with the bounds issues or give build
logs showing failures. It's then up to you to modify your package.
If a package is not modified in a timely manner, it may be temporarily
removed from Stackage by the curator team, at which point it is your
responsibility to add it back in via a new pull request. We typically
use fairly long windows on this, but at a minimum:
We recommend that you also follow the dependencies of your packages on
[Packdeps](http://packdeps.haskellers.com/) (typically using the RSS
feeds) as well as that often gives you notice ahead of stackage
issues. There are cases where we will not notice a new release of a
package because of other upper bounds that are in place.
* If restrictive version bounds are the only problem, we will give at
least a week to respond.
If a package is not updated in time, it may be temporarily removed
from Stackage by the curator team. We strive to notify you when this
happens. If it does you are always welcome to file another pull
request to add it back.
We typically use fairly long windows before disabling packages, but it
is decided on a case-by-case basis.
* If restrictive version bounds are the only problem, we will give
maintainers at least a week to respond.
* If there are real breaking changes, the curator team will retain
more discretion on how long a window to give before dropping
packages. Historically, this has usually waited until the cutting of
packages.
* We usually drop all upper bounds and disable packages when we create
a new Long Term Support (LTS) major version.
* There are rare cases where an upper bound or build failure are hard
to deal with so then we may disable
**NOTE** Previously, this maintainer agreement put a time limit on
maintainers, requiring a certain level of responsiveness for
modifications to be made. We have explicitly removed that: anyone is
free to add a package to Stackage regardless of responsiveness
guarantees. However, as stated above, we may elect to temporarily
remove a package if it is not updated in a timely manner.
**NOTE** Previously we had stricter time limits, but we decided to
remove that: Anyone is free to add a package to Stackage regardless of
responsiveness guarantees. However, as stated above, we may elect to
temporarily remove a package if it is not updated in a timely manner.
## Failing to meet the time limits
## Delays
Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack.
Maintainers are humans, humans get sick/have babies/go on
holiday. Sometimes a dependency upgrade is extra time consuming.
The soft time limits are intended to prevent people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for longer periods.
Consider finding a co-maintainer with access to SCM and Hackage to
help you out.
We appreciate if you notify us of any expected delays in the Github
issues, some kind soul might decide to help out.
## Upgrading to a new GHC version
The Stackage curation team tries to move Stackage Nightly to new versions of GHC quickly as they become available, while keeping LTS Haskell on a regular release schedule. For package maintainers, the most important impacts of a new GHC release are:
The Stackage curation team tries to move Stackage Nightly to new
versions of GHC quickly as they become available, while keeping LTS
Haskell on a regular release schedule. For package maintainers, the
most important impacts of a new GHC release are:
* We will typically do a sweep through the Stackage upper bounds and aggressively remove packages that block them. This is because, in most cases, we will need to move to the newest versions of a package to get support for the latest GHC, and asking package maintainers to backport their fixes is an undue burden
* We will definitely do this at a GHC major version release, and may do so at a minor version release
* Packages that are incompatible with the newest GHC version will be temporarily blocked
* We will typically do a sweep through the Stackage upper bounds and
remove blocking packages. We prefer to do it this way rather than
ask other maintainers to backport fixes.
* We will definitely do this at a GHC major version release, and
may do so at a minor version release
* Packages that are incompatible with the newest GHC version will be
temporarily disabled
If your package ends up being temporarily removed from Stackage Nightly, please simply send a pull request to add it back once it and its dependencies are compatible with the newest GHC version.
If your package ends up being temporarily disabled from Stackage
Nightly, please simply send a pull request to add it back once it and
its dependencies are compatible with the newest GHC version.
Note that it is _not_ a goal of LTS Haskell to track the latest
version of GHC. If you want the latest and greatest, Stackage Nightly
is your best bet. In particular, LTS Haskell will often&mdash;but not
always&mdash;avoid upgrading to the first point release of GHC
releases (e.g., 8.2.1) to allow further testing and to get the
benefits of the first bugfix release (e.g., 8.2.2).
## Adding a package to an LTS snapshot
@ -108,8 +170,53 @@ If you would like to get your package added to an existing LTS Haskell major
release (e.g., if `lts-8.9` is out, you would want your package to appear in
`lts-8.10`), please do the following in addition to the steps above:
* Check that your package can be built with that LTS major version (e.g. `stack build --test --bench --haddock --resolver lts-8.10`)
* Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
* Specify the LTS major version you would like your package to go into (e.g., lts-8)
* Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
* Be patient! The LTS releases are less frequent than Nightly and by their nature more conservative, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue as soon as possible, but it may take some time.
* To add a package to more than one LTS version please file separate tickets for each major LTS release, since they will typically be built and added at different times.
* Specify the LTS major versions you would like your packages to go into (e.g. lts-8)
* Provide a list of packages you would like added
* If relevant, mention any upper bounds that are needed on those packages
* Be patient! The LTS releases are less frequent than Nightly. The
Stackage curators will try to get to your issue as soon as possible,
but it may take some time.
* We gradually stop maintainng old LTS major versions, so your
request may take longer or be declined if it's for an old LTS.
## LTS package guarantees and exceptions
In general, we try to stick to some rules when it comes to the
packages included in LTS minor bumps. In particular:
* If a package exists in LTS-X.Y, it should also exist in LTS-X.(Y+1)
* We should not include a major version bump of a package between
LTS-X.Y and LTS-X.(Y+1)
However, there are some cases where exceptions may be made, based
purely on Stackage Curator discretion. The most common examples are:
* If a package does not follow the PVP in its version number policy,
applying the standard version bump rules would not necessarily
makes sense. As an example, suppose package `foo` decides to
follow SemVer instead of the PVP. By our standard rules of version
bumps, a change from `foo-1.2.0` to `foo-1.3.0` would be
considered a major version bump, and disallowed in an LTS minor
version bump. However, if a package is following SemVer, this
would not be a breaking change, and curators may elect to include
it.
* If a package has overly restrictive version bounds on a
dependency, in particular constraining a minor version
unnecessarily, we may drop that package instead of artificially
holding back the dependency. As an example: suppose `LTS-20.1`
includes `foo-1.2.0` and `bar-1`. `bar-1` has a dependency `foo >=
1.2.0 && < 1.2.1`, which is overly constrained on the minor
version number according to the PVP. Then `foo-1.2.1` is
released. The Stackage Curator team would have two choices:
* Reject `foo-1.2.1` from `LTS-20.2`, since that is what `bar-1`
requires.
* Drop `bar-1` from `LTS-20.2`, and allow `foo-1.2.1`.
Decisions will need to be taken on a case-by-case basis, and may
depend on such issues as whether an important bugfix or security
update in included. The curator team may also try to notify the
author of `bar` to try and get a patched version released.

15
README.md
View File

@ -1,7 +1,7 @@
stackage
========
[![Build Status](https://travis-ci.org/fpco/stackage.svg?branch=master)](https://travis-ci.org/fpco/stackage)
[![Build Status](https://travis-ci.org/commercialhaskell/stackage.svg?branch=master)](https://travis-ci.org/commercialhaskell/stackage)
"Stable Hackage": creating a vetted set of packages from Hackage.
This repository is for package authors and maintainers to get their packages into Stackage.
@ -16,12 +16,12 @@ Add your package
We welcome all packages, provided:
* The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
* The package author/maintainer agrees to the [maintainers agreement](https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md).
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
* The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
* The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package).
Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#adding-a-package).
__NOTE__: There is an approximate 30 minute delay between a package uploading
to Hackage and being available to the Travis build script to check upper
@ -37,7 +37,6 @@ project information. In addition, we have the following repositories:
* [stackage-server](https://github.com/fpco/stackage-server) [![Build Status](https://travis-ci.org/fpco/stackage-server.svg?branch=master)](https://travis-ci.org/fpco/stackage-server)
* [stackage-curator](https://github.com/fpco/stackage-curator) [![Build Status](https://travis-ci.org/fpco/stackage-curator.svg?branch=master)](https://travis-ci.org/fpco/stackage-curator)
* [stackage-types](https://github.com/fpco/stackage-types) [![Build Status](https://travis-ci.org/fpco/stackage-types.svg?branch=master)](https://travis-ci.org/fpco/stackage-types)
* [lts-haskell](https://github.com/fpco/lts-haskell)
* [stackage-nightly](https://github.com/fpco/stackage-nightly)
@ -51,17 +50,17 @@ Stackage both easier and more secure:
* [stackage-build-plan](https://github.com/fpco/stackage-build-plan) [![Build Status](https://travis-ci.org/fpco/stackage-build-plan.svg?branch=master)](https://travis-ci.org/fpco/stackage-build-plan)
Curious how it all fits together? See the [Stackage data
flow](https://github.com/fpco/stackage/blob/master/DATA-FLOW.md).
flow](https://github.com/commercialhaskell/stackage/blob/master/DATA-FLOW.md).
Build the package set
---------------------
Generally only the stackage build server run by the stackage curator
team and people intrested in incorporating stackage snapshots into an
team and people interested in incorporating stackage snapshots into an
OS distribution need to build the entire package set. If you're
interested in trying this yourself, please check out
[the curator guide](https://github.com/fpco/stackage/blob/master/CURATORS.md),
[the curator guide](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md),
though be aware that this is not a recommended practice and there
likely will be problems you will need to debug yourself.
@ -103,7 +102,7 @@ There are a number of answers to this question:
another update in the LTS 6 line
* Sometimes we have upper bounds in place because other packages have
problems with newer versions of dependencies. Open up the
[build-constraints file](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
[build-constraints file](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml)
and search for "Stackage upper bounds"
* Wired-in packages - those that ship with GHC and cannot be upgraded,
and packages depending on them - are fixed to GHC versions. Common

15
automated/build.sh
View File

@ -21,7 +21,6 @@ IMAGE=snoyberg/stackage:$TAG
PLAN_FILE=current-plan.yaml
DOCMAP_FILE=current-docmap.yaml
BUNDLE_FILE=current.bundle
CABAL_DIR=$ROOT/cabal
STACK_DIR=$ROOT/stack
@ -30,6 +29,7 @@ DOT_STACKAGE_DIR=$ROOT/dot-stackage
WORKDIR=$ROOT/$TAG/work
EXTRA_BIN_DIR=$ROOT/extra-bin
SSH_DIR=$ROOT/ssh-$SHORTNAME
USERID=$(id -u)
mkdir -p \
"$CABAL_DIR" \
@ -81,9 +81,9 @@ chmod +x stackage-curator
)
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
ARGS_PREBUILD="$ARGS_COMMON -u $USER -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
ARGS_UPLOAD="$ARGS_COMMON -u $USER -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
# Make sure we actually need this snapshot. We only check this for LTS releases
# since, for nightlies, we'd like to run builds even if they are unnecessary to
@ -118,10 +118,15 @@ fi
# * Do a single unpack to create the package index cache (again due to directory perms)
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
case $SHORTNAME in
lts) JOBS=1 ;;
nightly) JOBS=2 ;;
esac
# Now do the actual build. We need to first set the owner of the home directory
# correctly, so we run the command as root, change owner, and then use sudo to
# switch back to the current user
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs 4 --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs $JOBS --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET"
# Make sure we actually need this snapshot. We used to perform this check
# exclusively before building. Now we perform it after as well for the case of
@ -136,7 +141,7 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-
# * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
# * Upload the new plan .yaml file to the appropriate Github repo
# * Register as a new Hackage distro
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
echo -n "Completed at "
date

2
become-a-curator.md
View File

@ -37,7 +37,7 @@ The current curator team consists of:
We onboarded Luke as the newest member a few months ago and this
helped us iron out and document the process further.
You can read the curator documentation here: https://github.com/fpco/stackage/blob/master/CURATORS.md
You can read the curator documentation here: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md
To apply, please fill in this form: [REDACTED]

2933
build-constraints.yaml
View File

File diff suppressed because it is too large Load Diff

2
check
View File

@ -3,4 +3,4 @@
# Convenience script for checking constraints locally
cd `dirname $0`
exec stack exec --resolver ghc-8.2.1 stackage-curator check
exec stack exec --resolver ghc-8.4.3 stackage-curator check

77
debian-bootstrap.sh
View File

@ -4,7 +4,7 @@
# up-and-running on a freshly installed Debian-based system (including Ubuntu).
# Quick start:
# wget -O - https://raw.github.com/fpco/stackage/master/debian-bootstrap.sh | bash -ex
# wget -O - https://raw.github.com/commercialhaskell/stackage/master/debian-bootstrap.sh | bash -ex
# NOTE: Requires that GHC and Cabal are installed and on your PATH. For
# instructions, see:
@ -25,19 +25,25 @@ add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://down
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
GHCVER=8.2.1
GHCVER=8.4.3
apt-get update
apt-get install -y \
apt-transport-https \
build-essential \
cmake \
curl \
freeglut3-dev \
freetds-dev \
fsharp \
g++ \
gawk \
ghc-$GHCVER \
ghc-$GHCVER-dyn \
ghc-$GHCVER-htmldocs \
ghc-$GHCVER-prof \
git \
gnupg \
gradle \
hscolour \
libadns1-dev \
@ -74,6 +80,7 @@ apt-get install -y \
libicu-dev \
libimlib2-dev \
libjack-jackd2-dev \
libjavascriptcoregtk-4.0-dev \
libjudy-dev \
liblapack-dev \
libleveldb-dev \
@ -89,11 +96,13 @@ apt-get install -y \
libmysqlclient-dev \
libncurses-dev \
libnfc-dev \
liboath-dev \
libnotify-dev \
libopenal-dev \
libpango1.0-dev \
libpcap0.8-dev \
libpq-dev \
libre2-dev \
libsdl1.2-dev \
libsdl2-dev \
libsdl2-gfx-dev \
@ -117,6 +126,7 @@ apt-get install -y \
libxss-dev \
libyaml-dev \
libzip-dev \
libzstd-dev \
libzmq3-dev \
llvm-3.9 \
locales \
@ -124,6 +134,7 @@ apt-get install -y \
minisat \
mono-mcs \
nettle-dev \
ninja-build \
nodejs \
npm \
openjdk-8-jdk \
@ -135,12 +146,19 @@ apt-get install -y \
r-base-dev \
ruby-dev \
sudo \
unixodbc-dev \
wget \
xclip \
z3 \
zip \
zlib1g-dev
# odbc
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update
ACCEPT_EULA=Y apt-get install msodbcsql17 -y
locale-gen en_US.UTF-8
curl -sSL https://get.haskellstack.org/ | sh
@ -148,6 +166,18 @@ curl -sSL https://get.haskellstack.org/ | sh
# Put documentation where we expect it
mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
# llvm-5.0 for GHC (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-5.0 main" \
&& apt-get update \
&& apt-get install -y llvm-5.0
# llvm-6.0 for llvm-hs (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" \
&& apt-get update \
&& apt-get install -y llvm-6.0
# Buggy versions of ld.bfd fail to link some Haskell packages:
# https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
# faster anyways and uses less RAM.
@ -158,9 +188,14 @@ update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
# This version is tracked here:
# https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
#
# GHC 8.2 requires LLVM 3.9 tools (specifically, llc-3.9 and opt-3.9).
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.9" 50
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.9" 50
# GHC 8.4 requires LLVM 5.0 tools (specifically, llc-5.0 and opt-5.0).
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-5.0" 50
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-5.0" 50
# Made sure a "node" binary is in the path, as well as "nodejs".
# A historical naming collision on Debian means that the binary is called "nodejs",
# but some tools like tsc still expect "node" to exist.
ln -s /usr/bin/nodejs /usr/bin/node
# install ocilib dependencies then build and install ocilib
cd /tmp \
@ -189,11 +224,14 @@ cd /tmp \
echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
&& ldconfig
# llvm-4.0 for llvm-hs (separate since it needs wget)
wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
&& add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main" \
&& apt-get update \
&& apt-get install -y llvm-4.0
# Install erlang/otp platform and its dependencies
ERLANG_VERSION="20.2.2"
ERLANG_DEB_FILE="esl-erlang_${ERLANG_VERSION}-1~debian~jessie_amd64.deb"
pushd /tmp \
&& wget https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/${ERLANG_DEB_FILE} \
&& (dpkg -i ${ERLANG_DEB_FILE}; apt-get install -yf) \
&& rm ${ERLANG_DEB_FILE} \
&& popd
# Install version 3 of the protobuf compiler. (The `protobuf-compiler` package only
# supports version 2.)
@ -207,6 +245,15 @@ curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-l
&& rm libtensorflow.tar.gz \
&& ldconfig
# Install libsodium
curl https://download.libsodium.org/libsodium/releases/LATEST.tar.gz > libsodium.tar.gz \
&& sudo tar xfz libsodium.tar.gz -C /tmp \
&& rm libsodium.tar.gz \
&& cd /tmp/libsodium-stable \
&& ./configure \
&& make install
# NOTE: also update Dockerfile when cuda version changes
# Install CUDA toolkit
# The current version can be found at: https://developer.nvidia.com/cuda-downloads
CUDA_PKG=8.0.61-1 # update this on new version
@ -220,12 +267,12 @@ pushd /tmp \
&& apt-get install -y cuda-drivers cuda-core-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} cuda-cufft-dev-${CUDA_APT} cuda-cublas-dev-${CUDA_APT} cuda-cusparse-dev-${CUDA_APT} cuda-cusolver-dev-${CUDA_APT} \
&& rm cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
&& export CUDA_PATH=/usr/local/cuda-${CUDA_VER} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH+x} \
&& export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${LD_LIBRARY_PATH} \
&& export PATH=${CUDA_PATH}/bin:${PATH} \
&& popd
## non-free repo for mediabus-fdk-aac
#apt-add-repository multiverse \
# && apt-get update \
# && apt-get install -y libfdk-aac-dev
# non-free repo for mediabus-fdk-aac
apt-add-repository multiverse \
&& apt-get update \
&& apt-get install -y nvidia-cuda-dev

2
stackage/stackage.cabal
View File

@ -26,4 +26,4 @@ library
source-repository head
type: git
location: git://github.com/fpco/stackage.git
location: git://github.com/commercialhaskell/stackage.git