Merge pull request #1 from commercialhaskell/master

update fork to current master

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,9 @@
+Checklist:
+- [ ] Meaningful commit message - please not `Update build-constraints.yml`
+- [ ] At least 30 minutes have passed since Hackage upload
+- [ ] On your own machine, in a new directory, you have successfully run the following set of commands (replace `$package` with the name of the package that is submitted, `$version` is the version of the package you want to get into Stackage):
+
+      stack unpack $package-$version
+      cd $package-$version
+      stack init --resolver nightly
+      stack build --resolver nightly --haddock --test --bench --no-run-benchmarks

.travis.yml

@@ -11,7 +11,7 @@ addons:
     - libgmp-dev
 
 env:
-- GHCVER=8.2.1
+- GHCVER=8.4.3
 
 install:
  # Download and unpack the stack executable

CURATORS.md

@@ -1,20 +1,28 @@
 This is a collection of instructions covering the processes that the Stackage curators - the
-guys who maintain the Stackage project itself - should be doing on a regular basis.
+team who maintain the Stackage project itself - should be doing on a regular basis.
-Originally this was handled largely by Michael Snoyman,
-but now we are a team of 5 people handling requests weekly in rotation.
 Curation activities are mostly automated, and do not take up a significant amount of time.
+The following is the current list of curators, in alphabetical order:
+
+* Adam Bergmark (@bergmark)
+* Alexey Zabelin (@alexeyzab)
+* Chris Dornan (@cdornan)
+* Dan Burton (@danburton)
+* Jens Petersen (@juhp)
+* Joe Kachmar (@jkachmar)
+* Michael Snoyman (@snoyberg)
+* Mihai Maruseac (@mihaimaruseac)
 
 ## Workflow overview
 
 This section sketches out at a high level how the entire Stackage build/curation
 process works:
 
-* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
+* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
 * [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
 * stackage-curator can check that build plan to ensure all version bounds are consistent
-    * The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
+    * The [Travis job](https://github.com/commercialhaskell/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
-* Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
+* Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
-* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh)
+* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh)
 * When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
 * Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
 * Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
@@ -26,7 +34,7 @@ author only added packages under his/her own name, merge it.  If the
 build later fails (see [Adding Debian packages]), then block the
 package until it's fixed.
 
-[Adding Debian packages]: https://github.com/fpco/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
+[Adding Debian packages]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
 
 If benchmarks, haddocks, or test suites fails at this point we
 typically also block the package until these issues are fixed. This in
@@ -65,13 +73,13 @@ being included immediately. This also applies to when only benchmarks
 and tests are affected.
 
 * Copy the stackage-curator output and create a new issue, see e.g
-https://github.com/fpco/stackage/issues/2108
+https://github.com/commercialhaskell/stackage/issues/2108
 
 * Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
 
 ```yaml
     "Stackage upper bounds":
-        # https://github.com/fpco/stackage/issues/2108
+        # https://github.com/commercialhaskell/stackage/issues/2108
         - pipes < 4.3.0
 ```
 
@@ -82,7 +90,7 @@ https://github.com/fpco/stackage/issues/2108
 
 Sometimes releases for different packages are tightly coupled. Then it
 can make sense to combine them into one issue, as in
-https://github.com/fpco/stackage/issues/2143.
+https://github.com/commercialhaskell/stackage/issues/2143.
 
 If a dependency that is not explicitly in stackage is causing test or
 benchmark failures you can skip or expect them to fail (see "Skipping
@@ -115,7 +123,7 @@ enough time to upgrade for this case as well.
 With the `pipes` example above there was later a new release of
 `pipes-safe` that required the **newer** version of `pipes`. You can
 add that package to the same upper bounds section,
-(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
+(e.g. https://github.com/commercialhaskell/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
 as we want to lift them simultaneously.
 
 ### Skipping tests and benchmarks
@@ -141,7 +149,7 @@ dependencies and compile them.
 
 If there are no version bounds that would fix the issue or if you
 can't figure it out, file it
-(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
+(e.g. https://github.com/commercialhaskell/stackage/issues/2133) to ask the
 maintainer for help.
 
 ### Waiting for new releases
@@ -204,6 +212,9 @@ major version number (e.g., lts3 for lts-3.\*).
 Note that when starting a new LTS major release, you'll need to modify Docker
 Hub to create a new Docker tag for the relevant branch name.
 
+You'll need to update both the `PATH` in `Dockerfile` and the `GHCVER` variable
+in `debian-bootstrap.sh`.
+
 ### Getting the new image to the build server
 Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
 below). Instead of pulling an unbounded number of images, I typically just
@@ -217,7 +228,7 @@ docker rmi $(docker images -q)
 but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
 
 For a new GHC version you should also delete the cache directories on the stackage-build server to
-force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg:
+force all packages to be rebuilt. See: [issue#746](https://github.com/commercialhaskell/stackage/issues/746). Eg:
 ```
 rm -r nightly/work/builds/nightly/
 ```
@@ -230,7 +241,7 @@ following command:
 ```
 DIR=$(mktemp -d)
 (cd $DIR \
-  && git clone https://github.com/fpco/stackage \
+  && git clone https://github.com/commercialhaskell/stackage \
   && cd stackage \
   && docker build --tag snoyberg/stackage:nightly .)
 rm -rf $DIR
@@ -285,6 +296,7 @@ Valid arguments to include in this environment variable:
 * `--add-package` to add a brand new package
 * `--expect-test-failure` to expect tests to fail
 * `--expect-haddock-failure` to expect haddocks to fail
+* `--expect-bench-failure` to expect benchmarks to fail
 
 If a build fails for bounds reasons, see all of the advice above. If the code
 itself doesn't build, or tests fail, open up an issue and then either put in a
@@ -348,7 +360,7 @@ Then, clone the stackage repo, get the latest packages and run dependency
 resolution:
 
 ```
-$ git clone git@github.com:fpco/stackage.git
+$ git clone git@github.com:commercialhaskell/stackage.git
 $ stack update && stackage-curator check
 ```
 
@@ -360,12 +372,12 @@ test suites and benchmarks, to check whether bounds can be lifted, and to get
 errors for builds, tests and benchmarks.
 
 [the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
-[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
+[tell-me-when-its-released]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#waiting-for-new-releases
 
 ## Adding new curators
 
 1. Add public ssh key to `~/.ssh/authorized_keys` on build server
-2. Add to fpco/stackage project.
+2. Add to commercialhaskell/stackage project.
 
 ## Dealing with a new GHC release
 
@@ -389,4 +401,26 @@ After the first LTS release, the package pruning process may begin in the
 nightly build in order to move forward with getting the latest versions of
 packages compatible with the new GHC release.
 
-[GHC upgrade note]: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
+[GHC upgrade note]: https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
+
+## New LTS major bump
+
+Every 3-6 months, we make a new major release of LTS. The procedure we follow for this is:
+
+1. Write a blog post on stackage.org announcing the intent to cut a major
+   release. Give an estimated date two weeks in the future from the publication
+   date of the post.
+2. Spread the blog post on social media and mailing lists as much as possible.
+3. Expect maintainers to send significant requests for added packages and
+   relaxed upper bounds. There will likely be some hard decisions to be made
+   regarding relaxing a bound versus keeping more packages. All of these changes
+   occur on master and affect nightly.
+4. Once the estimated date hits, push a new `lts-XX` branch and trigger Docker
+   Hub to build a Docker image for the new release.
+5. Run the build procedure for the new LTS release.
+6. After the LTS build completes, more aggressively prune upper bounds from
+   `build-constraints.yaml`.
+7. Once both (5) and (6) are done, publish a new blog post on stackage.org
+   announcing the new LTS and Nightly, with links to the change pages on
+   stackage.org. Include a reminder that requests for packages to be added to LTS
+   may be made on commercialhaskell/lts-haskell.

DATA-FLOW.md

@@ -15,7 +15,7 @@ There are three inputs into the data flow:
   Hackage provides both cabal file metadata (via the 00-index.tar file) and
   tarballs of the individual packages.
 
-* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
+* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml)
   is the primary Stackage input file. This is where package maintainers can add
   packages to the Stackage package set. This also defines upper bounds, skipped
   tests, and a few other pieces of metadata.
@@ -60,7 +60,7 @@ The heart of running Stackage builds is the
 [stackage-curator](https://github.com/fpco/stackage-curator) tool. We run this
 on a daily basis on the Stackage build server for Stackage Nightly, and on a
 weekly basis for LTS Haskell. The build process is [highly
-automated](https://github.com/fpco/stackage/blob/master/automated/build.sh) and
+automated](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh) and
 leverages Docker quite a bit.
 
 stackage-curator needs to know about the most recent versions of all packages,
@@ -80,7 +80,7 @@ By combining these constraints with the current package data, stackage-curator
 can generate a build plan and check it. (As an aside, this build plan
 generation and checking also occurs every time you make a pull request to the
 stackage repo.) If there are version bounds problems, one of the [Stackage
-curators](https://github.com/fpco/stackage/blob/master/CURATORS.md) will open
+curators](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md) will open
 up a Github issue and will add upper bounds, temporarily block a package, or
 some other corrective action.
 

Dockerfile

@@ -2,7 +2,11 @@ FROM fpco/pid1:16.04
 
 ENV HOME /home/stackage
 ENV LANG en_US.UTF-8
-ENV PATH /opt/ghc/8.2.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+# NOTE: also update debian-bootstrap.sh when cuda version changes
+ENV PATH /usr/local/cuda-8.0/bin:/opt/ghc/8.4.3/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ENV CUDA_PATH /usr/local/cuda-8.0
+ENV LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/nvvm/lib64
 
 ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
 RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh

MAINTAINERS.md

@@ -12,89 +12,151 @@ Packages in Stackage are not patched: all package changes occur upstream in Hack
 
 Anyone can add any package to Stackage but you may only add packages under your own name. It's highly encouraged that the actual package maintainer is also the Stackage maintainer, if that is not the case you should drop the package maintainer a note first.
 
-To add your package, first fork this repository.
+To add your package you can edit [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) directly on github or fork the project. There's a section called `packages` where you would add yourself and your packages:
-In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`.
-To add a set of packages, you would add:
 
     "My Name <myemail@example.com> @mygithubuser":
         - package1
         - package2
         - package3
 
-Note that the `master` branch is used for Stackage Nightly (not the `nightly` branch, which is used for the nightly docker builder imagine).
+If your library depends on a C library, please add it to the `debian-bootstrap.sh` script.
 
-After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
+After doing that commit with a message like "add foo-bar" and send a pull request.
+
+The continuous integration job will do some checks to see if your package's dependencies are up-to-date.
+
+The CI job notably doesn't compile packages, run tests, build documentation, or find missing C libraries.
+If you want to be proactive or if CI fails, you can make sure that your package builds against the latest nightly:
 
-If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
 ```
+# Build from the tarball on Hackage to check for missing files
+$ stack unpack yourpackage && cd yourpackage-*
+# Generate a pristine stack.yaml, adding any missing extra-deps
+$ rm -f stack.yaml && stack init --resolver nightly --solver
+# Build, generate docs, test, and build benchmarks
+$ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
+```
+
+This approach works well, but has two limitations you should be aware
+of:
+
+* It won't notify you of restrictive upper bounds in your package if
+  Stackage has the same upper bounds. For that reason, we recommend
+  using [Packdeps](http://packdeps.haskellers.com/) (see "Following
+  dependency upgrades" below).
+* If the latest Stackage Nightly is missing some of the latest
+  packages, your build above may succeed whereas the Travis job may
+  fail. Again: Packdeps will help you detect this situation.
+
+Alternatively, you can build with `cabal`. Note that this may end up
+using older dependency versions:
+
+```
+$ ghc --version # Should be the same as the latest nightly, it's in the title of https://www.stackage.org/nightly
 $ cabal update
-$ ghc --version # Should give v8.0.2
+$ cabal get PACKAGE
-$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
+$ cd PACKAGE-*
-$ cd PACKAGE-VERSION
 $ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
 $ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
 $ cabal install --enable-tests --enable-benchmarks --allow-newer
 $ cabal test
+$ cabal haddock
 ```
 
-**NB** Please use commit messages like "add foo-bar" or "add johndev's packages"
+## Github and Notifications
-(`build-constraints.yaml` is the most frequently changed file in this git repo
-so commit messages like "update build-constraints.yaml" are not helpful).
 
-**NB2** There can be a delay of up to an hour before package versions
+Stackage uses Github notifications to reduce overhead of contacting individual
-newly uploaded to Hackage appear to our build server. If you just
+package maintainers through various channels. As a package maintainer, you will
-uploaded a package to Hackage that you're trying to get included, we
+receive notifications for a number of reasons, package build failures of
-recommend waiting an hour before opening the PR. You can verify this
+different sorts, blockages and bounds issues, etc.
-by making sure the latest version is listed at 
+
-https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/.
+Please note, Github does some throttling on the number people that can be
+notified within one issue. This means that on issues created with a large
+number of packages affected, maintainers may not receive a notification. This
+is not ideal, but Stackage is largely a manual process and done on a best
+effort basis.
 
 ## Uploading a new package version
 
-When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
+When a new version of a package in Stackage is uploaded to Hackage, we automatically try to include it in Stackage. That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and open a GitHub issue ticket to resolve the issue.
 
 If the new version doesn't compile then the package author should upload a fixed version.
 
 If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite.
 
-
 ## Following dependency upgrades
 
-If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
+If a new version of a dependency is released your package may fail to
+build or tests may fail. In most cases we will add an upper bound on
+the dependency so that your package remains in nightly. We'll also
+create a github issue pinging you with the bounds issues or give build
+logs showing failures. It's then up to you to modify your package.
 
-If a package is not modified in a timely manner, it may be temporarily
+We recommend that you also follow the dependencies of your packages on
-removed from Stackage by the curator team, at which point it is your
+[Packdeps](http://packdeps.haskellers.com/) (typically using the RSS
-responsibility to add it back in via a new pull request. We typically
+feeds) as well as that often gives you notice ahead of stackage
-use fairly long windows on this, but at a minimum:
+issues. There are cases where we will not notice a new release of a
+package because of other upper bounds that are in place.
 
-* If restrictive version bounds are the only problem, we will give at
+If a package is not updated in time, it may be temporarily removed
-  least a week to respond.
+from Stackage by the curator team. We strive to notify you when this
+happens. If it does you are always welcome to file another pull
+request to add it back.
+
+We typically use fairly long windows before disabling packages, but it
+is decided on a case-by-case basis.
+
+* If restrictive version bounds are the only problem, we will give
+  maintainers at least a week to respond.
 * If there are real breaking changes, the curator team will retain
   more discretion on how long a window to give before dropping
-  packages. Historically, this has usually waited until the cutting of
+  packages.
+* We usually drop all upper bounds and disable packages when we create
   a new Long Term Support (LTS) major version.
+* There are rare cases where an upper bound or build failure are hard
+  to deal with so then we may disable
 
-**NOTE** Previously, this maintainer agreement put a time limit on
+**NOTE** Previously we had stricter time limits, but we decided to
-maintainers, requiring a certain level of responsiveness for
+remove that: Anyone is free to add a package to Stackage regardless of
-modifications to be made. We have explicitly removed that: anyone is
+responsiveness guarantees. However, as stated above, we may elect to
-free to add a package to Stackage regardless of responsiveness
+temporarily remove a package if it is not updated in a timely manner.
-guarantees. However, as stated above, we may elect to temporarily
-remove a package if it is not updated in a timely manner.
 
-## Failing to meet the time limits
+## Delays
 
-Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack.
+Maintainers are humans, humans get sick/have babies/go on
+holiday. Sometimes a dependency upgrade is extra time consuming.
 
-The soft time limits are intended to prevent people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for longer periods.
+Consider finding a co-maintainer with access to SCM and Hackage to
+help you out.
+
+We appreciate if you notify us of any expected delays in the Github
+issues, some kind soul might decide to help out.
 
 ## Upgrading to a new GHC version
 
-The Stackage curation team tries to move Stackage Nightly to new versions of GHC quickly as they become available, while keeping LTS Haskell on a regular release schedule. For package maintainers, the most important impacts of a new GHC release are:
+The Stackage curation team tries to move Stackage Nightly to new
+versions of GHC quickly as they become available, while keeping LTS
+Haskell on a regular release schedule. For package maintainers, the
+most important impacts of a new GHC release are:
 
-* We will typically do a sweep through the Stackage upper bounds and aggressively remove packages that block them. This is because, in most cases, we will need to move to the newest versions of a package to get support for the latest GHC, and asking package maintainers to backport their fixes is an undue burden
+* We will typically do a sweep through the Stackage upper bounds and
-    * We will definitely do this at a GHC major version release, and may do so at a minor version release
+  remove blocking packages. We prefer to do it this way rather than
-* Packages that are incompatible with the newest GHC version will be temporarily blocked
+  ask other maintainers to backport fixes.
+    * We will definitely do this at a GHC major version release, and
+      may do so at a minor version release
+* Packages that are incompatible with the newest GHC version will be
+  temporarily disabled
 
-If your package ends up being temporarily removed from Stackage Nightly, please simply send a pull request to add it back once it and its dependencies are compatible with the newest GHC version.
+If your package ends up being temporarily disabled from Stackage
+Nightly, please simply send a pull request to add it back once it and
+its dependencies are compatible with the newest GHC version.
+
+Note that it is _not_ a goal of LTS Haskell to track the latest
+version of GHC. If you want the latest and greatest, Stackage Nightly
+is your best bet. In particular, LTS Haskell will often&mdash;but not
+always&mdash;avoid upgrading to the first point release of GHC
+releases (e.g., 8.2.1) to allow further testing and to get the
+benefits of the first bugfix release (e.g., 8.2.2).
 
 ## Adding a package to an LTS snapshot
 
@@ -108,8 +170,53 @@ If you would like to get your package added to an existing LTS Haskell major
 release (e.g., if `lts-8.9` is out, you would want your package to appear in
 `lts-8.10`), please do the following in addition to the steps above:
 
+* Check that your package can be built with that LTS major version (e.g. `stack build --test --bench --haddock --resolver lts-8.10`)
 * Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
-* Specify the LTS major version you would like your package to go into (e.g., lts-8)
+  * Specify the LTS major versions you would like your packages to go into (e.g. lts-8)
-* Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
+  * Provide a list of packages you would like added
-* Be patient! The LTS releases are less frequent than Nightly and by their nature more conservative, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue as soon as possible, but it may take some time.
+    * If relevant, mention any upper bounds that are needed on those packages
-* To add a package to more than one LTS version please file separate tickets for each major LTS release, since they will typically be built and added at different times.
+* Be patient! The LTS releases are less frequent than Nightly. The
+  Stackage curators will try to get to your issue as soon as possible,
+  but it may take some time.
+* We gradually stop maintainng old LTS major versions, so your
+  request may take longer or be declined if it's for an old LTS.
+
+## LTS package guarantees and exceptions
+
+In general, we try to stick to some rules when it comes to the
+packages included in LTS minor bumps. In particular:
+
+* If a package exists in LTS-X.Y, it should also exist in LTS-X.(Y+1)
+* We should not include a major version bump of a package between
+  LTS-X.Y and LTS-X.(Y+1)
+
+However, there are some cases where exceptions may be made, based
+purely on Stackage Curator discretion. The most common examples are:
+
+*   If a package does not follow the PVP in its version number policy,
+    applying the standard version bump rules would not necessarily
+    makes sense. As an example, suppose package `foo` decides to
+    follow SemVer instead of the PVP. By our standard rules of version
+    bumps, a change from `foo-1.2.0` to `foo-1.3.0` would be
+    considered a major version bump, and disallowed in an LTS minor
+    version bump. However, if a package is following SemVer, this
+    would not be a breaking change, and curators may elect to include
+    it.
+
+*   If a package has overly restrictive version bounds on a
+    dependency, in particular constraining a minor version
+    unnecessarily, we may drop that package instead of artificially
+    holding back the dependency. As an example: suppose `LTS-20.1`
+    includes `foo-1.2.0` and `bar-1`. `bar-1` has a dependency `foo >=
+    1.2.0 && < 1.2.1`, which is overly constrained on the minor
+    version number according to the PVP. Then `foo-1.2.1` is
+    released. The Stackage Curator team would have two choices:
+
+    * Reject `foo-1.2.1` from `LTS-20.2`, since that is what `bar-1`
+      requires.
+    * Drop `bar-1` from `LTS-20.2`, and allow `foo-1.2.1`.
+
+    Decisions will need to be taken on a case-by-case basis, and may
+    depend on such issues as whether an important bugfix or security
+    update in included. The curator team may also try to notify the
+    author of `bar` to try and get a patched version released.

README.md

@@ -1,7 +1,7 @@
 stackage
 ========
 
-[![Build Status](https://travis-ci.org/fpco/stackage.svg?branch=master)](https://travis-ci.org/fpco/stackage)
+[![Build Status](https://travis-ci.org/commercialhaskell/stackage.svg?branch=master)](https://travis-ci.org/commercialhaskell/stackage)
 
 "Stable Hackage": creating a vetted set of packages from Hackage.
 This repository is for package authors and maintainers to get their packages into Stackage.
@@ -16,12 +16,12 @@ Add your package
 
 We welcome all packages, provided:
 
-* The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
+* The package author/maintainer agrees to the [maintainers agreement](https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md).
 * The package is buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
 * The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
 * The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
 
-Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package).
+Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#adding-a-package).
 
 __NOTE__: There is an approximate 30 minute delay between a package uploading
 to Hackage and being available to the Travis build script to check upper
@@ -37,7 +37,6 @@ project information. In addition, we have the following repositories:
 
 * [stackage-server](https://github.com/fpco/stackage-server) [![Build Status](https://travis-ci.org/fpco/stackage-server.svg?branch=master)](https://travis-ci.org/fpco/stackage-server)
 * [stackage-curator](https://github.com/fpco/stackage-curator) [![Build Status](https://travis-ci.org/fpco/stackage-curator.svg?branch=master)](https://travis-ci.org/fpco/stackage-curator)
-* [stackage-types](https://github.com/fpco/stackage-types) [![Build Status](https://travis-ci.org/fpco/stackage-types.svg?branch=master)](https://travis-ci.org/fpco/stackage-types)
 * [lts-haskell](https://github.com/fpco/lts-haskell)
 * [stackage-nightly](https://github.com/fpco/stackage-nightly)
 
@@ -51,17 +50,17 @@ Stackage both easier and more secure:
 * [stackage-build-plan](https://github.com/fpco/stackage-build-plan) [![Build Status](https://travis-ci.org/fpco/stackage-build-plan.svg?branch=master)](https://travis-ci.org/fpco/stackage-build-plan)
 
 Curious how it all fits together? See the [Stackage data
-flow](https://github.com/fpco/stackage/blob/master/DATA-FLOW.md).
+flow](https://github.com/commercialhaskell/stackage/blob/master/DATA-FLOW.md).
 
 
 Build the package set
 ---------------------
 
 Generally only the stackage build server run by the stackage curator
-team and people intrested in incorporating stackage snapshots into an
+team and people interested in incorporating stackage snapshots into an
 OS distribution need to build the entire package set. If you're
 interested in trying this yourself, please check out
-[the curator guide](https://github.com/fpco/stackage/blob/master/CURATORS.md),
+[the curator guide](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md),
 though be aware that this is not a recommended practice and there
 likely will be problems you will need to debug yourself.
 
@@ -103,7 +102,7 @@ There are a number of answers to this question:
   another update in the LTS 6 line
 * Sometimes we have upper bounds in place because other packages have
   problems with newer versions of dependencies. Open up the
-  [build-constraints file](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
+  [build-constraints file](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml)
   and search for "Stackage upper bounds"
 * Wired-in packages - those that ship with GHC and cannot be upgraded,
   and packages depending on them - are fixed to GHC versions. Common

automated/build.sh

@@ -21,7 +21,6 @@ IMAGE=snoyberg/stackage:$TAG
 
 PLAN_FILE=current-plan.yaml
 DOCMAP_FILE=current-docmap.yaml
-BUNDLE_FILE=current.bundle
 
 CABAL_DIR=$ROOT/cabal
 STACK_DIR=$ROOT/stack
@@ -30,6 +29,7 @@ DOT_STACKAGE_DIR=$ROOT/dot-stackage
 WORKDIR=$ROOT/$TAG/work
 EXTRA_BIN_DIR=$ROOT/extra-bin
 SSH_DIR=$ROOT/ssh-$SHORTNAME
+USERID=$(id -u)
 
 mkdir -p \
 	"$CABAL_DIR" \
@@ -81,9 +81,9 @@ chmod +x stackage-curator
 )
 
 ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
-ARGS_PREBUILD="$ARGS_COMMON -u $USER -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
+ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
 ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
-ARGS_UPLOAD="$ARGS_COMMON -u $USER -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
+ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
 
 # Make sure we actually need this snapshot. We only check this for LTS releases
 # since, for nightlies, we'd like to run builds even if they are unnecessary to
@@ -118,10 +118,15 @@ fi
 # * Do a single unpack to create the package index cache (again due to directory perms)
 docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
 
+case $SHORTNAME in
+    lts) JOBS=1 ;;
+    nightly) JOBS=2 ;;
+esac
+
 # Now do the actual build. We need to first set the owner of the home directory
 # correctly, so we run the command as root, change owner, and then use sudo to
 # switch back to the current user
-docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs 4 --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
+docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs $JOBS --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET"
 
 # Make sure we actually need this snapshot. We used to perform this check
 # exclusively before building. Now we perform it after as well for the case of
@@ -136,7 +141,7 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-
 # * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
 # * Upload the new plan .yaml file to the appropriate Github repo
 # * Register as a new Hackage distro
-docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
+docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
 
 echo -n "Completed at "
 date

become-a-curator.md

@@ -37,7 +37,7 @@ The current curator team consists of:
 We onboarded Luke as the newest member a few months ago and this
 helped us iron out and document the process further.
 
-You can read the curator documentation here: https://github.com/fpco/stackage/blob/master/CURATORS.md
+You can read the curator documentation here: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md
 
 To apply, please fill in this form: [REDACTED]
 

check

@@ -3,4 +3,4 @@
 # Convenience script for checking constraints locally
 
 cd `dirname $0`
-exec stack exec --resolver ghc-8.2.1 stackage-curator check
+exec stack exec --resolver ghc-8.4.3 stackage-curator check

debian-bootstrap.sh

@@ -4,7 +4,7 @@
 # up-and-running on a freshly installed Debian-based system (including Ubuntu).
 
 # Quick start:
-# wget -O - https://raw.github.com/fpco/stackage/master/debian-bootstrap.sh | bash -ex
+# wget -O - https://raw.github.com/commercialhaskell/stackage/master/debian-bootstrap.sh | bash -ex
 
 # NOTE: Requires that GHC and Cabal are installed and on your PATH. For
 # instructions, see:
@@ -25,19 +25,25 @@ add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://down
 add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
 add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
 
-GHCVER=8.2.1
+GHCVER=8.4.3
 
 apt-get update
 apt-get install -y \
+    apt-transport-https \
     build-essential \
+    cmake \
     curl \
     freeglut3-dev \
+    freetds-dev \
     fsharp \
+    g++ \
+    gawk \
     ghc-$GHCVER \
     ghc-$GHCVER-dyn \
     ghc-$GHCVER-htmldocs \
     ghc-$GHCVER-prof \
     git \
+    gnupg \
     gradle \
     hscolour \
     libadns1-dev \
@@ -74,6 +80,7 @@ apt-get install -y \
     libicu-dev \
     libimlib2-dev \
     libjack-jackd2-dev \
+    libjavascriptcoregtk-4.0-dev \
     libjudy-dev \
     liblapack-dev \
     libleveldb-dev \
@@ -89,11 +96,13 @@ apt-get install -y \
     libmysqlclient-dev \
     libncurses-dev \
     libnfc-dev \
+    liboath-dev \
     libnotify-dev \
     libopenal-dev \
     libpango1.0-dev \
     libpcap0.8-dev \
     libpq-dev \
+    libre2-dev \
     libsdl1.2-dev \
     libsdl2-dev \
     libsdl2-gfx-dev \
@@ -117,6 +126,7 @@ apt-get install -y \
     libxss-dev \
     libyaml-dev \
     libzip-dev \
+    libzstd-dev \
     libzmq3-dev \
     llvm-3.9 \
     locales \
@@ -124,6 +134,7 @@ apt-get install -y \
     minisat \
     mono-mcs \
     nettle-dev \
+    ninja-build \
     nodejs \
     npm \
     openjdk-8-jdk \
@@ -135,12 +146,19 @@ apt-get install -y \
     r-base-dev \
     ruby-dev \
     sudo \
+    unixodbc-dev \
     wget \
     xclip \
     z3 \
     zip \
     zlib1g-dev
 
+# odbc
+curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
+curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
+apt-get update
+ACCEPT_EULA=Y apt-get install msodbcsql17 -y
+
 locale-gen en_US.UTF-8
 
 curl -sSL https://get.haskellstack.org/ | sh
@@ -148,6 +166,18 @@ curl -sSL https://get.haskellstack.org/ | sh
 # Put documentation where we expect it
 mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
 
+# llvm-5.0 for GHC (separate since it needs wget)
+wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
+    && add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-5.0 main" \
+    && apt-get update \
+    && apt-get install -y llvm-5.0
+
+# llvm-6.0 for llvm-hs (separate since it needs wget)
+wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
+    && add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main" \
+    && apt-get update \
+    && apt-get install -y llvm-6.0
+
 # Buggy versions of ld.bfd fail to link some Haskell packages:
 # https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
 # faster anyways and uses less RAM.
@@ -158,9 +188,14 @@ update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
 # This version is tracked here:
 # https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
 #
-# GHC 8.2 requires LLVM 3.9 tools (specifically, llc-3.9 and opt-3.9).
+# GHC 8.4 requires LLVM 5.0 tools (specifically, llc-5.0 and opt-5.0).
-update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.9" 50
+update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-5.0" 50
-update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.9" 50
+update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-5.0" 50
+
+# Made sure a "node" binary is in the path, as well as "nodejs".
+# A historical naming collision on Debian means that the binary is called "nodejs",
+# but some tools like tsc still expect "node" to exist.
+ln -s /usr/bin/nodejs /usr/bin/node
 
 # install ocilib dependencies then build and install ocilib
 cd /tmp \
@@ -189,11 +224,14 @@ cd /tmp \
 echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
     && ldconfig
 
-# llvm-4.0 for llvm-hs (separate since it needs wget)
+# Install erlang/otp platform and its dependencies
-wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
+ERLANG_VERSION="20.2.2"
-    && add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main" \
+ERLANG_DEB_FILE="esl-erlang_${ERLANG_VERSION}-1~debian~jessie_amd64.deb"
-    && apt-get update \
+pushd /tmp \
-    && apt-get install -y llvm-4.0
+    && wget https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/${ERLANG_DEB_FILE} \
+    && (dpkg -i ${ERLANG_DEB_FILE}; apt-get install -yf) \
+    && rm ${ERLANG_DEB_FILE} \
+    && popd
 
 # Install version 3 of the protobuf compiler.  (The `protobuf-compiler` package only
 # supports version 2.)
@@ -207,6 +245,15 @@ curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-l
     && rm libtensorflow.tar.gz \
     && ldconfig
 
+# Install libsodium
+curl https://download.libsodium.org/libsodium/releases/LATEST.tar.gz > libsodium.tar.gz \
+	&& sudo tar xfz libsodium.tar.gz -C /tmp \
+	&& rm libsodium.tar.gz \
+	&& cd /tmp/libsodium-stable \
+	&& ./configure \
+	&& make install
+
+# NOTE: also update Dockerfile when cuda version changes
 # Install CUDA toolkit
 # The current version can be found at: https://developer.nvidia.com/cuda-downloads
 CUDA_PKG=8.0.61-1         # update this on new version
@@ -220,12 +267,12 @@ pushd /tmp \
     && apt-get install -y cuda-drivers cuda-core-${CUDA_APT} cuda-cudart-dev-${CUDA_APT} cuda-cufft-dev-${CUDA_APT} cuda-cublas-dev-${CUDA_APT} cuda-cusparse-dev-${CUDA_APT} cuda-cusolver-dev-${CUDA_APT} \
     && rm cuda-repo-ubuntu1604_${CUDA_PKG}_amd64.deb \
     && export CUDA_PATH=/usr/local/cuda-${CUDA_VER} \
-    && export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH} \
+    && export LD_LIBRARY_PATH=${CUDA_PATH}/nvvm/lib64:${LD_LIBRARY_PATH+x} \
     && export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${LD_LIBRARY_PATH} \
     && export PATH=${CUDA_PATH}/bin:${PATH} \
     && popd
 
-## non-free repo for mediabus-fdk-aac
+# non-free repo for mediabus-fdk-aac
-#apt-add-repository multiverse \
+apt-add-repository multiverse \
-#    && apt-get update \
+    && apt-get update \
-#    && apt-get install -y libfdk-aac-dev
+    && apt-get install -y nvidia-cuda-dev

stackage/stackage.cabal

@@ -26,4 +26,4 @@ library
 
 source-repository head
   type:     git
-  location: git://github.com/fpco/stackage.git
+  location: git://github.com/commercialhaskell/stackage.git