Merge branch 'master' of github.com:commercialhaskell/stackage

2026-01-13 07:48:31 +01:00 · 2019-09-13 10:25:45 -05:00 · 2019-09-13 10:25:45 -05:00 · 95130edd80
commit 95130edd80
parent 1bafd11534 4056a7cef9
15 changed files with 420 additions and 322 deletions
--- a/.dockerignore
+++ b/.dockerignore
@ -6,3 +6,5 @@ cabal.sandbox.config
 tarballs
 *.yaml
 .git
+automated
+.github
--- a/.gitignore
+++ b/.gitignore
@ -5,3 +5,6 @@ lts-*.yaml
 *.swp
 check-plan.yaml
 *~
+/constraints.yaml
+/snapshot.yaml
+/snapshot-incomplete.yaml
--- a/CURATORS.md
+++ b/CURATORS.md
@ -18,8 +18,8 @@ This section sketches out at a high level how the entire Stackage build/curation
 process works:

 * [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
-* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
-* stackage-curator can check that build plan to ensure all version bounds are consistent
+* [curator](https://github.com/commercialhaskell/stack/tree/master/subs/curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
+* `curator` can check that build plan to ensure all version bounds are consistent
    * The [Travis job](https://github.com/commercialhaskell/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
 * Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://hub.docker.com/r/commercialhaskell/stackage/) for running builds
 * The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh)
@ -72,7 +72,7 @@ Most common technique, just prevent a new version of a library from
 being included immediately. This also applies to when only benchmarks
 and tests are affected.

-* Copy the stackage-curator output and create a new issue, see e.g
+* Copy the `curator` output and create a new issue, see e.g
 https://github.com/commercialhaskell/stackage/issues/2108

 * Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
@ -84,7 +84,7 @@ https://github.com/commercialhaskell/stackage/issues/2108
 ```

 * Commit (message e.g. "Upper bound for #2108")
-* Optionally: Verify with `stackage-curator check` locally
+* Optionally: Verify with `./check` locally
 * Push
 * Verify that everything works on the build server (you can restart the build or wait for it to to run again)

@ -110,7 +110,7 @@ new package may appear if its dependencies were part of this issue but
 have been updated since the last time we checked. We want to give
 these new packages ample time to be upgraded.

-If stackage-curator is happy commit the change ("Remove upper bounds
+If `curator` is happy commit the change ("Remove upper bounds
 and close #X"). After doing this the next nightly build may fail
 because some packages didn't have an upper bound in place, but
 compilation failed. In this case revert the previous commit so any
@ -367,30 +367,30 @@ We do not run the full stackage build locally as that might take too
 much time. However, some steps on the other hand are much faster to do
 yourself, e.g. verifying constraints without building anything.

-To get started, install `stackage-curator` via Git, or [the Linux binary]:
+To get started, install `curator` via Git:

 ```
-$ git clone git@github.com:fpco/stackage-curator.git
-$ cd stackage-curator && stack install
+$ git clone git@github.com:commercialhaskell/curator.git
+$ cd curator && stack install curator
 ```

-It is a good idea to upgrade `stackage-curator` at the start of your week.
+It is a good idea to upgrade `curator` at the start of your week.
 Then, clone the stackage repo, get the latest packages and run dependency
 resolution:

 ```
 $ git clone git@github.com:commercialhaskell/stackage.git
-$ stack update && stackage-curator check
+$ cd stackage
+$ ./check
 ```

 This can be used to make sure all version bounds are in place, including for
 test suites and benchmarks, to check whether bounds can be lifted, and to get
 [tell-me-when-its-released] notifications.

-`stackage-curator` does not build anything, so you wont see any compilation
+`curator` does not build anything, so you wont see any compilation
 errors for builds, tests and benchmarks.

-[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
 [tell-me-when-its-released]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#waiting-for-new-releases

 ## Adding new curators
--- a/MAINTAINERS.md
+++ b/MAINTAINERS.md
@ -35,7 +35,7 @@ If you want to be proactive or if CI fails, you can make sure that your package
 # Build from the tarball on Hackage to check for missing files
 $ stack unpack yourpackage && cd yourpackage-*
 # Generate a pristine stack.yaml, adding any missing extra-deps
-$ rm -f stack.yaml && stack init --resolver nightly --solver
+$ rm -f stack.yaml && stack init --resolver nightly
 # Build, generate docs, test, and build benchmarks
 $ stack build --resolver nightly --haddock --test --bench --no-run-benchmarks
 ```
--- a/automated/build-next.sh
+++ b/automated/build-next.sh
@ -5,6 +5,8 @@ set -eu +x -o pipefail
 ROOT=$(cd $(dirname $0) ; pwd)
 TARGET=$1

+source aws.sh
+
 # For nightly-YYYY-MM-DD, tag should be nightly
 # For lts-X.Y, tag should be ltsX
 SHORTNAME=$(echo $TARGET | cut -d- -f 1)
@ -61,25 +63,25 @@ require_400_file "$HACKAGE_CREDS"
 mkdir -p $ROOT/bin
 BINDIR=$(cd $ROOT/bin ; pwd)
 (
+# See etc/curator-exes/README.md
+CURATOR_EXES=95a207fb4d5bd2e2a255350ce18f55976344eeb71e6f27a25f63e8dba28a4cd1
 cd $BINDIR
-rm -f stackage-curator stackage-curator-2*.bz2
-CURATOR2=stackage-curator-2-f6258124cff9a7e92bcb5704164a70e149080e88
-wget "https://download.fpcomplete.com/stackage-curator-2/$CURATOR2.bz2"
-bunzip2 "$CURATOR2.bz2"
-chmod +x $CURATOR2
-mv $CURATOR2 stackage-curator
-./stackage-curator --version
+rm -f curator stack *.bz2
+wget "https://s3.amazonaws.com/download.fpcomplete.com/curator-exes/curator-exes-$CURATOR_EXES.tar.bz2"
+tar xf "curator-exes-$CURATOR_EXES.tar.bz2"
+echo curator: $(./curator --version)
+echo stack: $(./stack --version)
 )

 # We share pantry directory between snapshots while the other content in .stack
 # is stored separately (because e.g. Ubuntu releases between LTS and nightly
 # could differ). Also the order of binds is important.
-ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $BINDIR/stack:/usr/bin/stack:ro -v $STACK_DIR:$HOME/.stack -v $PANTRY_DIR:$HOME/.stack/pantry"
+ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/curator:/usr/bin/curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $BINDIR/stack:/usr/bin/stack:ro -v $STACK_DIR:$HOME/.stack -v $PANTRY_DIR:$HOME/.stack/pantry"
 ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
 ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $GHC_DIR:$HOME/.ghc:ro"
 # instance-data is an undocumented feature of S3 used by amazonka,
 # see https://github.com/brendanhay/amazonka/issues/271
-ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro --add-host instance-data:169.254.169.254"
+ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY"

 # Make sure we actually need this snapshot. We only check this for LTS releases
 # since, for nightlies, we'd like to run builds even if they are unnecessary to
@ -87,21 +89,9 @@ ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $HACKAGE_CREDS:/hackage-cr
 # below for why this is safe.)
 if [ $SHORTNAME = "lts" ]
 then
-  docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
+  docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator check-target-available --target $TARGET"
 fi

-# Get latest stack
-(
-cd $BINDIR
-rm -f stack stack-*.bz2
-STACK=stack-f6258124cff9a7e92bcb5704164a70e149080e88
-wget "https://download.fpcomplete.com/stackage-curator-2/$STACK.bz2"
-bunzip2 "$STACK.bz2"
-chmod +x $STACK
-mv $STACK stack
-./stack --version
-)
-

 # Determine the new build plan unless NOPLAN is set
 #
@ -111,10 +101,12 @@ if [ "${NOPLAN:-}x" = "x" ]
 then
    if [ $SHORTNAME = "lts" ]
    then
-        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator constraints --target $TARGET && stackage-curator snapshot-incomplete && stackage-curator snapshot"
+        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator constraints --target $TARGET && curator snapshot-incomplete --target $TARGET && curator snapshot"
    else
-        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator update && stackage-curator constraints --target $TARGET && stackage-curator snapshot-incomplete && stackage-curator snapshot"
+        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator update && curator constraints --target $TARGET && curator snapshot-incomplete --target $TARGET && curator snapshot"
    fi
+else
+    docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "curator snapshot"
 fi


@ -122,7 +114,7 @@ fi
 #
 # * Check that the snapshot is valid
 # * Fetch and unpack all needed tarballs (the build step does not have write access to the tarball directory)
-docker run $ARGS_PREBUILD $IMAGE /bin/bash -c 'GHCVER=$(sed -n "s/^ghc-version: \(.*\)/\1/p" constraints.yaml) && stack setup ghc-$GHCVER --verbosity=error && stack exec --resolver=ghc-$GHCVER stackage-curator check-snapshot && stackage-curator unpack'
+docker run $ARGS_PREBUILD $IMAGE /bin/bash -c 'GHCVER=$(sed -n "s/^ghc-version: \(.*\)/\1/p" constraints.yaml) && stack setup ghc-$GHCVER --verbosity=error && stack exec --resolver=ghc-$GHCVER curator check-snapshot && curator unpack'

 case $SHORTNAME in
    lts) JOBS=1 ;;
@ -132,25 +124,24 @@ esac
 # Now do the actual build. We need to first set the owner of the home directory
 # correctly, so we run the command as root, change owner, and then use sudo to
 # switch back to the current user
-docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator build --jobs $JOBS" 2>&1 | tee "$SHORTNAME-build.log"
+docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" curator build --jobs $JOBS" 2>&1 | tee "$SHORTNAME-build.log"

 # Make sure we actually need this snapshot. We used to perform this check
 # exclusively before building. Now we perform it after as well for the case of
 # nightly, where we don't perform this check beforehand. This is also slightly
 # safer, in case someone else already uploaded a specific snapshot while we
 # were building.
-docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
+docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec curator check-target-available --target $TARGET"

 # Successful build, so we need to:
 #
 # * Upload the docs to S3
 # * Upload the new snapshot .yaml file to the appropriate Github repo, also upload its constraints
 # * Register as a new Hackage distro (currently disabled)
-docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET && exec stackage-curator upload-github --target $TARGET"
-# FIXME - add back "stackage-curator hackage-distro --target $TARGET" when we will be ready to publish
+docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "curator upload-docs --target $TARGET && curator upload-github --target $TARGET && exec curator hackage-distro --target $TARGET"
 # information about the new snapshots on Hackage

-$BINDIR/stackage-curator legacy-bulk --stackage-snapshots dot-stackage/curator/stackage-snapshots/ --lts-haskell dot-stackage/curator/lts-haskell/ --stackage-nightly dot-stackage/curator/stackage-nightly/
+$BINDIR/curator legacy-bulk --stackage-snapshots dot-stackage/curator/stackage-snapshots/ --lts-haskell dot-stackage/curator/lts-haskell/ --stackage-nightly dot-stackage/curator/stackage-nightly/

 (

--- a/automated/build.sh
+++ b/automated/build.sh
@ -1,148 +0,0 @@
-#!/usr/bin/env bash
-
-set -eu +x
-
-ROOT=$(cd $(dirname $0) ; pwd)
-TARGET=$1
-
-source aws.sh
-
-# For nightly-YYYY-MM-DD, tag should be nightly
-# For lts-X.Y, tag should be ltsX
-SHORTNAME=$(echo $TARGET | cut -d- -f 1)
-if [ $SHORTNAME = "lts" ]
-then
-    TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@')
-else
-    TAG=$SHORTNAME
-fi
-
-IMAGE=commercialhaskell/stackage:$TAG
-
-PLAN_FILE=current-plan.yaml
-DOCMAP_FILE=current-docmap.yaml
-
-CABAL_DIR=$ROOT/cabal
-STACK_DIR=$ROOT/stack
-GHC_DIR=$ROOT/ghc
-DOT_STACKAGE_DIR=$ROOT/dot-stackage
-WORKDIR=$ROOT/$TAG/work
-EXTRA_BIN_DIR=$ROOT/extra-bin
-SSH_DIR=$ROOT/ssh-$SHORTNAME
-USERID=$(id -u)
-
-mkdir -p \
-	"$CABAL_DIR" \
-	"$STACK_DIR" \
-	"$GHC_DIR" \
-	"$DOT_STACKAGE_DIR" \
-	"$WORKDIR" \
-	"$EXTRA_BIN_DIR" \
-	"$SSH_DIR"
-
-GITCONFIG=$ROOT/gitconfig
-cat >$GITCONFIG <<EOF
-[user]
-	email = michael+stackage-build@fpcomplete.com
-	name = Stackage Build host
-EOF
-
-cat >$SSH_DIR/known_hosts <<EOF
-|1|Qn0iij8BnxGZXbyFSozS9zWkH+Q=|YrKKNp2KHO3/oc4UBFIe1zOvhDc= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
-|1|RxBEt2ljiEppr019szMIhbY12m0=|0FZ2Oji1LphRbPLLEQhFzTmL69I= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
-EOF
-
-AUTH_TOKEN=$ROOT/auth-token
-HACKAGE_CREDS=$ROOT/hackage-creds
-
-function require_400_file {
-    if [ ! -f "$1" ]
-    then
-        echo File not found: "$1"
-        exit 1
-    fi
-
-    chmod 400 "$1"
-}
-
-require_400_file "$SSH_DIR/id_rsa"
-require_400_file "$AUTH_TOKEN"
-require_400_file "$HACKAGE_CREDS"
-
-mkdir -p $ROOT/bin
-BINDIR=$(cd $ROOT/bin ; pwd)
-(
-cd $BINDIR
-rm -f stackage-curator stackage-curator.bz2
-wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
-bunzip2 stackage-curator.bz2
-chmod +x stackage-curator
-./stackage-curator --version
-)
-
-ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
-ARGS_PREBUILD="$ARGS_COMMON -u $USERID -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
-ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
-ARGS_UPLOAD="$ARGS_COMMON -u $USERID -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
-
-# Make sure we actually need this snapshot. We only check this for LTS releases
-# since, for nightlies, we'd like to run builds even if they are unnecessary to
-# get early warning information of upcoming failures. (See the duplicate check
-# below for why this is safe.)
-if [ $SHORTNAME = "lts" ]
-then
-  docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
-fi
-
-# Get latest compatible stack
-curl -L https://github.com/commercialhaskell/stack/releases/download/v1.9.3/stack-1.9.3-linux-x86_64-static.tar.gz | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
-
-
-# Determine the new build plan unless NOPLAN is set
-#
-# * Update the package index (unless LTS)
-# * Create a new plan
-if [ "${NOPLAN:-}x" = "x" ]
-then
-    if [ $SHORTNAME = "lts" ]
-    then
-        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
-    else
-        docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
-    fi
-fi
-
-# Do the rest of the pre-build actions:
-#
-# * Check that the plan is valid
-# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
-# * Do a single unpack to create the package index cache (again due to directory perms)
-docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
-
-case $SHORTNAME in
-    lts) JOBS=1 ;;
-    nightly) JOBS=2 ;;
-esac
-
-# Now do the actual build. We need to first set the owner of the home directory
-# correctly, so we run the command as root, change owner, and then use sudo to
-# switch back to the current user
-docker run $ARGS_BUILD $IMAGE nice -n 15 /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --jobs $JOBS --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET"
-
-# Make sure we actually need this snapshot. We used to perform this check
-# exclusively before building. Now we perform it after as well for the case of
-# nightly, where we don't perform this check beforehand. This is also slightly
-# safer, in case someone else already uploaded a specific snapshot while we
-# were building.
-docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
-
-# Successful build, so we need to:
-#
-# * Upload the docs to S3
-# * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
-# * Upload the new plan .yaml file to the appropriate Github repo
-# * Register as a new Hackage distro
-docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
-
-echo -n "Completed at "
-date
--- a/automated/new-stackage-format/convert.sh
+++ b/automated/new-stackage-format/convert.sh
@ -4,7 +4,7 @@ set -eux

 if [[ ! -f convert ]]
 then
-    curl https://s3.amazonaws.com/www.snoyman.com/convert-old-stackage-f780174e2c84e4fb171f6526228d9243beb7fd71.bz2 > convert.bz2
+    curl https://s3.amazonaws.com/www.snoyman.com/convert-old-stackage-c9c4d06b31cb7aafedd23aa316b8008c45e4d4dd.bz2 > convert.bz2
    chmod +x convert.bz2
    bunzip2 convert.bz2
 fi
@ -21,7 +21,10 @@ do
    fi
 done

+(
+export HOME=$(pwd)
 ./convert
+)

 cd stackage-snapshots
 git add lts nightly
--- a/build-constraints.yaml
+++ b/build-constraints.yaml
@ -9,8 +9,28 @@ cabal-format-version: "2.4"

 # Constraints for brand new builds
 packages:
+    "Tobias Reinhart <tobi.reinhart@fau.de> @TobiReinhart":
+        - sparse-tensor
+
+    "Stephan Schiffels <stephan_schiffels@mac.com> @stschiff":
+        - sequence-formats
+        - pipes-ordered-zip
+        - sequenceTools
+
+    "YongJoon Joe <developer@quietjoon.net> @QuietJoon":
+        - doldol
+        - ENIG
+
+    "Chris Penner <christopher.penner@gmail.com> @ChrisPenner":
+        - lens-regex-pcre
+
+    "Emily Pillmore <emilypi@cohomolo.gy> @topos":
+        - lens-process
+        - microlens-process

    "Matthieu Monsch <mtth@apache.org> @mtth":
+        - flags-applicative
+        - more-containers
        - tracing

    "Robert Vollmert <rob@vllmrt.net> @robx":
@ -24,6 +44,9 @@ packages:
        - interpolatedstring-qq2
        - prospect
        - do-notation
+        - unagi-chan
+        - type-errors
+        - loopbreaker

    "William Yao <williamyaoh@gmail.com> @williamyaoh":
        - string-interpolate
@ -64,6 +87,9 @@ packages:
    "Preetham Gujjula <preetham.gujjula@gmail.com> @pgujjula":
        - modular

+    "Guillaume Bouchard <guillaum.bouchard@gmail.com> @guibou":
+        - PyF
+
    "Erik Schnetter <schnetter@gmail.com> @eschnett":
        - mpi-hs

@ -80,7 +106,7 @@ packages:
        - parsec-numbers

    "Joshua Grosso <jgrosso256@gmail.com> @jgrosso":
-        - axel
+        - axel < 0 # https://github.com/axellang/axel/issues/24

    "Varun Gandhi <theindigamer15@gmail.com> @theindigamer":
        - edit < 0
@ -121,6 +147,11 @@ packages:
        - rhine-gloss
        - dunai-core
        - finite-typelits
+        - essence-of-live-coding
+        - essence-of-live-coding-pulse
+        - essence-of-live-coding-gloss
+        - essence-of-live-coding-quickcheck
+        - pulse-simple

    "Paul Johnson <paul@cogito.org.uk> @PaulJohnson":
        - geodetics
@ -248,7 +279,7 @@ packages:
    "Richard Cook <rcook@rcook.org> @rcook":
        - hidden-char
        - oset
-        - req-url-extra
+        - req-url-extra < 0 # req < 2.1.0
        - sexpr-parser

    "Vanessa McHale <tmchale@wisc.edu> @vmchale":
@ -265,6 +296,7 @@ packages:
        - apportionment
        - audacity
        - bibtex
+        - board-games
        - buffer-pipe
        - calendar-recycling
        - checksum
@ -367,6 +399,7 @@ packages:
        - newtype-generics
        - bsb-http-chunked
        - coercible-utils
+        - hspec-parsec

    "Joe M <joe9mail@gmail.com> @joe9":
        - logger-thread
@ -387,6 +420,7 @@ packages:
        - yeshql

    "Yair Chuchem <yairchu@gmail.com> @yairchu":
+        - generic-constraints
        - List
        - ListTree

@ -415,6 +449,7 @@ packages:
    "James M.C. Haver II <mchaver@gmail.com> @mchaver":
        - quickcheck-arbitrary-adt
        - hspec-golden-aeson
+        - quickcheck-arbitrary-template

    "Winter Han <winterland1989@gmail.com> @winterland1989":
        - if
@ -430,6 +465,7 @@ packages:
        - unboxed-ref

    "Harendra Kumar <harendra.kumar@gmail.com> @harendra-kumar":
+        - bench-show
        - monad-recorder
        - packcheck
        - streamly
@ -494,9 +530,10 @@ packages:
        - persistent-postgresql
        - persistent-sqlite
        - persistent-template
-        # persistent-test # https://github.com/commercialhaskell/stackage/pull/4492
+        - persistent-test # https://github.com/commercialhaskell/stackage/pull/4492
        # - stackage-curator # http-conduit 2.3 via amazonka
        - store
+        - wai-extra
        - wai-websockets
        - warp-tls
        - yesod
@ -536,7 +573,7 @@ packages:
        - hybrid-vectors
        - indents
        - language-c
-        - persistent-mongoDB < 0 # GHC 8.4 via mongoDB
+        - persistent-mongoDB
        - pretty-class
        - th-expand-syns
        - th-lift
@ -567,7 +604,8 @@ packages:
        - githash

        - time-manager
-        - pantry-tmp
+        - pantry
+        - mega-sdist
        - http-download
        - hi-file-parser
        - rio-prettyprint
@ -981,7 +1019,6 @@ packages:
        - dhall-bash
        - dhall-json
        # - dhall-nix # deriving-compat via hnix
-        - dhall-text

    "Andrew Thaddeus Martin <andrew.thaddeus@gmail.com> @andrewthad":
        - colonnade
@ -1098,13 +1135,15 @@ packages:
        - DAV
        - hopenpgp-tools
        - opensource
+        - debian
+        - cabal-debian

    "Piyush P Kurur <ppk@cse.iitk.ac.in> @piyush-kurur":
-        - raaz < 0 # GHC 8.4 via base-4.11.0.0
-        - naqsha < 0 # GHC 8.4 via base-4.11.0.0
+        - raaz
+        - naqsha

    "Joey Hess <id@joeyh.name> @joeyh":
-        # - git-annex # conduit 1.3
+        # - git-annex # Depends on many packages not tracked by Stackage
        - concurrent-output
        - mountpoints
        - disk-free-space
@ -1171,9 +1210,8 @@ packages:
        #
        - hledger-lib
        - hledger
-        - hledger-ui < 0  # via brick, via data-clist, via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - hledger-ui
        - hledger-web
-        - hledger-api
        #
        - quickbench
        - regex-compat-tdfa
@ -1234,6 +1272,7 @@ packages:
        - kind-generics
        - kind-generics-th
        - simplistic-generics
+        - wl-pprint

    "Matvey Aksenov <matvey.aksenov@gmail.com> @supki":
        - terminal-size
@ -1398,10 +1437,10 @@ packages:
        - HaskellNet-SSL < 0 # mime-mail-0.5 via HaskellNet

    "Jens Petersen <juhpetersen@gmail.com> @juhp":
-        - cabal-rpm
+        - cabal-rpm < 0 # https://github.com/commercialhaskell/stackage/issues/4666
+        - dl-fedora
        - fedora-dists
-        - fedora-haskell-tools
-        - fedora-img-dl
+        - fedora-haskell-tools < 0 # https://github.com/commercialhaskell/stackage/issues/4662
        - hkgr
        - http-directory
        - rpmbuild-order
@ -1450,7 +1489,7 @@ packages:
        - pipes-binary
        - pipes-network
        - pipes-network-tls
-        - safe-money < 0
+        - safe-money
        - vector-bytes-instances
        - xmlbf-xeno
        - xmlbf-xmlhtml < 0 # GHC 8.4 via xmlhtml via hspec-2.5.0
@ -1744,7 +1783,7 @@ packages:
        - cubicspline
        - maximal-cliques

-    "Alexander Bondarenko <aenor.realm@gmail.com> @wiz":
+    "Alexander Bondarenko <aenor.realm@gmail.com> @dpwiz":
        - hedn
        - soap
        - soap-tls
@ -1940,6 +1979,7 @@ packages:
        - configurator-export
        - decidable
        - emd
+        - functor-products
        - hamilton
        - hmatrix-backprop
        - hmatrix-vector-sized
@ -2019,6 +2059,7 @@ packages:
        - glabrous
        - google-oauth2-jwt
        - IPv6DB < 0
+        - gothic

    "koral koral@mailoo.org @k0ral":
        - atom-conduit < 0
@ -2156,12 +2197,14 @@ packages:
        - monad-par
        - monad-par-extras
        - mtl-compat
+        - ordered-containers
        - proxied
        - singleton-nats
        - text-show
        - text-show-instances
        - th-abstraction
        - thread-local-storage
+        - type-equality

    "Kirill Zaborsky <qrilka@gmail.com> @qrilka":
        - xlsx
@ -2176,6 +2219,7 @@ packages:
        - esqueleto
        - persistent-typed-db
        - persistent-qq
+        - persistent-pagination

    "Matthew Pickering <matthewtpickering@gmail.com> @mpickering":
        - refact
@ -2254,6 +2298,7 @@ packages:
        - leancheck
        - leancheck-instances
        - fitspec
+        - express
        - speculate
        - extrapolate
        - percent-format
@ -2265,22 +2310,19 @@ packages:
        - irc

    "Dennis Gosnell <cdep.illabout@gmail.com> @cdepillabout":
-        - emailaddress < 0 # opaleye
        - envelope
        - from-sum
        - hailgun
-        - hailgun-simple < 0 # via hailgun
        - natural-transformation
-        # - opaleye-trans # product-profunctors 0.9
        - password
        - password-instances
        - pretty-simple
        - read-env-var
-        - servant-checked-exceptions < 0 # https://github.com/cdepillabout/servant-checked-exceptions/issues/29
-        - servant-checked-exceptions-core < 0 # servant 0.16
-        - servant-rawm < 0 # https://github.com/cdepillabout/servant-rawm/issues/9
+        - servant-checked-exceptions
+        - servant-checked-exceptions-core
+        - servant-rawm
        - servant-static-th
-        - termonad < 0
+        - termonad
        - world-peace
        - xml-html-qq
        - xml-indexed-cursor
@ -2339,7 +2381,9 @@ packages:
        - http-common
        - http-streams
        - locators
-        - unbeliever
+        - core-text
+        - core-data
+        - core-program

    "Sean Hunt <scshunt@csclub.uwaterloo.ca @scshunt":
        - cheapskate
@ -2430,14 +2474,14 @@ packages:
        - identicon
        - lame
        - megaparsec
-        - megaparsec-tests < 0 # via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - megaparsec-tests
        - mmark
        - mmark-cli
        - mmark-ext
        - modern-uri
        - pagination
        - parser-combinators
-        # parser-combinators-tests # https://github.com/commercialhaskell/stackage/issues/4540
+        - parser-combinators-tests
        - path
        - path-io
        - req
@ -2654,9 +2698,9 @@ packages:
        - dawg-ord < 0

    "Amit Levy <amit@amitlevy.com> @alevy":
-        - simple < 0 # GHC 8.4 via simple-templates
-        - simple-templates < 0 # BuildFailureException Process exited with ExitFailure 1: ./Setup build
-        - simple-session < 0 # GHC 8.4 via simple
+        - simple
+        - simple-templates
+        - simple-session
        - postgresql-orm

    "Sergey Astanin <s.astanin@gmail.com> @astanin":
@ -2718,7 +2762,7 @@ packages:

    "Ozgun Ataman ozgun.ataman@soostone.com @ozataman":
        - string-conv
-        - rng-utils
+        - rng-utils < 0 # https://github.com/commercialhaskell/stackage/issues/4686
        - ua-parser
        - hs-GeoIP
        - retry
@ -2737,8 +2781,8 @@ packages:
        - xlsx-tabular

    "Mikhail Glushenkov <mikhail.glushenkov@gmail.com> @23Skidoo":
-        - Cabal
-        - cabal-install
+        - Cabal < 3 # https://github.com/commercialhaskell/stackage/issues/4761
+        - cabal-install < 3
        # - pointful # haskell-src-exts

    "Lennart Kolmodin <kolmodin@gmail.com> @kolmodin":
@ -2853,12 +2897,14 @@ packages:
        - repline
        - picosat < 0
        - aos-signature
-        - bulletproofs
+        # https://github.com/commercialhaskell/stackage/issues/4682
+        - bulletproofs < 1
        - pedersen-commitment
        - merkle-tree
        - oblivious-transfer
-        - pairing
+        - pairing < 0 # #4758
        - libraft
+        - galois-field

    "Daishi Nakajima <nakaji.dayo@gmail.com> @nakaji-dayo":
        - api-field-json-th
@ -2899,8 +2945,8 @@ packages:
        # - eventful-dynamodb # http-conduit 2.3 via amazonka
        - eventful-memory
        - eventful-postgresql < 0 # GHC 8.4 via persistent-postgresql
-        - eventful-sql-common
-        - eventful-sqlite
+        - eventful-sql-common < 0 # persistent 2.10.0 #4551
+        - eventful-sqlite < 0 # persistent 2.10.0 via eventful-sql-common
        - eventful-test-helpers
        - stratosphere
        - sum-type-boilerplate
@ -2975,6 +3021,7 @@ packages:
        - quickcheck-classes
        - arithmoi
        - bitvec
+        - poly

    "Ashley Yakeley <ashley@semantic.org> @AshleyYakeley":
        - countable
@ -3037,7 +3084,7 @@ packages:

    "Moritz Kiefer <moritz.kiefer@purelyfunctional.org> @cocreature":
        - lrucaching
-        - llvm-hs < 0 # https://github.com/commercialhaskell/stackage/issues/4238
+        - llvm-hs
        - llvm-hs-pure

    "Thierry Bourrillon <thierry.bourrillon@fpinsight.com> @tbourrillon":
@ -3101,7 +3148,7 @@ packages:
        - butcher
        - czipwith
        - data-tree-print
-        - brittany < 0 # via yaml-0.9.0 commercialhaskell/stackage#3823
+        - brittany

    "Ryan Mulligan <ryan@ryantm.com> @ryantm":
        - HDBC-mysql
@ -3129,7 +3176,7 @@ packages:
        - clock-extras
        - next-ref < 0
        - threads-extras < 0
-        - tmp-postgres
+        - tmp-postgres < 0.3 # https://github.com/commercialhaskell/stackage/issues/4790
        - pg-transact
        - hspec-pg-transact
        - postgresql-simple-queue
@ -3177,11 +3224,9 @@ packages:
        - salak-yaml
        - salak-toml
        - tensors
-        - yam
-        - yam-datasource
-        - yam-redis
        - menshen
        - crc32c
+        - boots

    "Tom Nielsen <tanielsen@gmail.com> @glutamate":
        - plotlyhs
@ -3199,10 +3244,11 @@ packages:
        # - hip # lens 4.16 via diagrams/chart
        - massiv
        - massiv-io
+        - massiv-test
        - scheduler

    "Hans-Peter Deifel <hpd@hpdeifel.de> @hpdeifel":
-        - hledger-iadd < 0  # via brick, via data-clist, via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - hledger-iadd

    "Roy Levien <royl@aldaron.com> @orome":
        - crypto-enigma
@ -3230,7 +3276,7 @@ packages:
    "Mitsutoshi Aoe <maoe@foldr.in> @maoe":
        - influxdb
        - sensu-run < 0 # GHC 8.4 via base-4.11.0.0
-        - viewprof < 0  # via brick, via data-clist, via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - viewprof

    "Dylan Simon <dylan-stack@dylex.net> @dylex":
        - postgresql-typed
@ -3319,6 +3365,11 @@ packages:
        - possibly
        - enum-text
        - rg
+        - columnar
+        - enum-text-rio
+        - fmt-for-rio
+        - no-value
+        - optparse-enum

    "Elliot Cameron <eacameron@gmail.com> @3noch":
        []
@ -3353,12 +3404,13 @@ packages:
        - pier-core < 0
        - pier < 0
        - haskeline
+        - ghc-source-gen

    "Christof Schramm <christof.schramm@campus.lmu.de>":
        - mnist-idx

    "Naushadh <naushadh@protonmail.com> @naushadh":
-        - persistent-mysql-haskell
+    - persistent-mysql-haskell < 0 # persistent 2.10.0 #4550

    "Moritz Schulte <mtesseract@silverratio.net> @mtesseract":
        - async-refresh
@ -3452,6 +3504,9 @@ packages:
        - tomland
        - typerep-map

+    "Dmitrii Kovanikov <kovanikov@gmail.com> @chshersh":
+        - type-errors-pretty
+
    "Holmusk <tech@holmusk.com> @arbus":
        - elm-street

@ -3469,6 +3524,7 @@ packages:
        - language-docker
        - docker-build-cacher < 0 # GHC 8.4 via turtle
        - mysql-haskell-nem
+        - hadolint

    "Phil Ruffwind <rf@rufflewind.com> @Rufflewind":
        - blas-hs
@ -3519,6 +3575,7 @@ packages:
        - path-text-utf8

    "Type Classes <hello@typeclasses.com> @argumatronic @chris-martin":
+        - aws-cloudfront-signed-cookies
        - stripe-concepts
        - stripe-signature
        - stripe-scotty
@ -3639,6 +3696,7 @@ packages:
    "Guru Devanla <grdvnl@gmail.com> @gdevanla":
        - pptable
        - cassava-records
+        - pandoc-markdown-ghci-filter

    "Lucas David Traverso <lucas6246@gmail.com> @ludat":
        - map-syntax < 0 # GHC 8.4 via base-4.11.0.0
@ -3650,11 +3708,11 @@ packages:
        - transformers-fix

    "Domen Kozar <domen@enlambda.com> @domenkozar":
-        - cachix < 0
+        - cachix < 0 # #4752
        - cachix-api
        - elm2nix
-        - stack2nix
-        - mixpanel-client < 0 # https://github.com/domenkozar/mixpanel-client/issues/3
+        - mixpanel-client < 0
+        - netrc
        - pretty-sop
        - servant-auth
        - servant-auth-server
@ -3703,12 +3761,12 @@ packages:
        - tasty-ant-xml

    "Eugene Smolanka <esmolanka@gmail.com> @esmolanka":
-        - sexp-grammar < 0
-        - invertible-grammar < 0
+        - sexp-grammar
+        - invertible-grammar

    "Maximilian Tagher <feedback.tagher@gmail.com> @MaxGabriel":
        - aeson-iproute
-        - persistent-iproute
+        - persistent-iproute < 0 # persistent-2.10.0 #4553

    "Damian Nadales <damian.nadales@gmail.com> @capitanbatata":
        - hierarchy < 0
@ -3758,6 +3816,7 @@ packages:
        - web3

    "Georg Rudoy <0xd34df00d@gmail.com> @0xd34df00d":
+        - can-i-haz
        - enum-subset-generate

    "Trevis Elser <trevis@silencedpoet.com> @telser":
@ -3848,7 +3907,7 @@ packages:
        - servant-auth-wordpress
        - ca-province-codes

-    "David Baynard <haskell@baynard.me> @dbaynard":
+    "David Baynard <haskell@baynard.dev> @dbaynard":
        - time-qq # see christian-marie/time-qq#3
        - ucam-webauth
        - ucam-webauth-types
@ -3896,6 +3955,22 @@ packages:
    "Elben Shira <elben@shira.im> @elben":
        - pencil

+    "ARATA Mizuki <minorinoki@gmail.com> @minoki":
+        - unboxing-vector
+
+    "Brandon Chinn <brandon@leapyear.io> @brandon-leapyear":
+        - th-test-utils
+
+    "Akshay Mankar <itsakshaymankar@gmail.com> @akshaymankar":
+        - jsonpath
+
+    "James Brock <jamesbrock@gmail.com> @jamesdbrock":
+        - replace-megaparsec
+        - replace-attoparsec
+
+    "Robbie McMichael @robbiemcmichael":
+        - http-client-overrides
+
    "Grandfathered dependencies":
        - Boolean
        - ChasingBottoms < 0 # due to QuickCheck, https://github.com/commercialhaskell/stackage/issues/4444
@ -3944,7 +4019,7 @@ packages:
        - blaze-svg
        - blaze-textual
        - boring
-        - brick < 0  # via data-clist, via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - brick < 0.48 # https://github.com/commercialhaskell/stackage/issues/4789
        - buffer-builder
        - byteable
        - bytestring-builder
@ -3999,7 +4074,7 @@ packages:
        - data-binary-ieee754
        - data-bword
        - data-checked
-        - data-clist < 0 # via QuickCheck https://github.com/commercialhaskell/stackage/issues/4444
+        - data-clist
        - data-default
        - data-default-class
        - data-default-instances-containers
@ -4030,6 +4105,7 @@ packages:
        - easytest
        - ed25519
        - edit-distance
+        - elm-bridge
        - enclosed-exceptions
        - entropy
        - erf
@ -4050,6 +4126,7 @@ packages:
        - fmlist
        - friendly-time
        - functor-classes-compat
+        - functor-combinators
        - generic-arbitrary
        - generics-sop-lens
        - ghc-compact
@ -4295,6 +4372,7 @@ packages:
        - tonatona
        - transformers-base
        - tree-diff
+        - trivial-constraint
        - true-name
        - tuple-th
        - type-fun
@ -4327,7 +4405,6 @@ packages:
        - wai-app-static
        - wai-conduit
        - wai-eventsource
-        - wai-extra
        - wai-handler-launch
        - wai-logger
        - wai-session
@ -4454,7 +4531,6 @@ packages:
        - amazonka-core < 0 # via http-client-0.6.1
        - github < 0 # via http-client-0.6.1
        - hailgun < 0 # via http-client-0.6.1
-        - language-puppet < 0 # via http-client-0.6.1
        - mbug < 0 # via http-client-0.6.1

        - antiope-athena < 0 # via amazonka
@ -4464,7 +4540,6 @@ packages:
        - antiope-s3 < 0 # via amazonka
        - antiope-sns < 0 # via amazonka
        - antiope-sqs < 0 # via amazonka
-        - cachix-api < 0 # via amazonka
        - confcrypt < 0 # via amazonka
        - amazonka-apigateway < 0 # via amazonka-core
        - amazonka-application-autoscaling < 0 # via amazonka-core
@ -4575,10 +4650,11 @@ packages:
        - hsdev < 0.3.3.0
        - network-bsd < 2.8.1.0
        - MissingH < 1.4.2.0
-        - dbus < 1.2.8 # https://github.com/commercialhaskell/stackage/issues/4599
-        - xml-conduit < 1.9 # https://github.com/commercialhaskell/stackage/issues/4570
+        - dbus < 1.2.8
+        - xml-conduit < 1.9
+        - systemd < 2

-        # https://github.com/simonmichael/hledger/issues/983
+        # https://github.com/simonmichael/hledger/pull/987
        - easytest < 0.3

        # https://github.com/commercialhaskell/stackage/issues/4484
@ -4587,13 +4663,6 @@ packages:
        # https://github.com/commercialhaskell/stackage/issues/4519
        - case-insensitive < 1.2.1

-        # https://github.com/commercialhaskell/stackage/issues/4523
-        - doctest < 0.16.1
-
-        # https://github.com/commercialhaskell/stackage/issues/4527
-        - generics-sop <  0.5
-        - sop-core < 0.5
-
        # https://github.com/commercialhaskell/stackage/issues/4529
        - hashable < 1.3

@ -4610,30 +4679,11 @@ packages:
        - machines < 0.7

        # https://github.com/commercialhaskell/stackage/issues/4532
-        - profunctors < 5.4
+        - profunctors < 5.5

        # https://github.com/commercialhaskell/stackage/issues/4541
        - dependent-sum < 0.5

-        # causes build failure in persistent-mysql-haskell https://github.com/commercialhaskell/stackage/issues/4550
-        # causes build failure in eventful-sql-common https://github.com/commercialhaskell/stackage/issues/4551
-        # causes build failure in persistent-test https://github.com/commercialhaskell/stackage/issues/4552
-        # causes build failure in persistent-iproute https://github.com/commercialhaskell/stackage/issues/4553
-        - persistent < 2.10
-        - persistent-mysql < 2.10
-        - persistent-postgresql < 2.10
-        - persistent-sqlite < 2.10
-        - persistent-template < 2.7.0
-        - persistent-typed-db < 0.1.0.0
-
-        # https://github.com/commercialhaskell/stackage/issues/4557
-        - hedgehog < 1.0
-        - registry < 0.1.5.0
-        - hedgehog-fn < 1.0
-        - tasty-hedgehog < 1.0
-        - tomland < 1.0.1.0
-        - nonempty-containers < 0.3.0.0
-
        # https://github.com/commercialhaskell/stackage/issues/4564
        - primitive < 0.7.0.0
        - primitive-extras < 0.8
@ -4653,24 +4703,6 @@ packages:
        # https://github.com/commercialhaskell/stackage/issues/4579
        - quickcheck-classes < 0.6.2.0

-        # https://github.com/commercialhaskell/stackage/issues/4580
-        - http-api-data < 0.4.1
-
-        # https://github.com/commercialhaskell/stackage/issues/4588
-        - systemd < 2
-
-        # https://github.com/commercialhaskell/stackage/issues/4591
-        - servant-elm < 0.6
-
-        # https://github.com/commercialhaskell/stackage/issues/4592
-        - swagger2 < 2.4
-
-        # https://github.com/commercialhaskell/stackage/issues/4594
-        - servant < 0.16.1
-        - servant-server < 0.16.1
-        - servant-cli < 0.1.0.1
-        - singleton-bool < 0.1.5
-
        # https://github.com/commercialhaskell/stackage/issues/4596
        - tls < 1.5.0
        - tls-debug < 0.4.6
@ -4678,11 +4710,12 @@ packages:
        # https://github.com/commercialhaskell/stackage/issues/4604
        - inline-c < 0.8

-        # https://github.com/commercialhaskell/stackage/issues/4613
-        - fixed < 0.3
-
        # https://github.com/commercialhaskell/stackage/issues/4619
        - hw-balancedparens < 0.2.1.0
+        # https://github.com/commercialhaskell/stackage/issues/4676
+        - hw-json < 1.1
+        # https://github.com/commercialhaskell/stackage/issues/4657
+        - hw-rankselect < 0.13.1.0

        # https://github.com/commercialhaskell/stackage/issues/4632
        - microlens < 0.4.11
@ -4694,8 +4727,110 @@ packages:
        # https://github.com/commercialhaskell/stackage/issues/4633
        - hw-packed-vector < 0.1

-        # New http2 not supported by Warp yet
+        # https://github.com/commercialhaskell/stackage/issues/4645
+        - warp < 3.3.0
        - http2 < 2
+
+        # https://github.com/commercialhaskell/stackage/issues/4673
+        - hlint < 2.2
+
+        # https://github.com/commercialhaskell/stackage/issues/4693
+        - optparse-applicative < 0.15
+
+        # https://github.com/commercialhaskell/stackage/issues/4702
+        - generic-lens < 1.2.0.0
+
+        # https://github.com/commercialhaskell/stackage/issues/4709
+        - base-unicode-symbols < 0.2.4
+
+        # https://github.com/commercialhaskell/stackage/issues/4725
+        - lens-family < 2
+        - lens-family-core < 2
+
+        # https://github.com/commercialhaskell/stackage/issues/4731
+        - ghc-lib-parser < 8.8.0.20190723
+        - ghc-lib        < 8.8.0.20190723
+
+        # https://github.com/commercialhaskell/stackage/issues/4737
+        - parser-combinators       < 1.2.0
+        - parser-combinators-tests < 1.2.0
+
+        # https://github.com/commercialhaskell/stackage/issues/4739
+        - doctemplates < 0.3
+
+        # https://github.com/commercialhaskell/stackage/issues/4743
+        - monoidal-containers < 0.5
+
+        # https://github.com/commercialhaskell/stackage/issues/4749
+        - call-stack < 0.2.0
+
+        # https://github.com/commercialhaskell/stackage/issues/4750
+        - gi-gdk < 4
+        - gi-gtk < 4
+
+        # https://github.com/commercialhaskell/stackage/issues/4753
+        - dependent-map < 0.3
+        - dependent-sum-template < 0.1
+
+        # https://github.com/commercialhaskell/stackage/issues/4775
+        - polyparse < 1.13
+
+        # https://github.com/commercialhaskell/stackage/issues/4781
+        - generic-deriving < 1.13
+
+        # See https://github.com/commercialhaskell/stackage/issues/4646#issuecomment-525517773
+        - singletons < 2.6
+        - eliminators < 0.6
+        - th-desugar < 1.10
+
+        # https://github.com/commercialhaskell/stackage/issues/4785
+        - ghc-exactprint < 0.6.2
+
+        # https://github.com/commercialhaskell/stackage/issues/4786
+        - shelly < 1.9
+
+        # https://github.com/commercialhaskell/stackage/issues/4796
+        - jailbreak-cabal < 1.3.5
+        - cabal2spec < 2.3
+
+        # https://github.com/commercialhaskell/stackage/issues/4798
+        - ansi-terminal < 0.10
+
+        # https://github.com/commercialhaskell/stackage/issues/4799
+        - memory < 0.15
+
+        # https://github.com/commercialhaskell/stackage/issues/4801
+        - base-compat < 0.11
+
+        # https://github.com/commercialhaskell/stackage/issues/4802
+        - base-compat-batteries < 0.11
+
+        # https://github.com/commercialhaskell/stackage/issues/4803
+        - hspec-wai < 0.10.0
+        - hspec-wai-json < 0.10.0
+
+        # https://github.com/commercialhaskell/stackage/issues/4804
+        - lens < 4.18
+
+        # https://github.com/commercialhaskell/stackage/issues/4805
+        - lens-aeson < 1.1
+
+        # https://github.com/commercialhaskell/stackage/issues/4807
+        - aeson < 1.4.5.0
+
+        # https://github.com/commercialhaskell/stackage/issues/4809
+        - elm-street < 0.1
+
+        # https://github.com/commercialhaskell/stackage/issues/4816
+        - trifecta < 2.1
+
+        # https://github.com/commercialhaskell/stackage/issues/4823
+        - universe < 1.2
+        - universe-dependent-sum < 1.2
+
+        # https://github.com/commercialhaskell/stackage/issues/4824
+        - bindings-GLFW < 3.3
+
 # end of packages

 # Package flags are applied to individual packages, and override the values of
@ -4789,10 +4924,6 @@ package-flags:
        # https://github.com/haskell/cabal/issues/4883
        native-dns: false

-    # https://github.com/commercialhaskell/stackage/issues/3666
-    safe-money:
-        serialise: false
-
    # https://github.com/fpco/stackage/issues/3619
    transformers-compat:
        five-three: true
@ -4877,6 +5008,7 @@ skipped-tests:
    - ip # hspec 2.5 https://github.com/andrewthad/haskell-ip/issues/33
    - language-ecmascript # testing-feat 1.1.0.0
    - lattices # via tasty-1.2
+    - netrc # via tasty-1.2
    - makefile # GHC 8.2
    - mmtf # via hspec-2.7.0
    - next-ref # hspec 2.3
@ -5002,6 +5134,8 @@ skipped-tests:
    - uri-bytestring # Could not deduce (SOP.All (SOP.All Arbitrary) xs) arising from a use of ‘SOP.hcpure’
    - cron # Could not deduce (SOP.All (SOP.All Arbitrary) xss) arising from a use of ‘SOP.hcpure’
    - config-ini # https://github.com/aisamanra/config-ini/issues/22
+    - polysemy-plugin # https://github.com/commercialhaskell/stackage/issues/4733
+

    # Runtime issues
    - blank-canvas # Never finishes https://github.com/ku-fpg/blank-canvas/issues/73
@ -5090,8 +5224,16 @@ skipped-tests:
    # Requires running pg-harness-server
    - peregrin

-    # Internal errors with move to the new curator tool?
-    - blaze-collonade # https://github.com/commercialhaskell/stackage/issues/4639
+    # Test errors with modules not being visible
+    - servant-ruby # https://github.com/commercialhaskell/stackage/issues/4650
+    - pretty-simple # https://github.com/commercialhaskell/stackage/issues/4652
+    - password-instances # https://github.com/commercialhaskell/stackage/issues/4653
+    - password # https://github.com/commercialhaskell/stackage/issues/4654
+    - haskell-gi # https://github.com/commercialhaskell/stackage/issues/4661
+    - envelope # https://github.com/commercialhaskell/stackage/issues/4669
+    - algebraic-graphs # https://github.com/commercialhaskell/stackage/issues/4670
+    - either-both # https://github.com/commercialhaskell/stackage/issues/4757
+    - bugsnag-haskell # https://github.com/commercialhaskell/stackage/issues/4759
 # end of skipped-tests

 # Tests listed in expected-test-failures configure correctly but may fail to run
@ -5203,7 +5345,7 @@ expected-test-failures:
    - odbc # "Need ODBC_TEST_CONNECTION_STRING environment variable"
    - opaleye # PostgreSQL
    - pandoc-pyplot # requires DISPLAY for tcltk
-    - pantry-tmp # https://github.com/commercialhaskell/stackage/issues/4628
+    - pantry # https://github.com/commercialhaskell/stackage/issues/4628
    - persistent-redis # redis - https://github.com/fpco/stackage/pull/1581
    - pipes-mongodb
    - postgresql-query # PostgreSQL
@ -5242,7 +5384,6 @@ expected-test-failures:
    - doctest-discover # 0.1.0.9 https://github.com/karun012/doctest-discover/issues/22
    - graylog # 0.1.0.1 https://github.com/fpco/stackage/pull/1254
    - tomland # https://github.com/kowainik/tomland/issues/141
-    - safe-json # 0.1.0 https://github.com/Vlix/safe-json/issues/1

    # Assertion failures, these can be real bugs or just limitations
    # in the test cases.
@ -5350,6 +5491,44 @@ expected-test-failures:

    # https://github.com/jgm/pandoc/issues/5582
    - pandoc
+
+    # https://github.com/commercialhaskell/stackage/issues/4642
+    - ghc-exactprint
+
+    # https://github.com/commercialhaskell/stackage/issues/4643
+    - world-peace
+
+    # https://github.com/commercialhaskell/stackage/issues/4646
+    - singletons
+
+    # Cannot reproduce locally, looks like it may be a bug in Stack or curator
+    - shake-language-c
+
+    # https://github.com/commercialhaskell/stackage/issues/4706
+    - dhall
+
+    # https://github.com/commercialhaskell/stackage/issues/4707
+    - blaze-colonnade
+
+    # https://github.com/commercialhaskell/stackage/issues/4722
+    - tasty-discover
+
+    # https://github.com/commercialhaskell/stackage/issues/4764
+    - persistent-mysql
+    # https://github.com/commercialhaskell/stackage/issues/4763
+    - persistent-postgresql
+
+    # https://github.com/commercialhaskell/stackage/issues/4784
+    - raaz
+
+    # Requires a running server
+    - persistent-mongoDB
+
+    # https://github.com/commercialhaskell/stackage/issues/4810
+    - dns
+
+    # https://github.com/commercialhaskell/stackage/issues/4815
+    - brittany
 # end of expected-test-failures

 # Benchmarks which are known not to build. Note that, currently we do not run
@ -5366,6 +5545,7 @@ expected-benchmark-failures:
    - cryptohash # https://github.com/vincenthz/hs-cryptohash/pull/43
    - http2
    - lz4 # https://github.com/fpco/stackage/issues/3510
+    - raaz # https://github.com/commercialhaskell/stackage/issues/4766
    - thyme
    - xmlgen # https://github.com/skogsbaer/xmlgen/issues/6

@ -5569,6 +5749,7 @@ skipped-benchmarks:
    - extensible # via freer-effects https://github.com/fumieval/extensible/issues/12
    - hw-bits # https://github.com/haskell-works/hw-bits/issues/8
    - hslua # https://github.com/commercialhaskell/stackage/issues/4526
+    - persistent-template # https://github.com/commercialhaskell/stackage/issues/4760

    # Cyclic dependencies
    - cassava
@ -5847,7 +6028,7 @@ no-revisions:
 - gauge
 - stack
 - http-download
- pantry-tmp
+- pantry
 - rio-prettyprint
 - hi-file-parser
 # https://github.com/commercialhaskell/stackage/issues/3706:
@ -5855,7 +6036,6 @@ no-revisions:
 - hledger-lib
 - hledger-ui
 - hledger-web
- hledger-api


 # Do not build these packages in parallel with others. Useful for high memory
--- a/12
+++ b/12
@ -1,6 +1,14 @@
-#!/bin/sh
+#!/bin/bash

 # Convenience script for checking constraints locally

 cd `dirname $0`
-exec stack exec --resolver ghc-8.6.5 stackage-curator check
+
+export GHCVER=$(sed -n "s/^ghc-version: \"\(.*\)\"/\1/p" "build-constraints.yaml")
+
+NIGHTLY="nightly-$(date +%Y-%m-%d)"
+curator update &&
+  curator constraints --target=$NIGHTLY &&
+  curator snapshot-incomplete --target=$NIGHTLY &&
+  curator snapshot &&
+  stack --resolver ghc-$GHCVER exec curator check-snapshot
--- a/debian-bootstrap.sh
+++ b/debian-bootstrap.sh
@ -98,6 +98,7 @@ apt-get install -y \
    libopenmpi-dev \
    libpango1.0-dev \
    libpcap0.8-dev \
+    libpcre2-dev \
    libpq-dev \
    libprotobuf-dev \
    libre2-dev \
@ -131,6 +132,7 @@ apt-get install -y \
    libzstd-dev \
    libzmq3-dev \
    llvm-6.0 \
+    llvm-8 \
    locales \
    m4 \
    minisat \
@ -163,13 +165,13 @@ curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources
 apt-get update
 ACCEPT_EULA=Y apt-get install msodbcsql17 -y

-locale-gen en_US.UTF-8
+# llvm for llvm-hs
+curl https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
+add-apt-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main"
+apt-get update
+apt-get install llvm-9-dev -y

-# llvm-7.0 for llvm-hs (separate since it needs wget)
-wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
-    && add-apt-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-7 main" \
-    && apt-get update \
-    && apt-get install -y llvm-7
+locale-gen en_US.UTF-8

 # Buggy versions of ld.bfd fail to link some Haskell packages:
 # https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
@ -217,10 +219,9 @@ echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf
    && ldconfig

 # Install erlang/otp platform and its dependencies
-ERLANG_VERSION="20.2.2"
-ERLANG_DEB_FILE="esl-erlang_21.2-1~ubuntu~bionic_amd64.deb"
+ERLANG_DEB_FILE="esl-erlang_21.1-1~ubuntu~bionic_amd64.deb"
 pushd /tmp \
-    && wget http://packages.erlang-solutions.com/site/esl/esl-erlang/FLAVOUR_1_general/${ERLANG_DEB_FILE} \
+    && wget https://packages.erlang-solutions.com/erlang/debian/pool/${ERLANG_DEB_FILE} \
    && (dpkg -i ${ERLANG_DEB_FILE}; apt-get install -yf) \
    && rm ${ERLANG_DEB_FILE} \
    && popd
@ -275,6 +276,12 @@ echo /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server > /etc/ld.so.conf.d/
 echo /usr/lib/llvm-3.7/lib > /etc/ld.so.conf.d/llvm.conf

 ldconfig
+
+# Install librdkafka (Apache Kafka C/C++ library)
+wget -qO - https://packages.confluent.io/deb/5.2/archive.key | apt-key add -
+add-apt-repository "deb https://packages.confluent.io/deb/5.2 stable main"
+apt-get update && apt install -y librdkafka-dev
+
 # EOF: don't build anything below this line

 # Cleanup
--- a/etc/curator-exes/.gitignore
+++ b/etc/curator-exes/.gitignore
@ -0,0 +1 @@
+/bin/
--- a/etc/curator-exes/Dockerfile
+++ b/etc/curator-exes/Dockerfile
@ -0,0 +1,13 @@
+FROM fpco/stack-build:lts-14
+
+# Prime the snapshot so rebuilds are faster
+RUN mkdir -p /artifacts
+RUN stack update
+RUN stack build --no-install-ghc --resolver lts-13.28 --only-dependencies stack amazonka-s3
+
+RUN git clone --depth 1 https://github.com/commercialhaskell/stack && \
+    git clone --depth 1 https://github.com/commercialhaskell/curator
+
+COPY stack.yaml .
+
+RUN stack --no-install-ghc install --local-bin-path /artifacts
--- a/etc/curator-exes/README.md
+++ b/etc/curator-exes/README.md
@ -0,0 +1,5 @@
+Scripts to create and upload the curator-exes.tar.bz2 files. Make sure to
+update the hash in automated/build-next.sh.
+
+NOTE: Depending on what's exactly on the master branches when running the
+`./run.sh` script, the `stack.yaml` file may need some tweaks.
--- a/etc/curator-exes/run.sh
+++ b/etc/curator-exes/run.sh
@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+set -eux
+
+rm -rf bin
+mkdir -p bin
+
+docker build . --tag curator-exes
+
+docker run --rm -v $(pwd)/bin:/output curator-exes cp /artifacts/* output
+aws s3 cp curators-exes/* s3://download.fpcomplete.com/curator-exes/
--- a/etc/curator-exes/stack.yaml
+++ b/etc/curator-exes/stack.yaml
@ -0,0 +1,22 @@
+resolver: lts-13.28
+
+packages:
+- stack
+- curator
+
+extra-deps:
+- hi-file-parser-0.1.0.0@sha256:3fc95852035c8e0fa6d5f9d22e5108067c8f4816ac84675bddc971b20144a9ae,2317
+- http-download-0.1.0.0@sha256:b02d76d28364b74fb90d3ec3c46fd81f9d77bcf71fa40c1d341a7251a0fc40e8,2023
+- pantry-0.1.1.2@sha256:6f06556f8e934afd517559d6f3dae947db91ef3801ee4e5cee9916c83a0c4d52,4628
+- rio-prettyprint-0.1.0.0@sha256:764e1a357275c3c62d57c494d038feae01cf745788e7d0e97367ba793c9670c3,1359
+- ansi-terminal-0.9.1@sha256:48f53532d0f365ffa568c8cf0adc84c66f800a7d80d3329e4f04fa75392f4af1,3225
+- rio-0.1.12.0@sha256:a0b42682a455ffde6a4d894b135117e379aea7ed1810c2cfed471b8f5879968c,3633
+- unliftio-0.2.12@sha256:b089fbc2ff2628a963c2c4b12143f2020874e3e5144ffd6c62b25639a0ca1483,3325
+
+- github: fpco/casa
+  commit: a0cd05c1a9ce33acbb1bdfb9324370f884916e88
+  # sha256: 812e05de8f9abdbd1101fc94895dcfd5c35962c4d67a43cc20cbc06eb59ea1f0
+  # size: 9508
+  subdirs:
+     - casa-client
+     - casa-types