Update from parent

CURATORS.md

@@ -1,7 +1,7 @@
 This is a collection of instructions covering the processes that the Stackage curators - the
 guys who maintain the Stackage project itself - should be doing on a regular basis.
 Originally this was handled largely by Michael Snoyman,
-but now we are a team of 4 people handling requests weekly in rotation.
+but now we are a team of 5 people handling requests weekly in rotation.
 Curation activities are mostly automated, and do not take up a significant amount of time.
 
 ## Workflow overview
@@ -223,6 +223,26 @@ rm -r nightly/work/builds/nightly/
 ```
 This should also be done when moving the Nightly docker image to a new version of Ubuntu.
 
+If you're impatient and would like to build the Docker image on the
+build server instead of waiting for Docker Hub, you can run the
+following command:
+
+```
+DIR=$(mktemp -d)
+(cd $DIR \
+  && git clone https://github.com/fpco/stackage \
+  && cd stackage \
+  && docker build --tag snoyberg/stackage:nightly .)
+rm -rf $DIR
+```
+
+Note that we do a clean clone of the `stackage` repo instead of using
+the existing checkout because of how `docker build` works: it will
+send the entire local directory contents as context to the Docker
+daemon, which in the case of the build tree is a _lot_ of content. (We
+can discuss the wisdom—or lack thereof—of Docker's
+approach separately.)
+
 ## stackage-build server
 
 You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
@@ -275,8 +295,9 @@ develop this advice over time. For now: if you're not sure, ask for guidance.
 __`NOPLAN=1`__ If you wish to rerun a build without recalculating a
 build plan, you can set the environment variable `NOPLAN=1`. This is
 useful for such cases as an intermittent test failure, out of memory
-condition, or manually tweaking the plan file. This is the default for
-LTS builds.
+condition, or manually tweaking the plan file.
+
+Note LTS builds inherit the current Hackage data (stack updated for Nigthly) to avoid excess extra rebuilding.
 
 ### Timing
 

MAINTAINERS.md

@@ -2,7 +2,7 @@ This project is built around the concept of maintainers taking responsibility fo
 
 The idea behind Stackage is that, if all packages work with the newest versions of dependencies, we avoid dependency hell. Specifically, we aim for:
 
-* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
+* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
 * All packages are compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
 * All packages in a snapshot are compatible with the versions of libraries that ship with the GHC used in the snapshot ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
 
@@ -21,6 +21,8 @@ To add a set of packages, you would add:
         - package2
         - package3
 
+Note that the `master` branch is used for Stackage Nightly (not the `nightly` branch, which is used for the nightly docker builder imagine).
+
 After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
 
 If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
@@ -103,10 +105,11 @@ following the above steps, you can get your package into the next major LTS
 Haskell release.
 
 If you would like to get your package added to an existing LTS Haskell major
-release (e.g., if `lts-3.21` is out, you would want your package to appear in
-`lts-3.22`), please do the following in addition to the steps above:
+release (e.g., if `lts-8.9` is out, you would want your package to appear in
+`lts-8.10`), please do the following in addition to the steps above:
 
 * Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
-* Specify the LTS major version you would like your package to go into (e.g., lts-3)
+* Specify the LTS major version you would like your package to go into (e.g., lts-8)
 * Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
-* Be patient! The LTS releases are by their nature more conservative than nightly, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue quickly, but there may be some delay.
+* Be patient! The LTS releases are less frequent than Nightly and by their nature more conservative, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue as soon as possible, but it may take some time.
+* To add a package to more than one LTS version please file separate tickets for each major LTS release, since they will typically be built and added at different times.

README.md

@@ -17,7 +17,7 @@ Add your package
 We welcome all packages, provided:
 
 * The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
-* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
+* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](https://docs.haskellstack.org/en/stable/travis_ci/), which ensures a package is not accidentally incomplete.
 * The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
 * The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
 
@@ -123,3 +123,7 @@ That said, we do maintain the capability to keep multiple LTS runs
 operational in parallel, and with LTS 6 and 7 in fact did so. We
 aren't changing our guarantees yet on longevity of a release, but are
 trying to push out the bounds a bit farther.
+
+__What time are Stackage snapshots published?__
+
+Stackage Nightly and LTS are not released at a fixed time of day, they get pushed to stackage.org (and the metadata to the stackage-nightly and stackage-lts github repos) when their builds finish on the Stackage build server and the latest built haddocks have been synced over. This time varies greatly depending on build times for package updates, bounds breakage, problems with new packages being added and other build issues, etc. There are days when a release does not happen. LTS releases tend to happen over the weekend or early in the week.

automated/build.sh

@@ -138,4 +138,5 @@ docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-
 # * Register as a new Hackage distro
 docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
 
+echo -n "Completed at "
 date

become-a-curator.md

@@ -0,0 +1,44 @@
+## We are no longer accepting applications, please wait until next time
+
+We are looking to expand the stackage curator team!
+
+As a stackage curator you will help the community by curating and
+publishing the widely used nightly and LTS snapshots: Collections of
+packages that are guaranteed to play well together. All curators are
+volunteers.
+
+This is a great opportunity to contribute to the haskell community and get
+familiar with the tools of the trade! You will also be in regular
+contact with experienced community members.
+
+We don't have any prerequisites, but familiarity with any of Haskell,
+Cabal, Stack, Linux, Git, and Docker is a plus.
+
+We work in shifts of one week, which means you will be on duty every
+6th week. Our suggestion is to pair you up with another curator to
+help out in the beginning.
+
+The workload is usually about 30 minutes per day, whenever you have
+time. The work consists of keeping the builds running on the build
+server, filing issues for packages needing updates when new releases
+occur, and closing resolved issues. The LTS snapshots are prepared
+once a week. All communication with package maintainers happen on
+github.
+
+We also have a curator slack channel where we help each other out.
+
+The current curator team consists of:
+* Adam Bergmark
+* Dan Burton
+* Jens Petersen
+* Luke Murphy
+* Michael Snoyman
+
+We onboarded Luke as the newest member a few months ago and this
+helped us iron out and document the process further.
+
+You can read the curator documentation here: https://github.com/fpco/stackage/blob/master/CURATORS.md
+
+To apply, please fill in this form: [REDACTED]
+
+If you applied last time, we encourage you to do so again!

debian-bootstrap.sh

@@ -20,10 +20,11 @@ apt-get install -y software-properties-common
 
 add-apt-repository ppa:hvr/ghc -y
 add-apt-repository -y ppa:marutter/rrutter
-# not sure what this was needed for
-#add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse
+apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
+add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy main'
+add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-apache24-compat main'
+add-apt-repository -y --keyserver hkp://keyserver.ubuntu.com:80 'deb http://download.mono-project.com/repo/debian wheezy-libjpeg62-compat main'
 
-# Set the GHC version
 GHCVER=8.0.2
 
 apt-get update
@@ -37,6 +38,7 @@ apt-get install -y \
     sudo \
     curl \
     freeglut3-dev \
+    fsharp \
     git \
     gradle \
     libadns1-dev \
@@ -69,6 +71,7 @@ apt-get install -y \
     libgtksourceview-3.0-dev \
     libhidapi-dev \
     libicu-dev \
+    libimlib2-dev \
     libjudy-dev \
     liblapack-dev \
     libleveldb-dev \
@@ -78,6 +81,8 @@ apt-get install -y \
     libmagickcore-dev \
     libmagickwand-dev \
     libmarkdown2-dev \
+    libmono-2.0-dev \
+    libmp3lame-dev \
     libmpfr-dev \
     libmysqlclient-dev \
     libncurses-dev \
@@ -94,6 +99,7 @@ apt-get install -y \
     libsdl2-ttf-dev \
     libsnappy-dev \
     libsndfile1-dev \
+    libsox-dev \
     libsqlite3-dev \
     libssl-dev \
     libsystemd-dev \
@@ -112,10 +118,16 @@ apt-get install -y \
     llvm-3.7 \
     locales \
     m4 \
+    minisat \
+    mono-mcs \
     nettle-dev \
     nodejs \
     npm \
     openjdk-8-jdk \
+    python-mpltoolkits.basemap \
+    python3-matplotlib \
+    python3-numpy \
+    python3-pip \
     r-base \
     r-base-dev \
     ruby-dev \
@@ -172,3 +184,26 @@ cd /tmp \
 # Add JDK to system paths.
 echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
     && ldconfig
+
+# llvm-4.0 for llvm-hs (separate since it needs wget)
+wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \
+    && add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main" \
+    && apt-get update \
+    && apt-get install -y llvm-4.0
+
+# Install version 3 of the protobuf compiler.  (The `protobuf-compiler` package only
+# supports version 2.)
+curl -OL https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip \
+  && sudo unzip -o protoc-3.3.0-linux-x86_64.zip -d /usr bin/protoc \
+  && rm -f protoc-3.3.0-linux-x84_64.zip
+
+# Install the TensorFlow C API.
+curl https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.1.0.tar.gz > libtensorflow.tar.gz \
+    && sudo tar zxf libtensorflow.tar.gz -C /usr \
+    && rm libtensorflow.tar.gz \
+    && ldconfig
+
+## non-free repo for mediabus-fdk-aac
+#apt-add-repository multiverse \
+#    && apt-get update \
+#    && apt-get install -y libfdk-aac-dev