mirror of
https://github.com/commercialhaskell/stackage.git
synced 2026-01-11 23:08:30 +01:00
Gregory Schwartz
GitHub
commit
b8ce12aaa4
1
.gitignore
vendored
1
.gitignore
vendored
@ -3,3 +3,4 @@
|
||||
nightly-*.yaml
|
||||
lts-*.yaml
|
||||
*.swp
|
||||
check-plan.yaml
|
||||
|
||||
45
.travis.yml
45
.travis.yml
@ -1,21 +1,40 @@
|
||||
env:
|
||||
- CABALVER=1.22 GHCVER=7.10.1
|
||||
# Use new container infrastructure to enable caching
|
||||
sudo: false
|
||||
|
||||
# Note: the distinction between `before_install` and `install` is not important.
|
||||
before_install:
|
||||
- travis_retry sudo add-apt-repository -y ppa:hvr/ghc
|
||||
- travis_retry sudo apt-get update
|
||||
- travis_retry sudo apt-get install cabal-install-$CABALVER ghc-$GHCVER # see note about happy/alex
|
||||
- export PATH=/opt/ghc/$GHCVER/bin:/opt/cabal/$CABALVER/bin:$PATH
|
||||
# Choose a lightweight base image; we provide our own build tools.
|
||||
language: generic
|
||||
|
||||
# GHC depends on GMP. You can add other dependencies here as well.
|
||||
addons:
|
||||
apt:
|
||||
packages:
|
||||
- libgmp-dev
|
||||
|
||||
env:
|
||||
- GHCVER=8.0.1
|
||||
|
||||
install:
|
||||
- cabal --version
|
||||
- echo "$(ghc --version) [$(ghc --print-project-git-commit-id 2> /dev/null || echo '?')]"
|
||||
- travis_retry cabal update
|
||||
# Download and unpack the stack executable
|
||||
- mkdir -p ~/.local/bin
|
||||
- export PATH=$HOME/.local/bin:$PATH
|
||||
- travis_retry curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'
|
||||
|
||||
# Get stackage-curator
|
||||
- wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
- bunzip2 stackage-curator.bz2
|
||||
- chmod +x stackage-curator
|
||||
- mv stackage-curator ~/.local/bin
|
||||
|
||||
# Install GHC and cabal-install
|
||||
- stack setup $GHCVER
|
||||
|
||||
# Update the index
|
||||
- travis_retry stack update
|
||||
|
||||
# Here starts the actual work to be performed for the package under test; any command which exits with a non-zero exit code causes the build to fail.
|
||||
script:
|
||||
- ./stackage-curator check
|
||||
- stack --resolver ghc-$GHCVER exec stackage-curator check
|
||||
|
||||
cache:
|
||||
directories:
|
||||
- $HOME/.stack
|
||||
- $HOME/.stackage/curator/cache
|
||||
|
||||
339
CURATORS.md
Normal file
339
CURATORS.md
Normal file
@ -0,0 +1,339 @@
|
||||
This is a collection of instructions covering the processes that the Stackage curators - the
|
||||
guys who maintain the Stackage project itself - should be doing on a regular basis.
|
||||
Originally this was handled largely by Michael Snoyman,
|
||||
but now we are a team of 4 people handling requests weekly in rotation.
|
||||
Curation activities are mostly automated, and do not take up a significant amount of time.
|
||||
|
||||
## Workflow overview
|
||||
|
||||
This section sketches out at a high level how the entire Stackage build/curation
|
||||
process works:
|
||||
|
||||
* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
|
||||
* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
|
||||
* stackage-curator can check that build plan to ensure all version bounds are consistent
|
||||
* The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
|
||||
* Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
|
||||
* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh)
|
||||
* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
|
||||
* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
|
||||
* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
|
||||
|
||||
## Pull requests
|
||||
|
||||
The typical story on pull requests is: If Travis accepts it and the
|
||||
author only added packages under his/her own name, merge it. If the
|
||||
build later fails (see "Adding Debian packages for required system tools or libraries"),
|
||||
then block the package until it's fixed.
|
||||
|
||||
If benchmarks, haddocks, or test suites fails at this point we
|
||||
typically also block the package until these issues are fixed. This in
|
||||
order to add packages with a clean slate.
|
||||
|
||||
Optionally we can check if packdeps says the package is up to date.
|
||||
Visit http://packdeps.haskellers.com/feed?needle=<package-name>
|
||||
|
||||
Builds may fail because of unrelated bounds changes. If this happens,
|
||||
first add any version bounds to get master into a passing state (see
|
||||
"Fixing bounds issues"), then re-run the travis build.
|
||||
|
||||
A common issue is that authors submit newly uploaded packages, it can
|
||||
take up to an hour before this has synced across the stack
|
||||
infrastructure. You can usually compare the versions of the package in
|
||||
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
|
||||
to what's on hackage to see if this is the case. Wait an hour and
|
||||
re-run the pull request.
|
||||
|
||||
Tests also commonly fail due to missing test files, and sometimes due
|
||||
to doctest limitations. You can point the maintainer to
|
||||
https://github.com/bergmark/blog/blob/master/2016/package-faq.md
|
||||
|
||||
## Fixing bounds issues
|
||||
|
||||
The most common activity you'll deal with in Stackage curation is a version
|
||||
bound issue, usually a restrictive upper bound. You fix this by opening an
|
||||
issue on the Stackage repo about the problem, and modifying the
|
||||
build-constraints.yaml file to work around it in one of the ways below. Be sure
|
||||
to refer to the issue for workarounds added to that file.
|
||||
|
||||
### Temporary upper bounds
|
||||
|
||||
Most common technique, just prevent a new version of a library from
|
||||
being included immediately. This also applies to when only benchmarks
|
||||
and tests are affected.
|
||||
|
||||
* Copy the stackage-curator output and create a new issue, see e.g
|
||||
https://github.com/fpco/stackage/issues/2108
|
||||
|
||||
* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
|
||||
|
||||
```yaml
|
||||
"Stackage upper bounds":
|
||||
# https://github.com/fpco/stackage/issues/2108
|
||||
- pipes < 4.3.0
|
||||
```
|
||||
|
||||
* Commit (message e.g. "Upper bound for #2108")
|
||||
* Optionally: Verify with `stackage-curator check` locally
|
||||
* Push
|
||||
* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
|
||||
|
||||
Sometimes releases for different packages are tightly coupled. Then it
|
||||
can make sense to combine them into one issue, as in
|
||||
https://github.com/fpco/stackage/issues/2143.
|
||||
|
||||
If a dependency that is not explicitly in stackage is causing test or
|
||||
benchmark failures you can skip or expect them to fail (see "Skipping
|
||||
tests and benchmarks" and "Expecting test/benchmark/haddock
|
||||
failures"). Bonus points for reporting this upstream to that packages'
|
||||
maintainer.
|
||||
|
||||
### Lifting upper bounds
|
||||
|
||||
You can try this when you notice that a package has been updated. You
|
||||
can also periodically try to lift bounds (I think it's good to do this
|
||||
at the start of your week /@bergmark)
|
||||
|
||||
If not all packages have been updated check if any of them are missing
|
||||
from the original issue and if so add a new comment mentioning them. A
|
||||
new package may appear if its dependencies were part of this issue but
|
||||
have been updated since the last time we checked. We want to give
|
||||
these new packages ample time to be upgraded.
|
||||
|
||||
If stackage-curator is happy commit the change ("Remove upper bounds
|
||||
and close #X"). After doing this the next nightly build may fail
|
||||
because some packages didn't have an upper bound in place, but
|
||||
compilation failed. In this case revert the previous commit so any
|
||||
disabled packages are enabled again, re-open the issue, and add a new
|
||||
comment with the failing packages. This is to give all maintainers
|
||||
enough time to upgrade for this case as well.
|
||||
|
||||
### Amending upper bounds
|
||||
|
||||
With the `pipes` example above there was later a new release of
|
||||
`pipes-safe` that required the **newer** version of `pipes`. You can
|
||||
add that package to the same upper bounds section,
|
||||
(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
|
||||
as we want to lift them simultaneously.
|
||||
|
||||
### Skipping tests and benchmarks
|
||||
|
||||
Sometimes tests and benchmark dependencies are forgotten or not cared
|
||||
for. To disable compilation for them add them to `skipped-tests` or
|
||||
`skipped-benchmarks`. If a package is added to these sections they
|
||||
won't be compiled, and their dependencies won't be taken into account.
|
||||
|
||||
There are sub sections under these headers that is used to group types
|
||||
of failures together, and also to document what type of failures
|
||||
exist.
|
||||
|
||||
### Expecting test/benchmark/haddock failures
|
||||
|
||||
The difference from the `skipped` sections is that items listed here
|
||||
are compiled and their dependencies are taken into account. These
|
||||
sections also have sub sections with groups and descriptions.
|
||||
|
||||
One big category of test suites in this section are those requiring
|
||||
running services. We don't want to run those, but we do want to check
|
||||
dependencies and compile them.
|
||||
|
||||
If there are no version bounds that would fix the issue or if you
|
||||
can't figure it out, file it
|
||||
(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
|
||||
maintainer for help.
|
||||
|
||||
### Waiting for new releases
|
||||
|
||||
Sometimes there is a failure reported on a (now possibly closed) issue
|
||||
on an external tracker. If an issue gets resolved but there is no
|
||||
hackage release yet we'd like to get notified when it's uploaded.
|
||||
|
||||
Add the package with its current version to the
|
||||
`tell-me-when-its-released` section. This will cause the build to stop
|
||||
when the new version is out.
|
||||
|
||||
### Excluding packages
|
||||
|
||||
In an extreme case of a non-responsive maintainer, you can remove the
|
||||
package entirely from Stackage. We try to avoid that whenever
|
||||
possible.
|
||||
|
||||
This typically happens when we move to a new major GHC release or when
|
||||
there are only a few packages waiting for updates on an upper bounds
|
||||
issue.
|
||||
|
||||
Comment out the offending packages from the "packages" section and add
|
||||
a comment saying why it was disabled:
|
||||
|
||||
```
|
||||
# - swagger # bounds: aeson 1.0
|
||||
```
|
||||
|
||||
If this causes reverse dependencies to be disabled we should notify
|
||||
the maintainers of those packages.
|
||||
|
||||
|
||||
## Updating the content of the Docker image used for building
|
||||
|
||||
### Adding Debian packages for required system tools or libraries
|
||||
Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
|
||||
Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
|
||||
since new packages are not added to the current lts release.
|
||||
|
||||
Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
|
||||
|
||||
Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
|
||||
It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
|
||||
|
||||
If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
|
||||
|
||||
### Upgrading GHC version
|
||||
The Dockerfile contains information on which GHC versions should be used. You
|
||||
can modify it and push it to Github to trigger a DockerHub build. The nightly
|
||||
branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the
|
||||
major version number (e.g., lts3 for lts-3.\*).
|
||||
|
||||
Note that when starting a new LTS major release, you'll need to modify Docker
|
||||
Hub to create a new Docker tag for the relevant branch name.
|
||||
|
||||
### Getting the new image to the build server
|
||||
Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
|
||||
below). Instead of pulling an unbounded number of images, I typically just
|
||||
delete all of the old images and let the new ones get downloaded:
|
||||
|
||||
```
|
||||
docker rm $(docker ps -a -q)
|
||||
docker rmi $(docker images -q)
|
||||
```
|
||||
|
||||
but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
|
||||
|
||||
For a new GHC version you should also delete the cache directories on the stackage-build server to
|
||||
force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg:
|
||||
```
|
||||
rm -r nightly/work/builds/nightly/
|
||||
```
|
||||
This should also be done when moving the Nightly docker image to a new version of Ubuntu.
|
||||
|
||||
## stackage-build server
|
||||
|
||||
You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
|
||||
|
||||
```
|
||||
Host stackage-build
|
||||
User curators
|
||||
Hostname build.stackage.org
|
||||
```
|
||||
|
||||
### Running the build script
|
||||
|
||||
We currently run the builds manually so make it easy to see when there are
|
||||
bounds issues that need to be corrected. Automated this would be even better,
|
||||
we're just not there yet.
|
||||
|
||||
```
|
||||
# Run a nightly build
|
||||
/opt/stackage-build/stackage/automated/run-nightly.sh
|
||||
|
||||
# Run an LTS minor bump
|
||||
/opt/stackage-build/stackage/automated/build.sh lts-2.17
|
||||
|
||||
# Run an LTS major bump
|
||||
/opt/stackage-build/stackage/automated/build.sh lts-3.0
|
||||
```
|
||||
|
||||
Recommended: run these from inside a `tmux` session. If you get version bound
|
||||
problems on nightly or LTS major, you need to fix build-constraints.yaml (see
|
||||
info above). For an LTS minor bump, you'll typically want to use the
|
||||
`CONSTRAINTS` environment variable, e.g.:
|
||||
|
||||
```
|
||||
CONSTRAINTS='--constraint "conduit < 1.4.5" --constraint "criterion < 1.2.3"' /opt/stackage-build/stackage/automated/build.sh lts-2.17
|
||||
```
|
||||
|
||||
Valid arguments to include in this environment variable:
|
||||
|
||||
* `--constraint` to modify an upper or lower bound
|
||||
* `--add-package` to add a brand new package
|
||||
* `--expect-test-failure` to expect tests to fail
|
||||
* `--expect-haddock-failure` to expect haddocks to fail
|
||||
|
||||
If a build fails for bounds reasons, see all of the advice above. If the code
|
||||
itself doesn't build, or tests fail, open up an issue and then either put in a
|
||||
version bound to avoid that version or something else. It's difficult to give
|
||||
universal advice on how to solve things, since each situation is unique. Let's
|
||||
develop this advice over time. For now: if you're not sure, ask for guidance.
|
||||
|
||||
__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
|
||||
build plan, you can set the environment variable `NOPLAN=1`. This is
|
||||
useful for such cases as an intermittent test failure, out of memory
|
||||
condition, or manually tweaking the plan file. This is the default for
|
||||
LTS builds.
|
||||
|
||||
### Timing
|
||||
|
||||
A looping script on the build server keeps trying to build nightly
|
||||
with `sleep 30m` interleaved. It only publishes the nightly once per
|
||||
day. This way new package versions or build failures can be caught
|
||||
early and hopefully the nightlies will be timely.
|
||||
|
||||
LTS minor bumps typically are run on Sundays.
|
||||
|
||||
### Website sync debugging (and other out of disk space errors)
|
||||
|
||||
* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
|
||||
* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
|
||||
* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
|
||||
|
||||
### Wiping the cache
|
||||
|
||||
Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
|
||||
You can wipe the nightly cache and rebuild everything by doing
|
||||
`rm -rf /var/stackage/stackage/automated/nightly`.
|
||||
Replace nightly with `lts7` to wipe the LTS 7 cache.
|
||||
|
||||
### Force a single package rebuild
|
||||
|
||||
You can force a single package to rebuild by deleting its "previous result"
|
||||
file, e.g.:
|
||||
|
||||
```
|
||||
$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
|
||||
```
|
||||
|
||||
## Local curator setup
|
||||
|
||||
We do not run the full stackage build locally as that might take too
|
||||
much time. However, some steps on the other hand are much faster to do
|
||||
yourself, e.g. verifying constraints without building anything.
|
||||
|
||||
To get started, install `stackage-curator` via Git, or [the Linux binary]:
|
||||
|
||||
```
|
||||
$ git clone git@github.com:fpco/stackage-curator.git
|
||||
$ cd stackage-curator && stack install
|
||||
```
|
||||
|
||||
It is a good idea to upgrade `stackage-curator` at the start of your week.
|
||||
Then, clone the stackage repo, get the latest packages and run dependency
|
||||
resolution:
|
||||
|
||||
```
|
||||
$ git clone git@github.com:fpco/stackage.git
|
||||
$ stack update && stackage-curator check
|
||||
```
|
||||
|
||||
This can be used to make sure all version bounds are in place, including for
|
||||
test suites and benchmarks, to check whether bounds can be lifted, and to get
|
||||
[tell-me-when-its-released] notifications.
|
||||
|
||||
`stackage-curator` does not build anything, so you wont see any compilation
|
||||
errors for builds, tests and benchmarks.
|
||||
|
||||
[the Linux binary]: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
[tell-me-when-its-released]: https://github.com/fpco/stackage/blob/master/CURATORS.md#waiting-for-new-releases
|
||||
|
||||
## Adding new curators
|
||||
|
||||
1. Add public ssh key to `~/.ssh/authorized_keys` on build server
|
||||
2. Add to fpco/stackage project.
|
||||
129
DATA-FLOW.md
Normal file
129
DATA-FLOW.md
Normal file
@ -0,0 +1,129 @@
|
||||
# The Stackage data flow
|
||||
|
||||
The Stackage project is really built on top of a number of different
|
||||
subcomponents. This page covers how they fit together. The Stackage data flow
|
||||
diagram gives a good bird's-eye view:
|
||||
|
||||
|
||||
|
||||
## Inputs
|
||||
|
||||
There are three inputs into the data flow:
|
||||
|
||||
* [Hackage](http://hackage.haskell.org/) is the upstream repository of all
|
||||
available open source Haskell packages that are part of our ecosystem.
|
||||
Hackage provides both cabal file metadata (via the 00-index.tar file) and
|
||||
tarballs of the individual packages.
|
||||
|
||||
* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
|
||||
is the primary Stackage input file. This is where package maintainers can add
|
||||
packages to the Stackage package set. This also defines upper bounds, skipped
|
||||
tests, and a few other pieces of metadata.
|
||||
|
||||
* [stackage-content](https://github.com/fpco/stackage-content) is a Github
|
||||
repository containing static file content served from stackage.org
|
||||
|
||||
## Travis
|
||||
|
||||
For [various
|
||||
reasons](https://www.fpcomplete.com/blog/2015/05/distributing-packages-without-sysadmin),
|
||||
we leverage Travis CI for running some processes. In particular:
|
||||
|
||||
* [all-cabal-files](https://github.com/commercialhaskell/all-cabal-files/blob/hackage/.travis.yml)
|
||||
clones all cabal files from Hackage's 00-index.tar file into a Git repository
|
||||
without any modification
|
||||
|
||||
* [all-cabal-hashes](https://github.com/commercialhaskell/all-cabal-hashes/blob/hackage/.travis.yml)
|
||||
is mostly the same, but also includes cryptographic hashes of the package
|
||||
tarballs for more secure download (as leveraged by
|
||||
[Stack](http://haskellstack.com). It is powered by [all-cabal-hashes-tool](https://github.com/commercialhaskell/all-cabal-hashes-tool)
|
||||
|
||||
* [all-cabal-packages](https://github.com/commercialhaskell/all-cabal-packages)
|
||||
uses [hackage-mirror](http://github.com/fpco/hackage-mirror) to populate the
|
||||
hackage.fpcomplete.com mirror of Hackage, which provides S3-backed high
|
||||
availability hosting of all package tarballs
|
||||
|
||||
* [all-cabal-metadata](https://github.com/commercialhaskell/all-cabal-metadata)
|
||||
uses
|
||||
[all-cabal-metadata-tool](https://github.com/commercialhaskell/all-cabal-metadata-tool)
|
||||
to query extra metadata from Hackage about packages and put them into YAML
|
||||
files. As we'll see later, this avoids the need to make a lot of costly calls
|
||||
to Hackage APIs
|
||||
|
||||
Travis does not currently provide a means of running jobs on a regular basis.
|
||||
Therefore, we have a simple cron job on the Stackage build server that triggers
|
||||
each of the above builds every 30 minutes.
|
||||
|
||||
## stackage-curator
|
||||
|
||||
The heart of running Stackage builds is the
|
||||
[stackage-curator](https://github.com/fpco/stackage-curator) tool. We run this
|
||||
on a daily basis on the Stackage build server for Stackage Nightly, and on a
|
||||
weekly basis for LTS Haskell. The build process is [highly
|
||||
automated](https://github.com/fpco/stackage/blob/master/automated/build.sh) and
|
||||
leverages Docker quite a bit.
|
||||
|
||||
stackage-curator needs to know about the most recent versions of all packages,
|
||||
their tarball contents, and some metadata, all of which it gets from the
|
||||
Travis-generated sources mentioned in the previous section. In addition, it
|
||||
needs to know about build constraints, which can come from one of two places:
|
||||
|
||||
* When doing an LTS Haskell minor version bump (e.g., building lts-5.13), it
|
||||
grabs the previous version (e.g., lts-5.12) and converts the previous package
|
||||
set into constraints. For example, if lts-5.12 contains the package foo-5.6.7,
|
||||
this will be converted into the constraint `foo >= 5.6.7 && < 5.7`.
|
||||
* When doing a Stackage Nightly build or LTS Haskell major version bump (e.g.,
|
||||
building lts-6.0), it grabs the latest version of the build-constraints.yaml
|
||||
file.
|
||||
|
||||
By combining these constraints with the current package data, stackage-curator
|
||||
can generate a build plan and check it. (As an aside, this build plan
|
||||
generation and checking also occurs every time you make a pull request to the
|
||||
stackage repo.) If there are version bounds problems, one of the [Stackage
|
||||
curators](https://github.com/fpco/stackage/blob/master/CURATORS.md) will open
|
||||
up a Github issue and will add upper bounds, temporarily block a package, or
|
||||
some other corrective action.
|
||||
|
||||
Once a valid build plan is found, stackage-curator will build all packages,
|
||||
build docs, and run test suites. Assuming that all succeeds, it generates some
|
||||
artifacts:
|
||||
|
||||
* Uploads the build plan as a YAML file to either
|
||||
[stackage-nightly](https://github.com/fpco/stackage-nightly) or
|
||||
[lts-haskell](https://github.com/fpco/lts-haskell)
|
||||
* Uploads the generated Haddock docs and a package index (containing all used
|
||||
.cabal files) to haddock.stackage.org.
|
||||
|
||||
## stackage-server-cron
|
||||
|
||||
On the Stackage build server, we run the [stackage-server-cron
|
||||
executable](https://github.com/fpco/stackage-server/blob/master/app/stackage-server-cron.hs)
|
||||
regularly, which generates:
|
||||
|
||||
* A [SQLite
|
||||
database](https://github.com/fpco/stackage-server/blob/master/Stackage/Database.hs)
|
||||
containing information on snapshots, the packages they contain, Hackage
|
||||
metadata about packages, and a bit more. This database is uploaded to S3.
|
||||
* A Hoogle database for each snapshot, which is also uploaded to S3
|
||||
|
||||
## stackage-server
|
||||
|
||||
The [software running stackage.org](https://github.com/fpco/stackage-server) is
|
||||
a relatively simple Yesod web application. It pulls data from the
|
||||
stackage-content repo, the SQLite database, the Hoogle databases, and the build
|
||||
plans for Stackage Nightly and LTS Haskell. It doesn't generate anything
|
||||
important of its own except for a user interface.
|
||||
|
||||
## Stack
|
||||
|
||||
[Stack](http://haskellstack.com) takes advantage of many of the pieces listed above as well:
|
||||
|
||||
* It by default uses the all-cabal-hashes repo for getting package metadata,
|
||||
and downloads package contents from the hackage.fpcomplete.com mirror (using
|
||||
the hashes in the repo for verification)
|
||||
* There are some metadata files in stackage-content which contain information
|
||||
on, for example, where to download GHC tarballs from to make `stack setup`
|
||||
work
|
||||
* Stack downloads the raw build plans for Stackage Nightly and LTS Haskell from
|
||||
the Github repo and uses them when deciding which packages to build for a
|
||||
given stack.yaml file
|
||||
21
Dockerfile
21
Dockerfile
@ -1,23 +1,8 @@
|
||||
FROM ubuntu:14.04
|
||||
FROM fpco/pid1:16.04
|
||||
|
||||
ENV HOME /home/stackage
|
||||
ENV LANG en_US.UTF-8
|
||||
|
||||
RUN mkdir /home/stackage -p
|
||||
RUN locale-gen en_US.UTF-8
|
||||
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get update
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y software-properties-common python-software-properties git
|
||||
RUN DEBIAN_FRONTEND=noninteractive add-apt-repository ppa:hvr/ghc -y
|
||||
ENV PATH /opt/ghc/8.0.1/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh
|
||||
RUN DEBIAN_FRONTEND=noninteractive bash /tmp/debian-bootstrap.sh
|
||||
RUN rm /tmp/debian-bootstrap.sh
|
||||
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cabal-install-1.22 ghc-7.10.1 ghc-7.10.1-htmldocs alex-3.1.3 happy-1.19.4 sudo
|
||||
|
||||
ENV PATH /home/stackage/.cabal/bin:/usr/local/sbin:/usr/local/bin:/opt/ghc/7.10.1/bin:/opt/cabal/1.22/bin:/opt/alex/3.1.3/bin:/opt/happy/1.19.4/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
RUN cabal update
|
||||
RUN cabal install hscolour cabal-install && cp $HOME/.cabal/bin/* /usr/local/bin && rm -rf $HOME/.cabal $HOME/.ghc /tmp/stackage
|
||||
RUN wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2 && bunzip2 stackage-curator.bz2 && chmod +x stackage-curator && mv stackage-curator /usr/local/bin
|
||||
RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh
|
||||
|
||||
96
MAINTAINERS.md
Normal file
96
MAINTAINERS.md
Normal file
@ -0,0 +1,96 @@
|
||||
This project is built around the concept of maintainers taking responsibility for making their packages work with the rest of the stable ecosystem, usually meaning the newest version of all dependencies. This is a social contract, and is not reflected in the codebase in any way.
|
||||
|
||||
The idea behind Stackage is that, if all packages work with the newest versions of dependencies, we avoid dependency hell. Specifically, we aim for:
|
||||
|
||||
* All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
|
||||
* All packages are compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
|
||||
* All packages in a snapshot are compatible with the versions of libraries that ship with the GHC used in the snapshot ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
|
||||
|
||||
## Adding a package
|
||||
|
||||
Anyone can add any package to Stackage but you may only add packages under your own name. It's highly encouraged that the actual package maintainer is also the Stackage maintainer, if that is not the case you should drop the package maintainer a note first.
|
||||
|
||||
To add your package, first fork this repository.
|
||||
In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`.
|
||||
To add a set of packages, you would add:
|
||||
|
||||
"My Name myemail@example.com @mygithubuser":
|
||||
- package1
|
||||
- package2
|
||||
- package3
|
||||
|
||||
After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly
|
||||
|
||||
If you want to make sure that the package builds against the newest versions of all dependecies you can do this:
|
||||
```
|
||||
$ cabal update
|
||||
$ ghc --version # Should give v8.0.1
|
||||
$ cabal get PACKAGE-VERSION # e.g. aeson-0.11.2.1
|
||||
$ cd PACKAGE-VERSION
|
||||
$ cabal sandbox init # Should give "Creating a new sandbox" and not "Using an existing sandbox".
|
||||
$ cabal install --enable-tests --enable-benchmarks --dry-run | grep latest # Should give no results
|
||||
$ cabal install --enable-tests --enable-benchmarks --allow-newer
|
||||
$ cabal test
|
||||
```
|
||||
|
||||
**NB** Please use commit messages like "add foo-bar" or "add johndev's packages"
|
||||
(`build-constraints.yaml` is the most frequently changed file in this git repo
|
||||
so commit messages like "update build-constraints.yaml" are not helpful).
|
||||
|
||||
**NB2** There can be a delay of up to an hour before package versions
|
||||
newly uploaded to Hackage appear to our build server. If you just
|
||||
uploaded a package to Hackage that you're trying to get included, we
|
||||
recommend waiting an hour before opening the PR. You can verify this
|
||||
by making sure the latest version is listed at
|
||||
https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/.
|
||||
|
||||
## Uploading a new package
|
||||
|
||||
When a new version of a package is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and raise GitHub issue tickets to resolve the issue.
|
||||
|
||||
If the new version doesn't compile then the package author should quickly (within 1 week) upload a fixed version.
|
||||
|
||||
If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite.
|
||||
|
||||
|
||||
## Following dependency upgrades
|
||||
|
||||
If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is highly recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds.
|
||||
|
||||
**If restrictive version bounds are the only problem** then you must quickly (within 1 week) upload a new version with relaxed version bounds. Note that unlike the PVP, Stackage does not require upper bounds.
|
||||
|
||||
**If the new dependency causes breaking changes** then all package authors should quickly assess the likely impact on their package (within 1 week) and then produce a new compatible version. The expected timeline for new versions varies between 1 week and 1 month, depending on the significance of the change, and thus the work required to produce those new versions.
|
||||
|
||||
|
||||
## Failing to meet the time limits
|
||||
|
||||
Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack.
|
||||
|
||||
The time limits are intended to stop people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for long.
|
||||
|
||||
## Upgrading to a new GHC version
|
||||
|
||||
The Stackage curation team tries to move Stackage Nightly to new versions of GHC quickly as they become available, while keeping LTS Haskell on a regular release schedule. For package maintainers, the most important impacts of a new GHC release are:
|
||||
|
||||
* We will typically do a sweep through the Stackage upper bounds and aggressively remove packages that block them. This is because, in most cases, we will need to move to the newest versions of a package to get support for the latest GHC, and asking package maintainers to backport their fixes is an undue burden
|
||||
* We will definitely do this at a GHC major version release, and may do so at a minor version release
|
||||
* Packages that are incompatible with the newest GHC version will be temporarily blocked
|
||||
|
||||
If your package ends up being temporarily removed from Stackage Nightly, please simply send a pull request to add it back once it and its dependencies are compatible with the newest GHC version.
|
||||
|
||||
## Adding a package to an LTS snapshot
|
||||
|
||||
The steps above affect the Stackage Nightly builds, but do not directly affect
|
||||
LTS Haskell builds. When we build a new LTS Haskell major version (anything
|
||||
ending in `.0`), the package set is taken from Stackage Nightly. Therefore, by
|
||||
following the above steps, you can get your package into the next major LTS
|
||||
Haskell release.
|
||||
|
||||
If you would like to get your package added to an existing LTS Haskell major
|
||||
release (e.g., if `lts-3.21` is out, you would want your package to appear in
|
||||
`lts-3.22`), please do the following in addition to the steps above:
|
||||
|
||||
* Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new)
|
||||
* Specify the LTS major version you would like your package to go into (e.g., lts-3)
|
||||
* Provide a list of packages you would like added, and if relevant, any upper bounds on those packages
|
||||
* Be patient! The LTS releases are by their nature more conservative than nightly, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue quickly, but there may be some delay.
|
||||
113
README.md
113
README.md
@ -3,11 +3,33 @@ stackage
|
||||
|
||||
[](https://travis-ci.org/fpco/stackage)
|
||||
|
||||
"Stable Hackage," tools for creating a vetted set of packages from Hackage.
|
||||
"Stable Hackage": creating a vetted set of packages from Hackage.
|
||||
This repository is for package authors and maintainers to get their packages into Stackage.
|
||||
If you simply want to use Stackage as an end user, please follow the instructions on [https://www.stackage.org/](https://www.stackage.org).
|
||||
|
||||
__NOTE__ This repository is for package authors to get their code into
|
||||
Stackage. If you simply want to use Stackage as an end user, please follow the
|
||||
instructions on [http://www.stackage.org/](http://www.stackage.org).
|
||||
We strongly recommend using the Haskell tool stack for doing builds, which
|
||||
includes built-in Stackage support: [stack](https://github.com/commercialhaskell/stack) [](https://travis-ci.org/commercialhaskell/stack).
|
||||
|
||||
|
||||
Add your package
|
||||
----------------
|
||||
|
||||
We welcome all packages, provided:
|
||||
|
||||
* The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md).
|
||||
* The package is buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete.
|
||||
* The package is compatible with the newest versions of all dependencies (You can find restrictive upper bounds by visiting http://packdeps.haskellers.com/feed?needle=PACKAGENAME).
|
||||
* The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
|
||||
|
||||
Full details on how to add and test a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package).
|
||||
|
||||
__NOTE__: There is an approximate 30 minute delay between a package uploading
|
||||
to Hackage and being available to the Travis build script to check upper
|
||||
bounds. If a pull request is marked as failed due to using an older version,
|
||||
please close and reopen the PR to retrigger a Travis build.
|
||||
|
||||
Other repos
|
||||
-----------
|
||||
|
||||
The Stackage project consists of multiple repositories. This repository
|
||||
contains the metadata on packages to be included in future builds and some
|
||||
@ -18,87 +40,30 @@ project information. In addition, we have the following repositories:
|
||||
* [stackage-types](https://github.com/fpco/stackage-types) [](https://travis-ci.org/fpco/stackage-types)
|
||||
* [lts-haskell](https://github.com/fpco/lts-haskell)
|
||||
* [stackage-nightly](https://github.com/fpco/stackage-nightly)
|
||||
|
||||
We also support some add-on tools to cabal-install to make its usage with
|
||||
Stackage both easier and more secure:
|
||||
|
||||
* [stackage-cli](https://github.com/fpco/stackage-cli) [](https://travis-ci.org/fpco/stackage-cli)
|
||||
* [stackage-update](https://github.com/fpco/stackage-update) [](https://travis-ci.org/fpco/stackage-update)
|
||||
* [stackage-upload](https://github.com/fpco/stackage-upload) [](https://travis-ci.org/fpco/stackage-upload)
|
||||
* [stackage-install](https://github.com/fpco/stackage-install) [](https://travis-ci.org/fpco/stackage-install)
|
||||
* [stackage-build-plan](https://github.com/fpco/stackage-build-plan) [](https://travis-ci.org/fpco/stackage-build-plan)
|
||||
|
||||
Get your package included
|
||||
-------------------------
|
||||
Curious how it all fits together? See the [Stackage data
|
||||
flow](https://github.com/fpco/stackage/blob/master/DATA-FLOW.md).
|
||||
|
||||
In order to get your package included in the set of stable packages, you should
|
||||
send a pull request against this repository. In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file,
|
||||
there's a section called `packages`. In general, to add a set of
|
||||
packages, you would add:
|
||||
|
||||
"My Name myemail@example.com @mygithubuser":
|
||||
- package1
|
||||
- package2
|
||||
- package3
|
||||
|
||||
You can follow the examples of the other sets of packages in that function.
|
||||
Once you've done this, you can send a pull request to get your package
|
||||
included.
|
||||
|
||||
__NOTE__: In order to ease the process of adding new packages, we no longer
|
||||
require new submissions to be tested on your own system before sending a pull
|
||||
request. If you believe your package works with the newest versions of all
|
||||
dependencies, you may send a pull request without testing first.
|
||||
|
||||
You should also read the [maintainers
|
||||
agreement](https://github.com/fpco/stackage/wiki/Maintainers-Agreement).
|
||||
|
||||
Package Author Guidelines
|
||||
-------------------------
|
||||
|
||||
There are some basic rules to get your package to play nice with Stackage. Here
|
||||
are some quick guidelines to hopefully make this easier:
|
||||
|
||||
* Make sure that your code is buildability and testable from Hackage. Often
|
||||
times, authors test their builds locally, but the tarball that gets uploaded
|
||||
to Hackage is missing some necessary files. The best way to do this is to
|
||||
set up a Travis job to do it for you. We recommend the
|
||||
[multi-ghc-travis](https://github.com/hvr/multi-ghc-travis) approach.
|
||||
* Make your code compatible with the newest versions of all dependencies.
|
||||
* Make your code compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)).
|
||||
|
||||
There are certainly many other tips that could be added here. If you think of
|
||||
any, please send a pull request!
|
||||
|
||||
Build the package set
|
||||
---------------------
|
||||
|
||||
Generally, building the package set should be done only by the Jenkins machine
|
||||
or by the official maintainers, as the process does require quite a bit of
|
||||
setup on the local machine. That said, you'll likely be able to get a stable
|
||||
build by running:
|
||||
|
||||
cabal update
|
||||
cabal install stackage
|
||||
stackage nightly
|
||||
|
||||
### Docker
|
||||
|
||||
Note: This method has been disabled for now, but may be enabled again in the future.
|
||||
|
||||
If you'd like to check a build plan, or perform an entire build, without
|
||||
specially configuring your system, Docker may be a good approach. To check if
|
||||
some modifications to `build-constraints.yaml` are valid, try the following:
|
||||
|
||||
1. Create a local clone of the `stackage` repo
|
||||
2. Make modifications to your local `build-constraints.yaml`
|
||||
3. Inside the `stackage` working directory, run the following:
|
||||
|
||||
```
|
||||
$ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage check'
|
||||
```
|
||||
|
||||
Similarly, if you'd like to perform an entire build, you can replace the last step with:
|
||||
|
||||
```
|
||||
$ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage nightly --skip-upload'
|
||||
```
|
||||
Generally only the stackage build server run by the stackage curator
|
||||
team and people intrested in incorporating stackage snapshots into an
|
||||
OS distribution need to build the entire package set. If you're
|
||||
interested in trying this yourself, please check out
|
||||
[the curator guide](https://github.com/fpco/stackage/blob/master/CURATORS.md),
|
||||
though be aware that this is not a recommended practice and there
|
||||
likely will be problems you will need to debug yourself.
|
||||
|
||||
## Processing
|
||||
|
||||
|
||||
2
automated/.gitignore
vendored
2
automated/.gitignore
vendored
@ -8,3 +8,5 @@
|
||||
/lts-*/
|
||||
/stackage-curator/
|
||||
/stackage-update/
|
||||
/aws.sh
|
||||
/crondir/
|
||||
|
||||
@ -1,34 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -eux
|
||||
set -eu +x
|
||||
|
||||
ROOT=$(cd $(dirname $0) ; pwd)
|
||||
TARGET=$1
|
||||
TAG=$(echo $TARGET | cut -d- -f 1)
|
||||
IMAGE=snoyberg/stackage:$TAG
|
||||
|
||||
if [ "$TAG" = "nightly" ]
|
||||
source aws.sh
|
||||
|
||||
# For nightly-YYYY-MM-DD, tag should be nightly
|
||||
# For lts-X.Y, tag should be ltsX
|
||||
SHORTNAME=$(echo $TARGET | cut -d- -f 1)
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
TROOT=$ROOT/nightly
|
||||
TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@')
|
||||
else
|
||||
TROOT=$ROOT/$(echo $TARGET | cut -d. -f 1)
|
||||
TAG=$SHORTNAME
|
||||
fi
|
||||
|
||||
IMAGE=snoyberg/stackage:$TAG
|
||||
|
||||
PLAN_FILE=current-plan.yaml
|
||||
DOCMAP_FILE=current-docmap.yaml
|
||||
BUNDLE_FILE=current.bundle
|
||||
|
||||
CABAL_DIR=$ROOT/cabal
|
||||
STACK_DIR=$ROOT/stack
|
||||
GHC_DIR=$ROOT/ghc
|
||||
DOT_STACKAGE_DIR=$ROOT/dot-stackage
|
||||
WORKDIR=$TROOT/work
|
||||
SSH_DIR=$ROOT/ssh-$(echo $TARGET | cut -d- -f 1)
|
||||
WORKDIR=$ROOT/$TAG/work
|
||||
EXTRA_BIN_DIR=$ROOT/extra-bin
|
||||
SSH_DIR=$ROOT/ssh-$SHORTNAME
|
||||
|
||||
mkdir -p \
|
||||
"$CABAL_DIR" \
|
||||
"$STACK_DIR" \
|
||||
"$GHC_DIR" \
|
||||
"$DOT_STACKAGE_DIR" \
|
||||
"$WORKDIR" \
|
||||
"$EXTRA_BIN_DIR" \
|
||||
"$SSH_DIR"
|
||||
|
||||
GITCONFIG=$ROOT/gitconfig
|
||||
@ -68,15 +77,65 @@ rm -f stackage-curator stackage-curator.bz2
|
||||
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
bunzip2 stackage-curator.bz2
|
||||
chmod +x stackage-curator
|
||||
./stackage-curator --version
|
||||
)
|
||||
|
||||
ARGS_COMMON="--rm -u $USER -v $WORKDIR:/home/stackage/work -w /home/stackage/work -v $BINDIR/stackage-curator:/usr/local/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro"
|
||||
ARGS_PREBUILD="$ARGS_COMMON -v $CABAL_DIR:/home/stackage/.cabal -v $GHC_DIR:/home/stackage/.ghc -v $DOT_STACKAGE_DIR:/home/stackage/.stackage"
|
||||
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:/home/stackage/.cabal:ro -v $GHC_DIR:/home/stackage/.ghc:ro"
|
||||
ARGS_UPLOAD="$ARGS_COMMON -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY -e AWS_SECRET_KEY=$AWS_SECRET_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:/home/stackage/.stackage -v $SSH_DIR:/home/ubuntu/.ssh:ro -v $GITCONFIG:/home/stackage/.gitconfig:ro -v $CABAL_DIR:/home/stackage/.cabal:ro"
|
||||
ARGS_COMMON="--rm -v $WORKDIR:$HOME/work -w $HOME/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR/stack:/usr/bin/stack:ro"
|
||||
ARGS_PREBUILD="$ARGS_COMMON -u $USER -e HOME=$HOME -v $CABAL_DIR:$HOME/.cabal -v $STACK_DIR:$HOME/.stack -v $GHC_DIR:$HOME/.ghc -v $DOT_STACKAGE_DIR:$HOME/.stackage"
|
||||
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro -v $GHC_DIR:$HOME/.ghc:ro"
|
||||
ARGS_UPLOAD="$ARGS_COMMON -u $USER -e HOME=$HOME -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v $AUTH_TOKEN:/auth-token:ro -v $HACKAGE_CREDS:/hackage-creds:ro -v $DOT_STACKAGE_DIR:$HOME/.stackage -v $SSH_DIR:$HOME/.ssh:ro -v $GITCONFIG:$HOME/.gitconfig:ro -v $CABAL_DIR:$HOME/.cabal:ro -v $STACK_DIR:$HOME/.stack:ro"
|
||||
|
||||
# Use cabal update first to initialize ~/.cabal.config, then use stackage-curator update to get it securely
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator check-target-available --target $TARGET"
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "cabal update && stackage-curator update && stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-} && stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cabal install random cabal-install"
|
||||
docker run $ARGS_BUILD $IMAGE stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
|
||||
# Make sure we actually need this snapshot. We only check this for LTS releases
|
||||
# since, for nightlies, we'd like to run builds even if they are unnecessary to
|
||||
# get early warning information of upcoming failures. (See the duplicate check
|
||||
# below for why this is safe.)
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
|
||||
fi
|
||||
|
||||
# Get latest stack
|
||||
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
|
||||
|
||||
# Determine the new build plan unless NOPLAN is set
|
||||
#
|
||||
# * Update the package index (unless LTS)
|
||||
# * Create a new plan
|
||||
if [ "${NOPLAN:-}x" = "x" ]
|
||||
then
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
|
||||
else
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stack update && exec stackage-curator create-plan --plan-file $PLAN_FILE --target $TARGET ${CONSTRAINTS:-}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Do the rest of the pre-build actions:
|
||||
#
|
||||
# * Check that the plan is valid
|
||||
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
|
||||
# * Do a single unpack to create the package index cache (again due to directory perms)
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "stackage-curator check --plan-file $PLAN_FILE && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && exec stack unpack random"
|
||||
|
||||
# Now do the actual build. We need to first set the owner of the home directory
|
||||
# correctly, so we run the command as root, change owner, and then use sudo to
|
||||
# switch back to the current user
|
||||
docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER $HOME && exec sudo -E -u $USER env \"HOME=$HOME\" \"PATH=\$PATH\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET"
|
||||
|
||||
# Make sure we actually need this snapshot. We used to perform this check
|
||||
# exclusively before building. Now we perform it after as well for the case of
|
||||
# nightly, where we don't perform this check beforehand. This is also slightly
|
||||
# safer, in case someone else already uploaded a specific snapshot while we
|
||||
# were building.
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "exec stackage-curator check-target-available --target $TARGET"
|
||||
|
||||
# Successful build, so we need to:
|
||||
#
|
||||
# * Upload the docs to S3
|
||||
# * Upload the 00-index.tar file to S3 (TODO: this is probably no longer necessary, since snapshots never modify .cabal files)
|
||||
# * Upload the new plan .yaml file to the appropriate Github repo
|
||||
# * Register as a new Hackage distro
|
||||
docker run $ARGS_UPLOAD $IMAGE /bin/bash -c "stackage-curator upload-docs --target $TARGET --bundle-file $BUNDLE_FILE && stackage-curator upload-index --plan-file $PLAN_FILE --target $TARGET && stackage-curator upload-github --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --target $TARGET && exec stackage-curator hackage-distro --plan-file $PLAN_FILE --target $TARGET"
|
||||
|
||||
date
|
||||
|
||||
72
automated/check-cabal.sh
Executable file
72
automated/check-cabal.sh
Executable file
@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# This script is intended for testing Cabal HEAD against a Stackage snapshot.
|
||||
|
||||
set -eux
|
||||
|
||||
ROOT=$(cd $(dirname $0) ; pwd)
|
||||
TARGET=$1
|
||||
|
||||
# For nightly-YYYY-MM-DD, tag should be nightly
|
||||
# For lts-X.Y, tag should be ltsX
|
||||
SHORTNAME=$(echo $TARGET | cut -d- -f 1)
|
||||
if [ $SHORTNAME = "lts" ]
|
||||
then
|
||||
TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@')
|
||||
PLAN_URL=https://raw.githubusercontent.com/fpco/lts-haskell/master/$TARGET.yaml
|
||||
else
|
||||
TAG=$SHORTNAME
|
||||
PLAN_URL=https://raw.githubusercontent.com/fpco/stackage-nightly/master/$TARGET.yaml
|
||||
fi
|
||||
|
||||
IMAGE=snoyberg/stackage:$TAG
|
||||
|
||||
PLAN_FILE=$TARGET-plan.yaml
|
||||
DOCMAP_FILE=$TARGET-docmap.yaml
|
||||
BUNDLE_FILE=$TARGET.bundle=
|
||||
|
||||
CABAL_DIR=$ROOT/cabal
|
||||
STACK_DIR=$ROOT/stack
|
||||
GHC_DIR=$ROOT/ghc
|
||||
DOT_STACKAGE_DIR=$ROOT/dot-stackage
|
||||
WORKDIR=$ROOT/$TAG/work
|
||||
EXTRA_BIN_DIR=$ROOT/extra-bin
|
||||
|
||||
mkdir -p \
|
||||
"$CABAL_DIR" \
|
||||
"$STACK_DIR" \
|
||||
"$GHC_DIR" \
|
||||
"$DOT_STACKAGE_DIR" \
|
||||
"$WORKDIR" \
|
||||
"$EXTRA_BIN_DIR"
|
||||
|
||||
curl "$PLAN_URL" > $WORKDIR/$PLAN_FILE
|
||||
|
||||
mkdir -p $ROOT/bin
|
||||
BINDIR=$(cd $ROOT/bin ; pwd)
|
||||
(
|
||||
cd $BINDIR
|
||||
rm -f stackage-curator stackage-curator.bz2
|
||||
wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
|
||||
bunzip2 stackage-curator.bz2
|
||||
chmod +x stackage-curator
|
||||
)
|
||||
|
||||
ARGS_COMMON="--rm -v $WORKDIR:/home/stackage/work -w /home/stackage/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR:/home/stackage/bin:ro"
|
||||
ARGS_PREBUILD="$ARGS_COMMON -u $USER -v $CABAL_DIR:/home/stackage/.cabal -v $STACK_DIR:/home/stackage/.stack -v $GHC_DIR:/home/stackage/.ghc -v $DOT_STACKAGE_DIR:/home/stackage/.stackage"
|
||||
ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:/home/stackage/.cabal:ro -v $STACK_DIR:/home/stackage/.stack:ro -v $GHC_DIR:/home/stackage/.ghc:ro"
|
||||
|
||||
# Get latest stack
|
||||
curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack'
|
||||
|
||||
# Do all of the pre-build actions:
|
||||
#
|
||||
# * Update the package index
|
||||
# * Fetch all needed tarballs (the build step does not have write access to the tarball directory)
|
||||
# * Do a single unpack to create the package index cache (again due to directory perms)
|
||||
docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "/home/stackage/bin/stack update && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && /home/stackage/bin/stack unpack random"
|
||||
|
||||
# Now do the actual build. We need to first set the owner of the home directory
|
||||
# correctly, so we run the command as root, change owner, and then use sudo to
|
||||
# switch back to the current user
|
||||
docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER /home/stackage && sudo -E -u $USER env \"PATH=\$PATH:/home/stackage/bin\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET --cabal-from-head"
|
||||
24
automated/cron.sh
Executable file
24
automated/cron.sh
Executable file
@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -eux
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
CRONDIR=$(pwd)/crondir
|
||||
mkdir -p $CRONDIR
|
||||
source aws.sh
|
||||
|
||||
IMAGE=fpco/stackage-server-prod:latest
|
||||
docker pull $IMAGE
|
||||
|
||||
stack update
|
||||
|
||||
date
|
||||
echo "Running stackage-server-cron..."
|
||||
docker run --rm \
|
||||
-v $CRONDIR:/home/ubuntu \
|
||||
-v $HOME/.stack/indices:/home/ubuntu/.stack/indices:ro \
|
||||
--workdir /home/ubuntu \
|
||||
-p 17834:17834 \
|
||||
$IMAGE \
|
||||
bash -c "useradd $(whoami) -u $(id -u); sudo -u $(whoami) env HOME=/home/ubuntu AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY bash -c '/usr/local/bin/stackage-server-cron 2>&1 | tee -a /home/ubuntu/stackage-server-cron.log'"
|
||||
10
automated/run-nightly.sh
Executable file
10
automated/run-nightly.sh
Executable file
@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
while true; do
|
||||
./build.sh nightly-$(date -u +%F)
|
||||
date
|
||||
sleep 30m
|
||||
echo
|
||||
done
|
||||
File diff suppressed because it is too large
Load Diff
6
check
Executable file
6
check
Executable file
@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Convenience script for checking constraints locally
|
||||
|
||||
cd `dirname $0`
|
||||
exec stack exec --resolver ghc-8.0.1 stackage-curator check
|
||||
@ -1,7 +1,7 @@
|
||||
#!/bin/bash -ex
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Work in progress: create a list of commands necessary to get Stackage
|
||||
# up-and-running on a freshly installed Debian-based system (includin Ubuntu).
|
||||
# up-and-running on a freshly installed Debian-based system (including Ubuntu).
|
||||
|
||||
# Quick start:
|
||||
# wget -O - https://raw.github.com/fpco/stackage/master/debian-bootstrap.sh | bash -ex
|
||||
@ -10,64 +10,161 @@
|
||||
# instructions, see:
|
||||
# http://www.stackage.org/install
|
||||
|
||||
add-apt-repository -y ppa:zoogie/sdl2-snapshots
|
||||
set -exu
|
||||
|
||||
mkdir /home/stackage -p
|
||||
locale-gen en_US.UTF-8
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y software-properties-common
|
||||
|
||||
add-apt-repository ppa:hvr/ghc -y
|
||||
add-apt-repository -y ppa:marutter/rrutter
|
||||
# not sure what this was needed for
|
||||
#add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse
|
||||
|
||||
# Set the GHC version
|
||||
GHCVER=8.0.1
|
||||
|
||||
# Get Stack
|
||||
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 575159689BEFB442
|
||||
echo 'deb http://download.fpcomplete.com/ubuntu xenial main'|tee /etc/apt/sources.list.d/fpco.list
|
||||
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
build-essential \
|
||||
libncurses-dev \
|
||||
git \
|
||||
wget \
|
||||
m4 \
|
||||
texlive-full \
|
||||
libgmp3-dev \
|
||||
zlib1g-dev \
|
||||
libedit2 \
|
||||
libedit-dev \
|
||||
ghc-$GHCVER \
|
||||
ghc-$GHCVER-dyn \
|
||||
ghc-$GHCVER-prof \
|
||||
ghc-$GHCVER-htmldocs \
|
||||
hscolour \
|
||||
sudo \
|
||||
curl \
|
||||
freeglut3-dev \
|
||||
libglu1-mesa-dev \
|
||||
libglib2.0-dev \
|
||||
libcairo2-dev \
|
||||
libpango1.0-dev \
|
||||
libgtk2.0-dev \
|
||||
zip \
|
||||
libdevil-dev \
|
||||
llvm \
|
||||
libbz2-dev \
|
||||
libjudy-dev \
|
||||
libsqlite3-dev \
|
||||
libmysqlclient-dev \
|
||||
libpq-dev \
|
||||
libicu-dev \
|
||||
libssl-dev \
|
||||
libgsl0-dev \
|
||||
git \
|
||||
libadns1-dev \
|
||||
libaio1 \
|
||||
libalut-dev \
|
||||
libasound2-dev \
|
||||
libblas-dev \
|
||||
liblapack-dev \
|
||||
libbz2-dev \
|
||||
libcairo2-dev \
|
||||
libclang-3.7-dev \
|
||||
libcurl4-openssl-dev \
|
||||
libdevil-dev \
|
||||
libedit-dev \
|
||||
libedit2 \
|
||||
libfftw3-dev \
|
||||
libflac-dev \
|
||||
libfreenect-dev \
|
||||
libnotify-dev \
|
||||
libgd2-xpm-dev \
|
||||
libyaml-dev \
|
||||
liblzma-dev \
|
||||
libsdl2-dev \
|
||||
libxss-dev \
|
||||
libgtk-3-dev \
|
||||
libxml2-dev \
|
||||
libgeoip-dev \
|
||||
libgirepository1.0-dev \
|
||||
libglfw3-dev \
|
||||
libglib2.0-dev \
|
||||
libglu1-mesa-dev \
|
||||
libgmp3-dev \
|
||||
libgnutls-dev \
|
||||
libgsasl7-dev \
|
||||
libphash0-dev \
|
||||
libzmq3-dev
|
||||
libgsl0-dev \
|
||||
libgtk-3-dev \
|
||||
libgtk2.0-dev \
|
||||
libgtksourceview-3.0-dev \
|
||||
libhidapi-dev \
|
||||
libicu-dev \
|
||||
libjudy-dev \
|
||||
liblapack-dev \
|
||||
libleveldb-dev \
|
||||
liblmdb-dev \
|
||||
liblzma-dev \
|
||||
libmagic-dev \
|
||||
libmagickcore-dev \
|
||||
libmagickwand-dev \
|
||||
libmarkdown2-dev \
|
||||
libmpfr-dev \
|
||||
libmysqlclient-dev \
|
||||
libncurses-dev \
|
||||
libnfc-dev \
|
||||
libnotify-dev \
|
||||
libopenal-dev \
|
||||
libpango1.0-dev \
|
||||
libpcap0.8-dev \
|
||||
libpq-dev \
|
||||
libsdl2-dev \
|
||||
libsnappy-dev \
|
||||
libsndfile1-dev \
|
||||
libsqlite3-dev \
|
||||
libssl-dev \
|
||||
libsystemd-dev \
|
||||
libtagc0-dev \
|
||||
libtre-dev \
|
||||
libudev-dev \
|
||||
libusb-1.0-0-dev \
|
||||
libwebkitgtk-3.0-dev \
|
||||
libxau-dev \
|
||||
libxml2-dev \
|
||||
libxrandr-dev \
|
||||
libxss-dev \
|
||||
libyaml-dev \
|
||||
libzip-dev \
|
||||
libzmq3-dev \
|
||||
llvm-3.7 \
|
||||
m4 \
|
||||
nettle-dev \
|
||||
nodejs \
|
||||
npm \
|
||||
openjdk-8-jdk \
|
||||
r-base \
|
||||
r-base-dev \
|
||||
ruby-dev \
|
||||
stack \
|
||||
wget \
|
||||
xclip \
|
||||
z3 \
|
||||
zip \
|
||||
zlib1g-dev
|
||||
|
||||
mkdir /tmp/nettle-build
|
||||
(
|
||||
cd /tmp/nettle-build
|
||||
wget https://ftp.gnu.org/gnu/nettle/nettle-2.7.1.tar.gz
|
||||
tar zxf nettle-2.7.1.tar.gz
|
||||
cd nettle-2.7.1
|
||||
./configure --prefix=/usr
|
||||
make
|
||||
make install
|
||||
# Put documentation where we expect it
|
||||
mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc
|
||||
|
||||
mkdir -p /usr/lib/x86_64-linux-gnu/
|
||||
ln -sfv /usr/lib/libnettle.so.4.7 /usr/lib/x86_64-linux-gnu/libnettle.so.4
|
||||
)
|
||||
rm -rf /tmp/nettle-build
|
||||
# Buggy versions of ld.bfd fail to link some Haskell packages:
|
||||
# https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is
|
||||
# faster anyways and uses less RAM.
|
||||
update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.gold" 20
|
||||
update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
|
||||
|
||||
# GHC requires a specific LLVM version on the system PATH for its LLVM backend.
|
||||
# This version is tracked here:
|
||||
# https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing
|
||||
#
|
||||
# GHC 8.0 requires LLVM 3.7 tools (specifically, llc-3.7 and opt-3.7).
|
||||
update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.7" 50
|
||||
update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.7" 50
|
||||
|
||||
# install ocilib dependencies then build and install ocilib
|
||||
cd /tmp \
|
||||
&& wget https://storage.googleapis.com/oracle.fpinsight.com/instantClient/oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
|
||||
&& dpkg -i oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
|
||||
&& rm -f oracle-instantclient12.1-basiclite_12.1.0.2.0-2_amd64.deb \
|
||||
&& wget https://storage.googleapis.com/oracle.fpinsight.com/instantClient/oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& dpkg -i oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& rm -f oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb \
|
||||
&& wget https://github.com/vrogier/ocilib/archive/v4.2.1.tar.gz \
|
||||
&& tar xvf v4.2.1.tar.gz \
|
||||
&& cd /tmp/ocilib-4.2.1 \
|
||||
&& ./configure --with-oracle-import=linkage \
|
||||
--with-oracle-charset=ansi \
|
||||
--with-oracle-headers-path=/usr/include/oracle/12.1/client64 \
|
||||
--with-oracle-lib-path=/usr/lib/oracle/12.1/client64/lib \
|
||||
&& make \
|
||||
&& make install \
|
||||
&& cd \
|
||||
&& rm -rf /tmp/ocilib-4.2.1 \
|
||||
&& echo "/usr/local/lib" > /etc/ld.so.conf.d/usr-local.conf \
|
||||
&& echo "/usr/lib/oracle/12.1/client64/lib" > /etc/ld.so.conf.d/oracle-client.conf \
|
||||
&& ldconfig
|
||||
|
||||
# Add JDK to system paths.
|
||||
echo "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/" > /etc/ld.so.conf.d/openjdk.conf \
|
||||
&& ldconfig
|
||||
|
||||
Loading…
Reference in New Issue
Block a user