More cleanup

.gitignore

@@ -1,26 +1,11 @@
 dist
-cabal-dev
 *.o
 *.hi
 *.chi
 *.chs.h
-.virthualenv
 *.swp
-/runtests/
-/build-plan.log
-/build.log
 /logs/
-/sandbox/
-/build-tools.log
-/logs-tools/
-build-plan.txt
-build-plan.csv
-hackage-map.txt
-module-name-conflicts.txt
-/exclusive
-/inclusive
-*.stackage
-/haddock/
-/build-summary/
 /.cabal-sandbox/
 cabal.sandbox.config
+nightly-*.yaml
+lts-*.yaml

README.md

@@ -42,41 +42,14 @@ agreement](https://github.com/fpco/stackage/wiki/Maintainers-Agreement).
 Build the package set
 ---------------------
 
-As this project is just starting, we don't really have a solid set of steps. In
+Generally, building the package set should be done only by the Jenkins machine
-general, the following set of commands should be good for getting started:
+or by the official maintainers, as the process does require quite a bit of
+setup on the local machine. That said, you'll likely be able to get a stable
+build by running:
 
     cabal update
-    git clone https://github.com/fpco/stackage
+    cabal install stackage
-    cd stackage
+    stackage nightly
-    cabal sandbox init # requires cabal-install 1.18
-    cabal install --only-dependencies
-    cabal configure
-    cabal build
-    ./patching/scripts/create-tarballs.sh
-    ./dist/build/stackage/stackage select
-    ./dist/build/stackage/stackage check
-    ./dist/build/stackage/stackage build # takes a *long* time
-    ./dist/build/stackage/stackage test # also takes a *long* time
 
-Notes
+The code itself
------
+---------------
-
-Make sure to have Cabal-1.16 installed in either your global or user database,
-regardless of any sandboxing, as custom build types require it to be present.
-You must build with cabal-install 1.16, due to several important bug fixes.
-
-Using a non-Haskell Platform versions of GHC
---------------------------------------------
-
-By default, Stackage bases itself off of the Haskell Platform for determining
-which packages are core packages, and locks down package versions to match the
-Haskell Platform selections. This works fine when you are compiling with the
-same version of GHC as the Haskell Platform was built on. If you're using a
-different version of GHC, you'll probably need to use the following options for
-the `select` call:
-
-    --no-platform --use-global-db
-
-The former says to disregard Haskell Platform package versions, and the latter
-says to determine which packages are core packages based on their presence in
-the global package database.

ROADMAP.md

@@ -1,61 +0,0 @@
-## Processing
-
-High level series of steps for processing
-
-### Nightlies
-
-1. Get list of core packages
-2. Get build constraints from list of maintained packages
-3. Load up package index
-4. Calculate build plan using newest versions of packages
-5. Write out a YAML file with complete build plan
-6. Verify that the build plan can be compiled
-7. Perform the build
-
-### LTS
-
-1. Load up most recent build plan
-2. Convert build plan into constraints for next build
-3. Continue from step (3) above
-
-## Code explanation
-
-We start off with *constraints*. Constraints state things like "package X has a
-given version range," who the maintainer is for a package, the description of
-the system/compiler being used, etc. `BuildConstraints` describes the build as
-a whole, whereas `PackageConstraints` describes the constraints on an
-individual package.
-
-There are two primary ways of getting a `BuildConstraints`.
-`defaultBuildConstraints` inspects the first GHC in the PATH environment variable to
-determine GHC version, core packages, core tools, etc. It then uses the
-`Stackage.Config` module to extract information on additional packages to be
-installed. The secondary approach is in `Stackage2.UpdateBuildPlan`, which will be
-discussed later.
-
-`BuildConstraints` does not specify a build completely. That is given by a
-`BuildPlan`, which is similarly broken down into `BuildPlan` and `PackagePlan`.
-In order to get a `BuildPlan`, we need two pieces of information: the
-`BuildConstraints`, and a package index. The package index (usually downloaded
-from Hackage) is a collection of all of the cabal files available.
-
-By applying a `BuildConstraints` to a package index (via `newBuildPlan`), we
-get a proposed `BuildPlan`. There is no guarantee that this `BuildPlan` is
-valid. To validate it, we use `checkBuildPlan`. A `BuildPlan` is an instance of
-both `ToJSON` and `FromJSON`, and therefore can be serialized to a file for
-later use.
-
-When dealing with LTS Haskell, we want to be able to take a `BuildPlan`, and
-update to a newer `BuildPlan` that keeps all packages at the same major
-version.  `updateBuildConstraints` turns a `BuildPlan` into a new
-`BuildConstraints` with that restriction, and `updateBuildPlan` applies
-`newBuildPlan` to that result. As mentioned previously: this is *not* a
-validated result, and therefore `checkBuildPlan` must be used.
-
-A `BuildPlan` can be acted on. This is done to check that all packages compile
-together, run relevant test suites, test Haddock documentation is correct, and
-produce as artifacts both a self-contained GHC binary package database and a
-set of Haddock documentation. (Not yet implemented.)
-
-A `BuildPlan` may be converted into a bundle to be uploaded to Stackage Server.
-(Not yet implemented.)

debian-bootstrap.sh

@@ -27,13 +27,3 @@ wget http://hackage.haskell.org/packages/archive/cabal-install/1.18.0.5/cabal-in
 tar zxfv cabal-install-1.18.0.5.tar.gz
 cd cabal-install-1.18.0.5/
 bash bootstrap.sh
-cd ..
-git clone --recursive https://github.com/fpco/stackage
-cd stackage
-cabal update
-cabal install
-
-stackage select && \
-    stackage check && \
-    stackage build && \
-    stackage test

full-run.sh

@@ -1,6 +0,0 @@
-#!/bin/bash -ex
-
-runghc app/stackage.hs select $*
-runghc app/stackage.hs check
-runghc app/stackage.hs build
-runghc app/stackage.hs test

jenkins-build.sh

@@ -1,22 +0,0 @@
-#!/bin/bash -ex
-
-cabal update
-rm -rf patching/tarballs
-(cd patching/ && ./scripts/create-tarballs.sh)
-cabal install
-cabal install Cabal-$(cabal --version | sed -n 's@using version \(.*\) of the Cabal library@\1@p')
-./dist/build/stackage/stackage select $*
-./dist/build/stackage/stackage check
-./dist/build/stackage/stackage build
-./dist/build/stackage/stackage test
-
-cabal install http-client --force-reinstalls
-
-for f in inclusive exclusive
-do
-    cd $f
-
-    bash create-snapshot.sh
-
-    cd ..
-done

patching/.gitignore

@@ -1,2 +0,0 @@
-/tarballs
-/work

patching/scripts/create-patches.sh

@@ -1,25 +0,0 @@
-#!/bin/bash -ex
-
-shopt -s nullglob
-
-mkdir -p patches work
-
-for f in work/*
-do
-    (
-    cd $f
-    PKG=$(basename $(pwd))
-    cabal sdist
-    cd ../..
-    rm -rf tmp
-    mkdir tmp
-    cd tmp
-    tar zxfv ../$f/dist/$PKG.tar.gz
-    mv $PKG new
-    cabal unpack $PKG
-    mv $PKG orig
-    diff -ruN orig new > ../patches/$PKG.patch || true
-    cd ..
-    rm -rf tmp
-    )
-done

patching/scripts/create-tarballs.sh

@@ -1,31 +0,0 @@
-#!/bin/bash -ex
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-(
-cd "$DIR/.."
-
-shopt -s nullglob
-
-mkdir -p tarballs
-
-for f in patches/*
-do
-    (
-    PKG1=$(basename $f)
-    PKG=${PKG1%.patch}
-    rm -rf tmp
-    mkdir tmp
-    (
-    cd tmp
-    cabal unpack $PKG
-    cd $PKG
-    patch -p1 < ../../$f
-    cabal sdist
-    mv dist/$PKG.tar.gz ../../tarballs
-    )
-    rm -rf tmp
-    )
-done
-
-)

patching/scripts/edit-package.sh

@@ -1,6 +0,0 @@
-#!/bin/bash -ex
-
-mkdir -p work
-cd work
-rm -rf $1
-cabal unpack $1

stackage-upload.hs

@@ -1,176 +0,0 @@
-{-# LANGUAGE OverloadedStrings #-}
-import           Control.Monad                         (filterM, when)
-import qualified Data.ByteString.Char8                 as S8
-import qualified Data.ByteString.Lazy                  as L
-import qualified Data.ByteString.Lazy.Char8            as L8
-import           Data.List                             (isInfixOf, isPrefixOf,
-                                                        sort)
-import           Network.HTTP.Client
-import           Network.HTTP.Client.MultipartFormData
-import           System.Directory                      (doesDirectoryExist,
-                                                        getDirectoryContents)
-import           System.Environment                    (getArgs, getEnv,
-                                                        getProgName)
-import           System.Exit                           (exitFailure)
-import           System.Exit                           (ExitCode (ExitSuccess))
-import           System.FilePath                       (takeDirectory, (</>))
-import           System.Process                        (createProcess, cwd,
-                                                        proc, waitForProcess)
-
-main :: IO ()
-main = withManager defaultManagerSettings $ \m -> do
-    args <- getArgs
-    token <- readFile "/auth-token"
-    (filepath, alias) <-
-        case args of
-            [x, y] -> return (x, y)
-            _ -> do
-                pn <- getProgName
-                putStrLn $ concat
-                    [ "Usage: "
-                    , pn
-                    , " <filepath> <alias name>"
-                    ]
-                exitFailure
-
-    let uploadDocs = "exclusive" `isInfixOf` alias
-        uploadHackageDistro = alias == "unstable-ghc78-exclusive"
-
-    putStrLn $ concat
-        [ "Uploading "
-        , filepath
-        , " as "
-        , alias
-        ]
-
-    req1 <- parseUrl "http://www.stackage.org/upload"
-    let formData =
-            [ partBS "alias" $ S8.pack alias
-            , partFileSource "stackage" filepath
-            ]
-    req2 <- formDataBody formData req1
-    let req3 = req2
-            { method = "PUT"
-            , requestHeaders =
-                [ ("Authorization", S8.pack token)
-                , ("Accept", "application/json")
-                ] ++ requestHeaders req2
-            , redirectCount = 0
-            , checkStatus = \_ _ _ -> Nothing
-            }
-    res <- httpLbs req3 m
-
-    snapid <-
-        case lookup "x-stackage-ident" $ responseHeaders res of
-            Just snapid -> do
-                putStrLn $ "New ident: " ++ S8.unpack snapid
-                return snapid
-            Nothing -> error $ "An error occurred: " ++ show res
-
-    when uploadDocs $ do
-        putStrLn "Generating index file"
-        let root = takeDirectory filepath </> "haddock"
-        contents <- getDirectoryContents root
-        dirs <- filterM (\n -> doesDirectoryExist $ root </> n)
-              $ filter (not . ("." `isPrefixOf`))
-              $ sort contents
-        writeFile (root </> "index.html") $ mkIndex (S8.unpack snapid) dirs
-        writeFile (root </> "style.css") styleCss
-
-        putStrLn "Creating tarball"
-        (Nothing, Nothing, Nothing, ph) <- createProcess
-            (proc "tar" $ "cJf" : "haddock.tar.xz" : "index.html" : "style.css" : dirs)
-                { cwd = Just root
-                }
-        ec <- waitForProcess ph
-        if ec == ExitSuccess
-            then putStrLn "Haddock tarball generated"
-            else error "Error generating Haddock tarball"
-
-        putStrLn "Uploading Haddocks"
-
-        req1 <- parseUrl $ "http://www.stackage.org/upload-haddock/"
-                        ++ S8.unpack snapid
-        let formData =
-                [ partFileSource "tarball" $ root </> "haddock.tar.xz"
-                ]
-        req2 <- formDataBody formData req1
-        let req3 = req2
-                { method = "PUT"
-                , requestHeaders =
-                    [ ("Authorization", S8.pack token)
-                    , ("Accept", "application/json")
-                    ] ++ requestHeaders req2
-                , redirectCount = 0
-                , checkStatus = \_ _ _ -> Nothing
-                }
-        httpLbs req3 m >>= print
-
-    when uploadHackageDistro $ do
-        lbs <- L.readFile $ takeDirectory filepath </> "build-plan.csv"
-        let req = "http://hackage.haskell.org/distro/Stackage/packages.csv"
-                { requestHeaders = [("Content-Type", "text/csv")]
-                , requestBody = RequestBodyLBS $ L.intercalate "\n" $ L8.lines lbs
-                , checkStatus = \_ _ _ -> Nothing
-                , method = "PUT"
-                }
-        httpLbs req m >>= print
-
-mkIndex :: String -> [String] -> String
-mkIndex snapid dirs = concat
-    [ "<!DOCTYPE html>\n<html lang='en'><head><title>Haddocks index</title>"
-    , "<link rel='stylesheet' href='https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css'>"
-    , "<link rel='stylesheet' href='style.css'>"
-    , "<link rel='shortcut icon' href='http://www.stackage.org/static/img/favicon.ico' />"
-    , "</head>"
-    , "<body><div class='container'>"
-    , "<div class='row'><div class='span12 col-md-12'>"
-    , "<h1>Haddock documentation index</h1>"
-    , "<p class='return'><a href=\"http://www.stackage.org/stackage/"
-    , snapid
-    , "\">Return to snapshot</a></p><ul>"
-    , concatMap toLI dirs
-    , "</ul></div></div></div></body></html>"
-    ]
-  where
-    toLI name = concat
-        [ "<li><a href='"
-        , name
-        , "/index.html'>"
-        , name
-        , "</a></li>"
-        ]
-
-styleCss :: String
-styleCss = concat
-    [ "@media (min-width: 530px) {"
-    , "ul { -webkit-column-count: 2; -moz-column-count: 2; column-count: 2 }"
-    , "}"
-    , "@media (min-width: 760px) {"
-    , "ul { -webkit-column-count: 3; -moz-column-count: 3; column-count: 3 }"
-    , "}"
-    , "ul {"
-    , "  margin-left: 0;"
-    , "  padding-left: 0;"
-    , "  list-style-type: none;"
-    , "}"
-    , "body {"
-    , "  background: #f0f0f0;"
-    , "  font-family: 'Lato', sans-serif;"
-    , "  text-shadow: 1px 1px 1px #ffffff;"
-    , "  font-size: 20px;"
-    , "  line-height: 30px;"
-    , "  padding-bottom: 5em;"
-    , "}"
-    , "h1 {"
-    , "  font-weight: normal;"
-    , "  color: #06537d;"
-    , "  font-size: 45px;"
-    , "}"
-    , ".return a {"
-    , "  color: #06537d;"
-    , "  font-style: italic;"
-    , "}"
-    , ".return {"
-    , "  margin-bottom: 1em;"
-    , "}"]