Add documentation for presigned operations (#68)

- Add full examples for presignedGetObjectUrl and
  presignedPutObjectUrl

- Lower-case `*URL` API functions to `*Url` to adopt a Haskell naming
  convention

Finishes and fixes #35, #36 and #37.
This commit is contained in:
Aditya Manthramurthy 2017-10-16 16:13:18 +00:00 committed by GitHub
parent c26af265ec
commit b7dfd0457d
7 changed files with 403 additions and 56 deletions

View File

@ -20,15 +20,15 @@ awsCI { connectAccesskey = "your-access-key"
```
|Bucket operations|Object Operations|
|:---|:---|
|[`listBuckets`](#listBuckets) |[`getObject`](#getObject)|
|[`makeBucket`](#makeBucket)|[`putObject`](#putObject)|
|[`removeBucket`](#removeBucket)|[`fGetObject`](#fGetObject)|
|[`listObjects`](#listObjects)|[`fPutObject`](#fPutObject)|
|[`listObjectsV1`](#listObjectsV1)|[`copyObject`](#copyObject)|
|[`listIncompleteUploads`](#listIncompleteUploads)|[`removeObject`](#removeObject)|
|[`bucketExists`](#bucketExists)||
|Bucket operations|Object Operations|Presigned Operations|
|:---|:---|:---|
|[`listBuckets`](#listBuckets) |[`getObject`](#getObject)|[`presignedGetObjectUrl`](#presignedGetObjectUrl)|
|[`makeBucket`](#makeBucket)|[`putObject`](#putObject)|[`presignedPutObjectUrl`](#presignedPutObjectUrl)|
|[`removeBucket`](#removeBucket)|[`fGetObject`](#fGetObject)|[`presignedPostPolicy`](#presignedPostPolicy)|
|[`listObjects`](#listObjects)|[`fPutObject`](#fPutObject)||
|[`listObjectsV1`](#listObjectsV1)|[`copyObject`](#copyObject)||
|[`listIncompleteUploads`](#listIncompleteUploads)|[`removeObject`](#removeObject)||
|[`bucketExists`](#bucketExists)|||
## 1. Connecting and running operations on the storage service
@ -685,9 +685,216 @@ In the expression `bucketExists bucketName` the parameters are:
| `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket |
<!-- ## 4. Presigned operations -->
## 4. Presigned operations
<!-- TODO -->
<a name="presignedGetObjectUrl"></a>
### presignedGetObjectUrl :: Bucket -> Object -> UrlExpiry -> Query -> RequestHeaders -> Minio ByteString
Generate a URL with authentication signature to GET (download) an
object. All extra query parameters and headers passed here will be
signed and are required when the generated URL is used. Query
parameters could be used to change the response headers sent by the
server. Headers can be used to set Etag match conditions among others.
For a list of possible request parameters and headers, please refer
to the GET object REST API AWS S3 documentation.
__Parameters__
In the expression `presignedGetObjectUrl bucketName objectName expiry queryParams headers`
the parameters are:
|Param |Type |Description |
|:---|:---| :---|
| `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket |
| `objectName` | _Object_ (alias for `Text`) | Name of the object |
| `expiry` | _UrlExpiry_ (alias for `Int`) | Url expiry time in seconds |
| `queryParams` | _Query_ (from package `http-types:Network.HTTP.Types`) | Query parameters to add to the URL |
| `headers` | _RequestHeaders_ (from package `http-types:Network.HTTP.Types` | Request headers that would be used with the URL |
__Return Value__
Returns the generated URL - it will include authentication
information.
|Return type |Description |
|:---|:---|
| _ByteString_ | Generated presigned URL |
__Example__
```haskell
{-# Language OverloadedStrings #-}
import Network.Minio
import qualified Data.ByteString.Char8 as B
main :: IO ()
main = do
let
bucket = "mybucket"
object = "myobject"
res <- runMinio minioPlayCI $ do
-- Set a 7 day expiry for the URL
presignedGetObjectUrl bucket object (7*24*3600) [] []
-- Print the URL on success.
putStrLn $ either
(("Failed to generate URL: " ++) . show)
B.unpack
res
```
<a name="presignedPutObjectUrl"></a>
### presignedPutObjectUrl :: Bucket -> Object -> UrlExpiry -> RequestHeaders -> Minio ByteString
Generate a URL with authentication signature to PUT (upload) an
object. Any extra headers if passed, are signed, and so they are
required when the URL is used to upload data. This could be used, for
example, to set user-metadata on the object.
For a list of possible headers to pass, please refer to the PUT object
REST API AWS S3 documentation.
__Parameters__
In the expression `presignedPutObjectUrl bucketName objectName expiry headers`
the parameters are:
|Param |Type |Description |
|:---|:---| :---|
| `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket |
| `objectName` | _Object_ (alias for `Text`) | Name of the object |
| `expiry` | _UrlExpiry_ (alias for `Int`) | Url expiry time in seconds |
| `headers` | _RequestHeaders_ (from package `http-types:Network.HTTP.Types` | Request headers that would be used with the URL |
__Return Value__
Returns the generated URL - it will include authentication
information.
|Return type |Description |
|:---|:---|
| _ByteString_ | Generated presigned URL |
__Example__
```haskell
{-# Language OverloadedStrings #-}
import Network.Minio
import qualified Data.ByteString.Char8 as B
main :: IO ()
main = do
let
bucket = "mybucket"
object = "myobject"
res <- runMinio minioPlayCI $ do
-- Set a 7 day expiry for the URL
presignedPutObjectUrl bucket object (7*24*3600) [] []
-- Print the URL on success.
putStrLn $ either
(("Failed to generate URL: " ++) . show)
B.unpack
res
```
<a name="presignedPostPolicy"></a>
### presignedPostPolicy :: PostPolicy -> Minio (ByteString, Map.Map Text ByteString)
Generate a presigned URL and POST policy to upload files via a POST
request. This is intended for browser uploads and generates form data
that should be submitted in the request.
The `PostPolicy` argument is created using the `newPostPolicy` function:
#### newPostPolicy :: UTCTime -> [PostPolicyCondition] -> Either PostPolicyError PostPolicy
In the expression `newPostPolicy expirationTime conditions` the parameters are:
|Param | Type| Description |
|:---|:---|:---|
| `expirationTime` | _UTCTime_ (from package `time:Data.Time.UTCTime`) | The expiration time for the policy |
| `conditions` | _[PostPolicyConditions]_ | List of conditions to be added to the policy |
The policy conditions are created using various helper functions -
please refer to the Haddocks for details.
Since conditions are validated by `newPostPolicy` it returns an
`Either` value.
__Return Value__
`presignedPostPolicy` returns a 2-tuple - the generated URL and a map
containing the form-data that should be submitted with the request.
__Example__
```haskell
{-# Language OverloadedStrings #-}
import Network.Minio
import qualified Data.ByteString as B
import qualified Data.ByteString.Char8 as Char8
import qualified Data.Map.Strict as Map
import qualified Data.Text.Encoding as Enc
import qualified Data.Time as Time
main :: IO ()
main = do
now <- Time.getCurrentTime
let
bucket = "mybucket"
object = "myobject"
-- set an expiration time of 10 days
expireTime = Time.addUTCTime (3600 * 24 * 10) now
-- create a policy with expiration time and conditions - since the
-- conditions are validated, newPostPolicy returns an Either value
policyE = newPostPolicy expireTime
[ -- set the object name condition
ppCondKey "photos/my-object"
-- set the bucket name condition
, ppCondBucket "my-bucket"
-- set the size range of object as 1B to 10MiB
, ppCondContentLengthRange 1 (10*1024*1024)
-- set content type as jpg image
, ppCondContentType "image/jpeg"
-- on success set the server response code to 200
, ppCondSuccessActionStatus 200
]
case policyE of
Left err -> putStrLn $ show err
Right policy -> do
res <- runMinio minioPlayCI $ do
(url, formData) <- presignedPostPolicy policy
-- a curl command is output to demonstrate using the generated
-- URL and form-data
let
formFn (k, v) = B.concat ["-F ", Enc.encodeUtf8 k, "=",
"'", v, "'"]
formOptions = B.intercalate " " $ map formFn $ Map.toList formData
return $ B.intercalate " " $
["curl", formOptions, "-F file=@/tmp/photo.jpg", url]
case res of
Left e -> putStrLn $ "post-policy error: " ++ (show e)
Right cmd -> do
putStrLn $ "Put a photo at /tmp/photo.jpg and run command:\n"
-- print the generated curl command
Char8.putStrLn cmd
```
<!-- ## 5. Bucket policy/notification operations -->

82
examples/PresignedGetObject.hs Executable file
View File

@ -0,0 +1,82 @@
#!/usr/bin/env stack
-- stack --resolver lts-9.1 runghc --package minio-hs
--
-- Minio Haskell SDK, (C) 2017 Minio, Inc.
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
{-# LANGUAGE OverloadedStrings #-}
import Network.Minio
import Control.Monad.IO.Class (liftIO)
import qualified Data.ByteString.Char8 as B
import Data.CaseInsensitive (original)
import qualified Data.Conduit.Combinators as CC
import qualified Data.Text.Encoding as E
-- | The following example uses minio's play server at
-- https://play.minio.io:9000. The endpoint and associated
-- credentials are provided via the libary constant,
--
-- > minioPlayCI :: ConnectInfo
--
main :: IO ()
main = do
let
bucket = "my-bucket"
object = "my-object"
kb15 = 15*1024
-- Set query parameter to modify content disposition response
-- header
queryParam = [("response-content-disposition",
Just "attachment; filename=\"your-filename.txt\"")]
res <- runMinio minioPlayCI $ do
liftIO $ B.putStrLn "Upload a file that we will fetch with a presigned URL..."
putObject bucket object (CC.repeat "a") (Just kb15)
liftIO $ putStrLn $ "Done. Object created at: my-bucket/my-object"
-- Extract Etag of uploaded object
(ObjectInfo _ _ etag _) <- statObject bucket object
-- Set header to add an if-match constraint - this makes sure
-- the fetching fails if the object is changed on the server
let headers = [("If-Match", E.encodeUtf8 etag)]
-- Generate a URL with 7 days expiry time - note that the headers
-- used above must be added to the request with the signed URL
-- generated.
url <- presignedGetObjectUrl "my-bucket" "my-object" (7*24*3600)
queryParam headers
return (headers, etag, url)
case res of
Left e -> putStrLn $ "presignedPutObject URL failed." ++ show e
Right (headers, etag, url) -> do
-- We generate a curl command to demonstrate usage of the signed
-- URL.
let
hdrOpt (k, v) = B.concat ["-H '", original k, ": ", v, "'"]
curlCmd = B.intercalate " " $
["curl --fail"] ++ map hdrOpt headers ++
["-o /tmp/myfile", B.concat ["'", url, "'"]]
putStrLn $ "The following curl command would use the presigned " ++
"URL to fetch the object and write it to \"/tmp/myfile\":"
B.putStrLn curlCmd

59
examples/PresignedPutObject.hs Executable file
View File

@ -0,0 +1,59 @@
#!/usr/bin/env stack
-- stack --resolver lts-9.1 runghc --package minio-hs
--
-- Minio Haskell SDK, (C) 2017 Minio, Inc.
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
{-# LANGUAGE OverloadedStrings #-}
import Network.Minio
import qualified Data.ByteString.Char8 as B
import Data.CaseInsensitive (original)
-- | The following example uses minio's play server at
-- https://play.minio.io:9000. The endpoint and associated
-- credentials are provided via the libary constant,
--
-- > minioPlayCI :: ConnectInfo
--
main :: IO ()
main = do
let
-- Use headers to set user-metadata - note that this header will
-- need to be set when the URL is used to make an upload.
headers = [("x-amz-meta-url-creator",
"minio-hs-presigned-put-example")]
res <- runMinio minioPlayCI $ do
-- generate a URL with 7 days expiry time
presignedPutObjectURL "my-bucket" "my-object" (7*24*3600) headers
case res of
Left e -> putStrLn $ "presignedPutObject URL failed." ++ show e
Right url -> do
-- We generate a curl command to demonstrate usage of the signed
-- URL.
let
hdrOpt (k, v) = B.concat ["-H '", original k, ": ", v, "'"]
curlCmd = B.intercalate " " $
["curl "] ++ map hdrOpt headers ++
["-T /tmp/myfile", B.concat ["'", url, "'"]]
B.putStrLn $ "The following curl command would use the presigned " ++
"URL to upload the file at \"/tmp/myfile\":"
B.putStrLn curlCmd

View File

@ -17,11 +17,10 @@
-- limitations under the License.
--
{-# Language OverloadedStrings #-}
{-# LANGUAGE OverloadedStrings #-}
import Network.Minio
import qualified Data.Conduit.Combinators as CC
import Prelude
-- | The following example uses minio's play server at
-- https://play.minio.io:9000. The endpoint and associated
@ -42,7 +41,7 @@ main = do
res1 <- runMinio minioPlayCI $
putObject bucket object (CC.repeat "a") (Just kb15)
case res1 of
Left e -> putStrLn $ "putObject failed." ++ show e
Left e -> putStrLn $ "putObject failed." ++ show e
Right () -> putStrLn "putObject succeeded."
@ -50,5 +49,5 @@ main = do
res2 <- runMinio minioPlayCI $
fPutObject bucket object localFile
case res2 of
Left e -> putStrLn $ "fPutObject failed." ++ show e
Left e -> putStrLn $ "fPutObject failed." ++ show e
Right () -> putStrLn "fPutObject succeeded."

View File

@ -79,9 +79,9 @@ module Network.Minio
-- * Presigned Operations
-------------------------
, UrlExpiry
, presignedPutObjectURL
, presignedGetObjectURL
, presignedHeadObjectURL
, presignedPutObjectUrl
, presignedGetObjectUrl
, presignedHeadObjectUrl
, PostPolicyCondition
, ppCondBucket
@ -102,11 +102,11 @@ module Network.Minio
This module exports the high-level Minio API for object storage.
-}
import qualified Data.Conduit as C
import qualified Data.Conduit.Binary as CB
import qualified Data.Conduit as C
import qualified Data.Conduit.Binary as CB
import qualified Data.Conduit.Combinators as CC
import Data.Default (def)
import qualified Data.Map as Map
import Data.Default (def)
import qualified Data.Map as Map
import Lib.Prelude

View File

@ -16,10 +16,10 @@
module Network.Minio.PresignedOperations
( UrlExpiry
, makePresignedURL
, presignedPutObjectURL
, presignedGetObjectURL
, presignedHeadObjectURL
, makePresignedUrl
, presignedPutObjectUrl
, presignedGetObjectUrl
, presignedHeadObjectUrl
, PostPolicyCondition(..)
, ppCondBucket
@ -54,17 +54,17 @@ import Network.Minio.Errors
import Network.Minio.Sign.V4
-- | Generate a presigned URL. This function allows for advanced usage
-- - for simple cases prefer the `presigned*URL` functions.
-- - for simple cases prefer the `presigned*Url` functions.
--
-- If region is Nothing, it is picked up from the connection
-- information (no check of bucket existence is performed).
--
-- All extra query parameters or headers are signed, and therefore are
-- required to be sent when the generated URL is actually used.
makePresignedURL :: UrlExpiry -> HT.Method -> Maybe Bucket -> Maybe Object
makePresignedUrl :: UrlExpiry -> HT.Method -> Maybe Bucket -> Maybe Object
-> Maybe Region -> HT.Query -> HT.RequestHeaders
-> Minio ByteString
makePresignedURL expiry method bucket object region extraQuery extraHeaders = do
makePresignedUrl expiry method bucket object region extraQuery extraHeaders = do
when (expiry > 7*24*3600 || expiry < 0) $
throwM $ MErrVInvalidUrlExpiry expiry
@ -98,10 +98,10 @@ makePresignedURL expiry method bucket object region extraQuery extraHeaders = do
--
-- For a list of possible headers to pass, please refer to the PUT
-- object REST API AWS S3 documentation.
presignedPutObjectURL :: Bucket -> Object -> UrlExpiry -> HT.RequestHeaders
presignedPutObjectUrl :: Bucket -> Object -> UrlExpiry -> HT.RequestHeaders
-> Minio ByteString
presignedPutObjectURL bucket object expirySeconds extraHeaders =
makePresignedURL expirySeconds HT.methodPut
presignedPutObjectUrl bucket object expirySeconds extraHeaders =
makePresignedUrl expirySeconds HT.methodPut
(Just bucket) (Just object) Nothing [] extraHeaders
-- | Generate a URL with authentication signature to GET (download) an
@ -113,10 +113,10 @@ presignedPutObjectURL bucket object expirySeconds extraHeaders =
--
-- For a list of possible request parameters and headers, please refer
-- to the GET object REST API AWS S3 documentation.
presignedGetObjectURL :: Bucket -> Object -> UrlExpiry -> HT.Query
presignedGetObjectUrl :: Bucket -> Object -> UrlExpiry -> HT.Query
-> HT.RequestHeaders -> Minio ByteString
presignedGetObjectURL bucket object expirySeconds extraQuery extraHeaders =
makePresignedURL expirySeconds HT.methodGet
presignedGetObjectUrl bucket object expirySeconds extraQuery extraHeaders =
makePresignedUrl expirySeconds HT.methodGet
(Just bucket) (Just object) Nothing extraQuery extraHeaders
-- | Generate a URL with authentication signature to make a HEAD
@ -126,10 +126,10 @@ presignedGetObjectURL bucket object expirySeconds extraQuery extraHeaders =
--
-- For a list of possible headers to pass, please refer to the HEAD
-- object REST API AWS S3 documentation.
presignedHeadObjectURL :: Bucket -> Object -> UrlExpiry
presignedHeadObjectUrl :: Bucket -> Object -> UrlExpiry
-> HT.RequestHeaders -> Minio ByteString
presignedHeadObjectURL bucket object expirySeconds extraHeaders =
makePresignedURL expirySeconds HT.methodHead
presignedHeadObjectUrl bucket object expirySeconds extraHeaders =
makePresignedUrl expirySeconds HT.methodHead
(Just bucket) (Just object) Nothing [] extraHeaders
-- | Represents individual conditions in a Post Policy document.
@ -239,7 +239,7 @@ showPostPolicy :: PostPolicy -> ByteString
showPostPolicy = toS . Json.encode
-- | Generate a presigned URL and POST policy to upload files via a
-- browser. On success, this function returns a URL and a POST
-- browser. On success, this function returns a URL and POST
-- form-data.
presignedPostPolicy :: PostPolicy
-> Minio (ByteString, Map.Map Text ByteString)

View File

@ -433,7 +433,7 @@ liveServerUnitTests = testGroup "Unit tests against a live server"
forM_ [src, copyObj] (removeObject bucket)
, presignedURLFunTest
, presignedUrlFunTest
, presignedPostPolicyFunTest
]
@ -505,8 +505,8 @@ basicTests = funTestWithBucket "Basic tests" $ \step bucket -> do
step "delete object"
deleteObject bucket object
presignedURLFunTest :: TestTree
presignedURLFunTest = funTestWithBucket "presigned URL tests" $
presignedUrlFunTest :: TestTree
presignedUrlFunTest = funTestWithBucket "presigned Url tests" $
\step bucket -> do
let obj = "mydir/myput"
obj2 = "mydir1/myfile1"
@ -514,8 +514,8 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $
-- manager for http requests
mgr <- liftIO $ NC.newManager NC.tlsManagerSettings
step "PUT object presigned URL - makePresignedURL"
putUrl <- makePresignedURL 3600 HT.methodPut (Just bucket)
step "PUT object presigned URL - makePresignedUrl"
putUrl <- makePresignedUrl 3600 HT.methodPut (Just bucket)
(Just obj) (Just "us-east-1") [] []
let size1 = 1000 :: Int64
@ -526,8 +526,8 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $
liftIO $ (NC.responseStatus putResp == HT.status200) @?
"presigned PUT failed"
step "GET object presigned URL - makePresignedURL"
getUrl <- makePresignedURL 3600 HT.methodGet (Just bucket)
step "GET object presigned URL - makePresignedUrl"
getUrl <- makePresignedUrl 3600 HT.methodGet (Just bucket)
(Just obj) (Just "us-east-1") [] []
getResp <- getR mgr getUrl
@ -540,39 +540,39 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $
"presigned put and get got mismatched data"
step "PUT object presigned - presignedPutObjectURL"
putUrl2 <- presignedPutObjectURL bucket obj2 3600 []
putUrl2 <- presignedPutObjectUrl bucket obj2 3600 []
let size2 = 1200
testFile <- mkRandFile size2
putResp2 <- putR size2 testFile mgr putUrl2
liftIO $ (NC.responseStatus putResp2 == HT.status200) @?
"presigned PUT failed (presignedPutObjectURL)"
"presigned PUT failed (presignedPutObjectUrl)"
step "HEAD object presigned URL - presignedHeadObjectURL"
headUrl <- presignedHeadObjectURL bucket obj2 3600 []
step "HEAD object presigned URL - presignedHeadObjectUrl"
headUrl <- presignedHeadObjectUrl bucket obj2 3600 []
headResp <- do req <- NC.parseRequest $ toS headUrl
NC.httpLbs (req {NC.method = HT.methodHead}) mgr
liftIO $ (NC.responseStatus headResp == HT.status200) @?
"presigned HEAD failed (presignedHeadObjectURL)"
"presigned HEAD failed (presignedHeadObjectUrl)"
-- check that header info is accurate
let h = Map.fromList $ NC.responseHeaders headResp
cLen = Map.findWithDefault "0" HT.hContentLength h
liftIO $ (cLen == show size2) @? "Head req returned bad content length"
step "GET object presigned URL - presignedGetObjectURL"
getUrl2 <- presignedGetObjectURL bucket obj2 3600 [] []
step "GET object presigned URL - presignedGetObjectUrl"
getUrl2 <- presignedGetObjectUrl bucket obj2 3600 [] []
getResp2 <- getR mgr getUrl2
liftIO $ (NC.responseStatus getResp2 == HT.status200) @?
"presigned GET failed (presignedGetObjectURL)"
"presigned GET failed (presignedGetObjectUrl)"
-- read content from file to compare with response above
bs2 <- CB.sourceFile testFile $$ CB.sinkLbs
liftIO $ (bs2 == NC.responseBody getResp2) @?
"presigned put and get got mismatched data (presigned*URL)"
"presigned put and get got mismatched data (presigned*Url)"
mapM_ (removeObject bucket) [obj, obj2]
@ -589,7 +589,7 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $
NC.httpLbs req mgr
presignedPostPolicyFunTest :: TestTree
presignedPostPolicyFunTest = funTestWithBucket "presigned URL tests" $
presignedPostPolicyFunTest = funTestWithBucket "Presigned Post Policy tests" $
\step bucket -> do
step "presignedPostPolicy basic test"