RESTful Access to Data Manager (version beta 4.3)

• Introduction
• HTTP Client Requirements
• Common Request Headers
• URLs and API Version
• Login and Authentication
• WADL
• GSFileMetadata, GSDataFormat and, GSDirectoryListing Objects
• Directories
• Uploading files
• Downloading Files
• File and Directory Ownership and Permissions
• Downloading Files in Different Data Formats
• Other File, Directory, and Format Operations
  • Delete a File or Directory
  • Copy a File or Directory
  • Convert a file on the server
  • Obtain Metadata on a File or Directory
  • To get HTTP headers on a file or directory
  • List all Data Formats
• Appendix A: GSFileMetadata JSON object(new)
• Appendix B: GSDirectoryListing JSON object
• Appendix C: GSDataFormat JSON object
• Appendix D: Data Manager Data Formats
• Appendix E: Available Data Format Converters
• Appendix F: ACL, ACE, and SID JSON formats
  • Access Control List (ACL) Object
  • Access Control Entry (ACE) Object
  • Security Identity (SID) Object
• Appendix G: S3FileUploadInfo JSON object
•  

Introduction

Part of the functionality that we are including in the GenomeSpace CDK is a Java API that provides client applications authentication and authorization services for the GenomeSpace system, as well as access to data file services. However, if your application is not written in Java and you want to add GenomeSpace support to it, you can still access these services by calling them directly over the web.

HTTP Client Requirements

We highly recommend you utilize a fully-featured HTTP client library, such as Apache HttpClient for Java or Httplib2 for Python. It will make your life and your code a lot simpler.
In addition to supporting the usual HTTP POST, GET, PUT, etc., your client will need to have the capabilities listed below.

Handle cookies

Accept cookies and send their values to the correct domains.

Support HTTP redirects

Accept as a response HTTP code 303 and automatically GET the response from the specified location URL.

Support Basic Authentication

Client should be ready at any time to respond to Basic authentication credentials challenge via HTTP response code 401.

Support JSON marshalling and unmarshalling

The Data Manager will generate responses using JSON format and some operations require the client to PUT or POST JSON payloads.

Support for HTTPS

All communications to the Identity Server and the Data Manager service occur through secure HTTP.  We have obtained third-party certificates for our servers, so updating trust stores should not be necessary in most cases.

Common Request Headers

Every HTTP request to the Data Manager should include the Accept header with a preference for the media types application/json and text/plain .

If you get responses that look like html instead of json, check that you are correctly setting the Accept header.

After having logged in, the gs-token cookie should be included in all requests. More on this below.

In general it is desirable (but optional) for tools to add a header (or request parameter, or cookie, any of these forms can be used) gs-toolname=MyToolName to all their requests.  This is optional but identifies which tool is making a request which permits more detailed usage reporting within the GenomeSpace system.

URLs and API Version

In this release of the Data Manager API we will start requiring the API version to be included in all calls.
The API version is specified in the base URL for all requests, e.g.:
http://dm.genomespace.org/datamanager/v1.0/dataformat

Calls that don’t include the URL will return an error code with a message.

The one exception we have allowed for the use of the new URL format is to GET files and directories.
If you GET
https://dm.genomespace.org/datamanager/files/users/test/myFile.txt
DM will forward the request to a new URL that includes the API version in the URL and return the contents of the file or directory.

The purpose of this exception is to keep  GenomeSpace URLs that are visible to end-users (i.e. the file URLs) as clean as possible.

However, for all API calls we strongly recommend you include the API version in the URL.

Login and Authentication

You can do an explicit login at the beginning of a session by issuing a GET to:
https://identity.genomespace.org/identityServer/basic
You will get a HTTP 401 response code with a request for credentials using Basic Authentication. Basic authentication is described in RFC2617.
If you are utilizing an HTTP library (and we highly recommend that you do), it is very likely that it already includes support for Basic authentication. In this case, you would just need to set the credentials on the web client, and when confronted with a Basic authentication challenge, the web client would automatically respond.
On successful submission of credentials, the server’s response will include a cookie named gs-token. The cookie should be included in all subsequent requests to the Data Manager. If your library has cookie handling enabled, the inclusion of the cookie in requests should happen automatically.
All requests to the Data Manager are authenticated via the gs-token cookie. If a request does not include a valid gs-token cookie or the session has expired, the Data Manager will redirect the client (HTTP code 302) to the identity server and confronted with a Basic Authentication challenge as described above. Once authentication is completed, the client will be redirected back to the originally requested URL.

WADL

All the operations available from the Data Manager  are described in Web Application Description Language (WADL) specification. The WADL can be accessed at:
https://dm.genomespace.org/datamanager/application.wadl

GSFileMetadata, GSDataFormat and, GSDirectoryListing Objects

Information about files managed by Data Manager will be returned in JSON objects. GSFileMetadata objects will provide information about an individual file or directory (name, path, size,etc.). GSDirectoryListing carries a GSFileMetadata object for the corresponding directory and a list of GSFileMetadata with one entry for each file and subdirectory.
GSDataFormat carries information about the format of the files and the available format transformations. ACL contains permissions information on a given object. Appendix A,B,C and D of this document include samples of each.

Directories(new)

Whenever a GET is submitted on a URL that corresponds to a directory, the DataManager will respond with a GSDirectoryListing.

Each user name will have a default directory assigned and created to reach this default directory.

will redirect to the  default directory URL. Currently, the default directory for all users is at URL:

https://dm.genomespace.org/datamanager/v1.0/file/Home

To redirected to the  logged in user's personal directory, where they would have full read/write privileges:

Currently, this would redirect to a URL that would look like this:

https://dm.genomespace.org/datamanager/v1.0/users/theUserName

To create a directory:

where newdirname is the new directory name. Note that DataManager expects the parent directory to exist (in this case, /users/test). If directory creation is successful, the GSFileMetaData for the new directory will be returned.

Uploading files

GenomeSpace offers two different methods to upload files. The simpler one we will refer to as "Single PUT upload". It is easier to implement on the client, but it is limited to files of up to 5GB. For larger files you will have to do a "Multipart file upload". In this second method, GenomeSpace provides the client with the information necessary for it to do an upload using Amazon S3's multipart upload protocol.

Single PUT upload

Uploading a file is a two step process. You must first obtain from Data Manager a signed Amazon S3 URL. Then you, will PUT your file to the generated URL.

To obtain the signed URL, you will do a GET with a URL that looks like:
Note that we changed the path from /uploadurls to /uploadurl

/datamanager/v1.0/uploadurl/users/test/mydir/AnotherLittleFile.txt?Content-Length=6&Content-MD5=nwYkOry4nHDgwzHGHYcfpw%3D%3D&Content-Type=application/octet-stream

Note that the base URL (https://dmtest.genomespace.org/datamanager) is followed by /uploadurl and then by the destination path and file name (/users/test/mydir/AnotherLittleFile.txt)

The 3 query parameters included in the URL

In response, the Data Manager service will return an Amazon S3 URL that you will use to PUT the file.
The returned URL might look something like this (included here for illustration, but what it looks like is not important. Just use whatever was returned by the web service):
https://genomespace-input.s3.amazonaws.com/users/test/mydir/AnotherLittleFile.txt?AWSAccessKeyId=AKIAIDXKHSCMYX5BHNLA&Expires=1296076583&Signature=G4tZJCObhPsvcdZxKJkBY7%2Bq378%3D

With your file PUT you will need to include 4 HTTP headers:
Content-Length
Content-MD5 (should not be URL encoded)
Content-Type

x-amz-meta-md5-hash is no longer needed or used. Do not include or upload will fail

Amazon S3 will return an HTTP status code of 200 on the successful completion of the upload.

Multipart File Upload

If you have a file larger than 5GB, you will have to do a multipart upload. In addition to overcoming the size limit of the "Single PUT upload", the multipart upload will allow for faster uploads. The S3 multipart protocol requires splitting the file into smaller chunks that can be uploaded concurrently a multithreaded implementation.

The S3 REST APIs are complicated. We highly recommend using an existing S3 library instead of dealing with the APIs directly. Amazon offers SDKs for Java, Ruby, PHP, and .NET.

The first step is to request the information needed for the upload from GenomeSpace.

The part that follows /uploadinfo/ is the destination path of the file you will be uploading. In response, the Data Manager will return a JSON S3FileUploadInfo object. This JSON response includes temporary Amazon credentials and S3 specific details you will be using during the upload.

Downloading Files

To download the file, you will need either the absolute directory file path or you would have obtained the URL for the file already.
Both the absolute file path and url are available in the GSFileMetada object (properties named path and url respectively).

The URL to GET will look something like below (minus protocol, server,port number):
/datamanager/v1.0/file/users/test/mydir/AnotherLittleFile.txt

To avoid problems with special characters, you should URL encode each URL path element and the file name itself. (Do not just do a URL encode on the whole URL or you will end up with a URL that will not work).

The Data Manager will respond with a redirect to the Amazon S3 location of the file. If your HTTP client library has redirects enabled, the redirection should happen automatically.

File and Directory Ownership and Permissions

Ownership of files and directories is established by who is the owner of the top level directory. Any file and directory that is directly or indirectly under a users home directory is owned by that user. As such, the owner user is able to read, write, delete, and grant permissions on any object below.

Example:
/users/fred is home directory for a user named “fred”.
User fred is the owner of directory /users/fred as well as /users/fred/dir1 and /users/fred/dir1/myfile.txt because the directory and the file are below the home directory.

Data Manager supports sharing of files and directories through the use of access control lists (ACLs). The owner of a file or directory can grant read and write permissions to any other user or group (for more information on users and groups see blabha).

Each ACL points to the object that it is associated with and includes a list of access control entries (ACE) identifying the user or group (generically referred as a Security Identity (SID) and the permission that has been granted.

Read permission on a file means that its contents can be downloaded. Write permissions mean that the file can be either updated/replaced or deleted.

On a directory read permission mean that the grantee can list the contents of the directory. Write permission mean that they can upload new files and delete and update existing ones.

ACL grants are inherited down the directory structure. So if you grant read permissions on a parent directory to another user, that user will be able read every file and directory below it in the hierarchy.

Example:
I user fred grants “read” permissions on directory /users/fred/dir1 to user kathy, she will be able to get a listing of /users/fred/dir1 and every directory below dir1, like /users/fred/dir1/dir2  and /users/fred/dir1/dir2/dir3.
Kathy will also be able to download /users/fred/dir1/myfile.txt since the file inherited the read permission from its parent directory.

If fred goes ahead and also grants write permissions to kathy on /users/fred/dir1, then kathy will be able to create and delete new directories and files below /users/fred/dir1.

There are two ways to create an ACL. The first, PUT an ACL object:

The other method is to  POST the ACEs associated with the the ACL

Both methods will return the resulting ACL JSON object.

For our running examples, the URLs would look something like:
https://dm.genomespace.org/datamanager/v1.0/acl/file/users/fred/dir1
and
https://dm.genomespace.org/datamanager/v1.0/acl/file/users/fred/dir1/dir2

Once in existence, as you would expect, you can GET the ACL:

You can also get all the ACLs that affect a particular file object by adding  the “hierarchy” query parameter to the URL:

In this case, instead of getting a single JSON object you will get a JSON array of ACLs, one for each ACL found in the directory hierarchy for the file object.

To delete an ACL:

or if you want to just remove a specific access control entry on an ACL:

You will need the ACE id that is included in the ACL JSON serialization.

To get the effective permissions on any file object  you can examine the “effectiveACL” property included in every GSFileMetadata object to obtain the consolidated permissions on that object. That is, who has read and write permissions on the object, either because of ownership, permissions granted directly on that object and permissions that have been inherited through the directory structure.
To emphasize, the ACL object attached to GSFileMetadata object is not a “real” ACL object, but flattened view of the hierarchical permissions and is included as a convenience.

Downloading Files in Different Data Formats

Data Manager has the ability to convert some files into formats that can be consumed by other applications.
The data format that the Data Manager thinks original file is identified in the GSFileMetadata property dataFormat (could be empty if it does not recognize the format).
GSFileMetadata also specifies the property availableDataFormats. This will be an array of GSDataFormat objects that identify the formats this file can be requested in.

To GET the file in a specific format you will build a URL that looks as follows:

Remember to URL-encode the value for the dataformat parameter.

The query parameter dataformat value is the URL for the format. This URL can be obtained from the GSDataFormat url property in the availableDataFormats array.

Other File, Directory, and Format Operations

Delete a File or Directory

Note: To delete a directory, it needs to be empty

Copy a File or Directory

Note: The URL identifies the new object that will be created by the copy.
The source file or directory is identified in the custom x-gs-copy-source header . The header value should have look like /dir1/dir3/sourceFile  (the base URL nor the URL path segment /datamanager/v1.0/file should be included).

Convert a file on the server

Note: The URL identifies the new object that will be the source of the conversion. Note the dataformat query parameter. This is expected to be URL for the destination format. The possible conversions for any file on GenomeSpace can be obtained from examining that file objects GSFileMetadata object (see Obtain Metadata on a File or Directory). Make sure you URL-encode the dataformat parameter.
Also, the destination URL file name is expected to be consistent with the request data format e.g., if your source file is a GCT file and you are converting it to a GXP file, then the URL for the file needs to end with .gxp .

The source file or directory is identified in the custom x-gs-copy-source header . The header value should have look like /dir1/dir3/sourceFile  (the base URL nor the URL path segment /datamanager/v1.0/file should be included).

Obtain Metadata on a File or Directory

This will return a JSON GSFileMetadata object. See Appendix A and B.

To get HTTP headers on a file or directory

If you want to get the response HTTP headers for a file or directory without actually GETting the file object, you can use the HEAD verb call. Can be useful, for example, to get the size of a file from the content-length header if for some reason you want to avoid looking at the GSFileMetadata object.

List all Data Formats

Will return an array of GSDataFormat objects (see Appendix C).

{
 "name":"twoRecords.gct",
 "url":"https:\/\/dev.broadinstitute.org\/datamanager\/file\/users\/test\/twoRecords.gct",
"parentUrl":"https:\/\/dev.broadinstitute.org\/datamanager\/v1.0\/file\/users\/test",
 "path":"\/users\/test\/twoRecords.gct",
 "owner":{
   "id":"test",
   "name":"test",
   "type":"User"
 },
 "size":225,
 "lastModified":"2011-04-20T11:53:00-04:00",
 "isDirectory":false,
 "isLink":false,
 "dataFormat":{
   "name":"gct",
   "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/gct\/0.0.0",
   "fileExtension":"gct"
 },
 "availableDataFormats":[
   {
     "name":"gxp",
     "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/gxp\/0.0.0",
     "fileExtension":"gxp"
   },
   {
     "name":"genomicatab",
     "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/genomicatab\/0.0.0",
     "fileExtension":"tab"
   },
   {
     "name":"gct",
     "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/gct\/0.0.0",
     "fileExtension":"gct"
   }
 ],
 "effectiveAcl":{
   "object":{
     "objectId":"\/users\/test\/twoRecords.gct",
     "objectType":"DataManagerFileObject"
   },
   "accessControlEntries":[
     {
       "sid":{
         "id":"test",
         "name":"test",
         "type":"User"
       },
       "permission":"R"
     },
     {
       "sid":{
         "id":"test",
         "name":"test",
         "type":"User"
       },
       "permission":"W"
     }
   ]
 }
}

{
 "directory":{
    "name":"testDir1",
    "url":"https:\/\/dev.broadinstitute.org\/datamanager\/file\/users\/test\/testDir1",
    "parentUrl":"https:\/\/dev.broadinstitute.org\/datamanager\/v1.0\/file\/users\/test",
    "path":"\/users\/test\/testDir1",
    "owner":{
       "id":"test",
       "name":"test",
       "type":"User"
    },
    "size":0,
    "isDirectory":true,
    "isLink":false,
    "effectiveAcl":{
       "object":{
          "objectId":"\/users\/test\/testDir1",
          "objectType":"DataManagerFileObject"
       },
       "accessControlEntries":[
          {
             "sid":{
                "id":"test",
                "name":"test",
                "type":"User"
             },
             "permission":"R"
          },
          {
             "sid":{
                "id":"test",
                "name":"test",
                "type":"User"
             },
             "permission":"W"
          }
       ]
    }
 },
 "contents":[
    {
       "name":"ASillyLittleFile.txt",
       "url":"https:\/\/dev.broadinstitute.org\/datamanager\/file\/users\/test\/testDir1\/ASillyLittleFile.txt",
       "path":"\/users\/test\/testDir1\/ASillyLittleFile.txt",
       "owner":{
          "id":"test",
          "name":"test",
          "type":"User"
       },
       "size":6,
       "lastModified":"2011-10-18T12:37:31-04:00",
       "isDirectory":false,
       "isLink":false,
       "dataFormat":{
          "name":"txt",
          "description":"Plain text format",
          "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/txt\/0.0.0",
          "fileExtension":"txt"
       },
       "availableDataFormats":[
          {
             "name":"uppercasetxt",
             "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/uppercasetxt\/0.0.0",
             "fileExtension":"uppertxt"
          },
          {
             "name":"lowercasetxt",
             "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/lowercasetxt\/0.0.0",
             "fileExtension":"lowertxt"
          },
          {
             "name":"txt",
             "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/txt\/0.0.0",
             "fileExtension":"txt"
          }
       ],
       "effectiveAcl":{
          "object":{
             "objectId":"\/users\/test\/testDir1\/ASillyLittleFile.txt",
             "objectType":"DataManagerFileObject"
          },
          "accessControlEntries":[
             {
                "sid":{
                   "id":"test",
                   "name":"test",
                   "type":"User"
                },
                "permission":"R"
             },
             {
                "sid":{
                   "id":"test",
                   "name":"test",
                   "type":"User"
                },
                "permission":"W"
             }
          ]
       }
    },
    {
       "name":"aLittleSubdir",
       "url":"https:\/\/dev.broadinstitute.org\/datamanager\/file\/users\/test\/testDir1\/aLittleSubdir",
       "path":"\/users\/test\/testDir1\/aLittleSubdir",
       "owner":{
          "id":"test",
          "name":"test",
          "type":"User"
       },
       "size":0,
       "isDirectory":true,
       "isLink":false,
       "effectiveAcl":{
          "object":{
             "objectId":"\/users\/test\/testDir1\/aLittleSubdir",
             "objectType":"DataManagerFileObject"
          },
          "accessControlEntries":[
             {
                "sid":{
                   "id":"test",
                   "name":"test",
                   "type":"User"
                },
                "permission":"R"
             },
             {
                "sid":{
                   "id":"test",
                   "name":"test",
                   "type":"User"
                },
                "permission":"W"
             }
          ]
       }
    }
 ]
}

{	
    "name":"txt",
    "description":"Plain text format",
    "url":"http:\/\/www.genomespace.org\/datamanager\/dataformat\/txt\/0.0.0",
    "fileExtension":"txt"
}

{
 "id":"03b00579-5356-414c-afc7-3432ece90029",
 "object":{
   "objectId":"\/users\/test\/dir1\/AnotherLittleFile.txt",
   "objectType":"DataManagerFileObject"
 },
 "accessControlEntries":[
   {
     "id":"9b02e70a-f25b-4951-90ae-6759517be31b",
     "sid":{
       "id":"test2",
       "name":"test2",
       "type":"User"
     },
     "permission":"R"
   },
   {
     "id":"df603747-8c5c-4e80-b187-c7e912bc39a6",
     "sid":{
       "id":"86d3ff94-7ada-44c5-bec2-afdbd7be54b4",
       "name":"DmClientTestGroup",
       "type":"Group"
     },
     "permission":"W"
   }
 ]
}

Appendix G: S3FileUploadInfo JSON object(new)

Note that version has been removed

Example:

{
    "path":"/users/test/clientTest1/AFileNamedDummy.txt", 
    "s3BucketName":"genomespace-dev", 
    "s3ObjectKey":"users/test/clientTest1/AFileNamedDummy.txt", 
    "genomeSpaceFileUrl":"https://dmdev.genomespace.org:8444/datamanager/file/users/test/clientTest1/AFileNamedDummy.txt", 
    "s3FileUrl":"https://genomespace-dev.s3.amazonaws.com/users/test/clientTest1/AFileNamedDummy.txt", 
    "amazonCredentials": {
        "secretKey":"1Gr3q/VAJRJFQTU3MzsTHnFH1Px1wz4gLjyF2ncQ", 
        "accessKey":"ASIAJXISRHBODWMCUCXQ", 
        "sessionToken":"AQoDYXdzEPNtiDYmMT6BA=="}
    }