GENI Storage and Archive Service (GSAS)

Participating Projects and Organizations

​Project 1663: Digital Object Registry (CNRI)

​Project 1856: GIMI I&M Tools (UMass Amherst, RENCI and others)

Technical Contacts

Giridhar Manepalli (CNRI) (​mailto:gmanepalli@cnri.reston.va.us / 703 620 8990)

Shu Huang (RENCI) (​mailto:shuang@renci.org)

GPO System Engineer: ​Harry Mussman

GPO Software Engineer: ​Jeanne Ohren

1) GENI Policy for Sharing Research Results

CNRI is drafting a GENI policy for sharing research results with the community, that follows evolving community and NSF practice.

The GSAS will provide the mechanisms necessary to implement the policy.

Task list:

ID	Description	Who	Due	Status	Demos	Notes
1	Establish GENI policy on sharing research results
1a	DRAFT policy ( 2 pages)	Larry/Giridhar	5/15/13
1b	Review with GPO	Larry/Giridhar	6/1/13
1c	Review with community	Larry/Giridhar	GEC17
1d
1e

2) Goals for GENI Storage and Archive Service (GSAS)

Goals for GSAS:

1. A structured place to store all of the objects (artifacts) for an experiment, with descriptors (metadata), that is easy to access, with short to medium term storage, and the ability to search. (Note: this goes well beyond just measurement data objects.)
   

2. A separate long-term archive, with controlled access from the outside world, using a DOI (handle) as a persistent identifier
   

3. Include most of the functionality provided by the Measurement Data Archive (MDA) prototype, built by CNRI.
   

4. Establish multiple federated iRODS services, starting at RENCI and UMass Amherst, and operate for GENI users (experimenters).

5. Establish persistent accounts for each user, and use icommands to store and retrieve objects (artifacts) for each user in the storage service.

6. Establish authentication for each user based on username/password, certificates, and also proxy (delegated) certificates.
   

7. Establish a directory structure in the storage service for each user to accommodate multiple experiments, and a directory structure for each experiment (consider "bag") to include all objects (artifacts) associated with that experiment, including one or more descriptors (metadata) within XML files (following the GENI descriptor schema).
   

8. Provide multiple interfaces (including icommand and web) to allow an authenticated user to view, search and curate their objects (artifacts).
   

9. Provide interface to allow a user to define an object (artifact) to be archived (where the object (artifact) may range from a large directory to a single file), include a descriptor (following the GENI ObjectDescriptor Schema), assign a persistent Digital Object Identifier (DOI, or "handle"), and decide when to push it to archive service.
   

10. Establish an archive service that provides long-term and reliable storage, with public access via a DOI from the global handle service.
    

11. Include a search function in the archive service, so that an outside user can search for and then retrieve an object, but allow the object’s owner to disable search, so that an outside user needs the DOI of the object to retrieve it.
    

Goals for GENI ObjectDescriptor Schema:

11. Useful for all types of objects, not just MeasurementData objects.
    

12. Keep it simple, with the minimum number of mandatory fields.
    

13. Where possible, values for fields should be automatically generated by Experiment Management Tools.
    

Use of “DataCite Schema for the Publication and Citation of Research Data”:

14. When an object (artifact) is archived in the Archive Service with public access from the outside world via the Internet, using a DOI (handle) as a persistent identifier, and include descriptors (metadata) that follows DataCite Schema (ref)
    

2) Overview of the GENI Storage and Archive Service (GSAS)

Document:

An overview of the structure and use of the GSAS is contained in this document:

GENI Storage and Archive Service: Storing and Archiving Experiment Objects

1) Goals
2) Configuration
3) Use Cases
4) Structure of Directories and Files in the GSAS
5) Access to the Structure in GSAS
6) Adding Descriptors
7) Searching Structure in GSAS
8) Creating a Bag and a .tar File
9) Archiving an Object
10) Overview of v1.x GENI ObjectDescriptor Schemas

This document is based on the early Measurement Data Archive (MDA) service prototype developed by CNRI, and many discussions within the GENI I&M community.

Versions of document:

​042613 v1.1 GENI ObjectDescriptor Schema

​042913 v1.2 GENI ObjectDescriptor Schema

Task list:

ID	Description	Who	Due	Status	Demos	Notes
2	"Storing and Archiving Experiment Objects" document
2a	Review v1.1	Group	4/29/13	completed
2b	Draft v1.2	Harry	5/1/13	completed
2c	Review with GPO	Harry/Jeanne	?
2d	Review with GEMINI	Harry/Jeanne	?
2e	Review with GIMI	Harry/Jeanne	?
2f
3	Resolve design issues
3a	Issue 2.1: How are persistent accounts established for each user in iRODSs?	Shu, Jeanne
3b	Issue 2.2: How are storage capacity limits established and enforced for each IRODS user? Are older objects (artifacts) flagged for removal?	Shu
3c	Issue 2.3: How are archive capacity limits established and enforced for each IRODS user? Are older objects (artifacts) flagged for removal?	Shu
3d	Issue 5.1: Where is the proxy certificate created? How is the proxy certificate transferred to the service?	Ezra
3e	Issue 5.2: What happens if the proxy certificate expires? Is the user notified? How can they load an updated proxy certificate?	Ezra
3f	Issue 5.3: How is the target information transferred to the service?	Ezra
3g	Issue 5.4: How is the iticket transferred to the service?	Cong
3h	Issue 5.5: What happens if the iticket certificate expires? Is the user notified? How can they load an updated proxy certificate?	Cong
3i	Issue 5.6: How is all of this target information transferred to the service agent?	Cong
3j	Issue 6.1: Need to establish rules if there is a discrepancy in descriptors.	Shu
3k	Issue 6.2: Need to establish rules for changing or removing metadata.xml files.	Shu
3l	Issue 7.1: When using a browser in the Experiment Management Environment (or elsewhere) to view artifacts (files and directories) in the GSAS, how will the associated descriptors (metadata) will be displayed?	Shu
3m	Issue 8.1: After the bag and .tar file have been created and used, is there some cleanup that should be done?	Shu
3n	Issue 8.2: After changes have been made to directories and files, what is the process for recreating the bag and .tar file?	Shu
3o	Issue 10.1: Is there a way to derive the descriptors in an archive.xml file from descriptors in the other types of metadata.xml files, or at least an initial set of descriptors for the archive.xml file?	Giridhar
3p
3q

3) GENI ObjectDescriptor Schema

CNRI is developing a comprehensive GENI ObjectDescriptor Schema, that fully implements the schema outlined in the GSAS overview document.

Versions of .xsd schema:

​v1.1 GENIObject

Examples of metadata.xml files:

​Project1
​Experiment1
​ProjectSerialized
​Step1
​Artifact1

​ArchiveofProject
​ArchiveofExperiment
​ArchiveofStep
​ArchiveofArtifact

Task list:

ID	Description	Who	Due	Status	Demos	Notes
4	"GENI ObjectDescriptor Schema"
4a	Review v1.1	Group	4/29/13	completed
4b	Issue v1.2	Giridhar	6/1/13			Need to include controlled vocabularies
4c	Review with group	Giridhar	?
4d	Review with GPO	Giridhar	?
4e	Review with community	Giridhar	GEC17
4f

4) DataCite Schema

When GENI research results are shared with the research community, they will use the DataCite metadata schema, which has been established for the research community.

References:

​document
​metadata example

5) iRODS Design and Deployment

As part of the GIMI project, RENCI is developing and deploying iRODS, with special extensions to implement the GSAS, as specified in the overview document.

References:

​iRODS fact sheet
​iRODS overview
​iRODS configuration
​iRODS use cases

Plan:

​011513 iRODS plan
​041513 addendum to the iRODS plan

Task list:

ID	Description	Who	Due	Status	Demos	Notes
5	"iRODS"
5a	Establish multiple federated iRODS services, starting at RENCI and UMass Amherst, and operate for GENI users (experimenters)	Shu and Cong	GEC14	completed
5b	Establish persistent accounts for each user, use icommands to store and retrieve measurement data objects for each user in a storage service	Shu and Cong	GEC14	completed
5c	Establish authentication for each user based on certificates, and also proxy (delegated) certificates.	Shu and Jeanne	GEC15	completed
5d	Establish directory structure in storage service for each user to accommodate multiple experiments, and directory structure for each experiment (consider "bag") to include all objects (artifacts) associated with that experiment, including one or more descriptors (metadata) within XML files (following the GENI descriptor schema)	Shu, Harry and Giridhar	GEC15	in progress
5e	Provide multiple interfaces (including icommand, restful http, and web) to allow authenticated user to view, search and curate their objects (artifacts)	Shu	GEC15	in progress		restful http can come later
5f	Provide interface to allow user to define an object to be archived (where the object may range from a large directory to a single file), include a descriptor (following the GENI descriptor schema), assign a persistent Digital Object Identifier (DOI, or "handle"), and decide when to push it to archive service.	Shu	GEC16	in progress
5g	Establish an archive service that provides long-term and reliable storage, with access via a DOI from the global handle service, with access following a local policy included in archived object. (Include at least two policies: give the object to anyone, or give the object only to its owner.)	Shu	GEC17	in progress
5h	"iRODS"	Shu
5i	"iRODS"	Shu
5j	"iRODS"	Shu
5k	"iRODS"	Shu
5l	"iRODS"	Shu
5m	"iRODS"	Shu

6) Access to GSAS from UserWorkspace using Experiment Management Tools

As explained in the overview of the GSAS, an experimenter (user) can access the GSAS from their UserWorkspace using Experiment Management Tools

An "Experiment Artifact Management Tool" is required that includes:

• iclient to access the GSAS
• ability to manage the experimenter's artifacts, and push them to the GSAS
• ability to formulate descriptors, put them into metadata.xml files, and push them to the GSAS

Who is going to prototype such a tool? when will it be available?

Versions:

Task list:

ID	Description	Who	Due	Status	Demos	Notes
6	"Experiment Artifact Management Tool"
6a
6b
6c
6d

7) Access to GSAS from GEMINI I&M Tools

As explained in the overview of the GSAS, a service acting on behalf of an experimenter (user) can access the GSAS using an agent holding a proxy certificate. This is the arrangement that is being used within the GEMINI I&M tools.

An early version of the agent is available, but it must now be extended to push all measurement data objects, and associated metadata, as specified in the overview of the GSAS. The issues identified in the overview must be resolved.

Versions:

Task list:

ID	Description	Who	Due	Status	Demos	Notes
7	"GEMINI agent to access GSAS"
7a		Ezra
7b
7c
7d

8) Access to GSAS from GIMI I&M Tools

As explained in the overview of the GSAS, a service acting on behalf of an experimenter (user) can access the GSAS using an agent holding a an iticket. This is the arrangement that is being used within the GIMI I&M tools.

An prototype version of the agent must be built, to push all measurement data objects, and associated metadata, as specified in the overview of the GSAS. The issues identified in the overview must be resolved.

Who will do this? Cong?

Versions:

Task list:

ID	Description	Who	Due	Status	Demos	Notes
8	"GIMI agent to access GSAS"
8a		Cong
8b
8c
8d

9) Acceptance Tests

Integrated acceptance tests must be done between all tools.agents and the GSAS.

Jeanne will coordinate.

Task list:

ID	Description	Who	Due	Status	Demos	Notes
9	"Integration of tools/agents with GSAS"
9a		Cong
9b
9c
9d

10) Tutorials and Experimenter Support

A) Technical References

I&M Tools: Basic Test/Tutorial/Experiment Storyboard
I&M Tools: Basic Test/Tutorial/Experiment Workflow