Version 22 (modified by 12 years ago) (diff) | ,
---|
-
GENI Storage and Archive Service (GSAS)
- Participating Projects and Organizations
- Technical Contacts
- GENI Policy for Sharing Research Results
- 1) Goals
- 2) Architecture and Design
- 3) GENI ObjectDescriptor Schema
- 4) DataCite Schema
- 5) iRODS Design
- 6) iRODS Deployment
- 7) Access to GSAS from UserWorkspace
- 8) Access to GSAS from GEMINI I&M Tools
- 9) Access to GSAS from GIMI I&M Tools
- 10) Acceptance Tests
- 11) Tutorials and Experimenter Support
- A) Technical References
GENI Storage and Archive Service (GSAS)
Participating Projects and Organizations
Project 1663: Digital Object Registry (CNRI)
Project 1856: GIMI I&M Tools (UMass Amherst, RENCI and others)
Technical Contacts
Giridhar Manepalli (CNRI) (mailto:gmanepalli@cnri.reston.va.us / 703 620 8990)
Shu Huang (RENCI) (mailto:shuang@renci.org)
GPO System Engineer: Harry Mussman
GPO Software Engineer: Jeanne Ohren
GENI Policy for Sharing Research Results
GENI is developing a policy for sharing research results with the community that follows evolving practice. The GSAS will provide the mechanisms necessary to implement the policy.
Current overview:
Versions:
Open issues:
Next steps:
1) Goals
For GSAS:
- A structured place to store all of the objects (artifacts) for an experiment, with descriptors (metadata), that is easy to access, with short to medium term storage, and the ability to search. (Note: this goes well beyond just measurement data objects.)
- A separate long-term archive, with controlled access from the outside world, using a DOI (handle) as a persistent identifier
- Include most of the functionality provided by the Measurement Data Archive (MDA) prototype, built by CNRI.
- Establish multiple federated iRODS services, starting at RENCI and UMass Amherst, and operate for GENI users (experimenters).
- Establish persistent accounts for each user, and use icommands to store and retrieve objects (artifacts) for each user in the storage service.
- Establish authentication for each user based on username/password, certificates, and also proxy (delegated) certificates.
- Establish a directory structure in the storage service for each user to accommodate multiple experiments, and a directory structure for each experiment (consider "bag") to include all objects (artifacts) associated with that experiment, including one or more descriptors (metadata) within XML files (following the GENI descriptor schema).
- Provide multiple interfaces (including icommand and web) to allow an authenticated user to view, search and curate their objects (artifacts).
- Provide interface to allow a user to define an object (artifact) to be archived (where the object (artifact) may range from a large directory to a single file), include a descriptor (following the GENI ObjectDescriptor Schema), assign a persistent Digital Object Identifier (DOI, or "handle"), and decide when to push it to archive service.
- Establish an archive service that provides long-term and reliable storage, with public access via a DOI from the global handle service.
- Include a search function in the archive service, so that an outside user can search for and then retrieve an object, but allow the object’s owner to disable search, so that an outside user needs the DOI of the object to retrieve it.
Goals for GENI ObjectDescriptor Schema
- Useful for all types of objects, not just MeasurementData objects.
- Keep it simple, with the minimum number of mandatory fields.
- Where possible, values for fields should be automatically generated by Experiment Management Tools.
Use of “DataCite Schema for the Publication and Citation of Research Data”:
- When an object (artifact) is archived in the Archive Service with public access from the outside world via the Internet, using a DOI (handle) as a persistent identifier, and include descriptors (metadata) that follows DataCite Schema (ref)
2) Architecture and Design
An overview of the architecture and design of the GSAS is contained in this document:
GENI Storage and Archive Service: Storing and Archiving Experiment Objects
1) Goals
2) Configuration
3) Use Cases
4) Structure of Directories and Files in the GSAS
5) Access to the Structure in GSAS
6) Adding Descriptors
7) Searching Structure in GSAS
8) Creating a Bag and a .tar File
9) Archiving an Object
10) Overview of v1.x GENI ObjectDescriptor Schemas
This document is based on the early Measurement Data Archive (MDA) service prototype developed by CNRI, and many discussions within the GENI I&M community.
Versions:
Task list:
ID | Description | Who | Due | Status | Demos | Notes |
1 | "Storing and Archiving Experiment Objects" document | |||||
1a | Review v1.1 | Group | 4/29/13 | completed | ||
1b | Issue v1.2 | Harry | 5/1/13 | completed | ||
1c | Review with GPO | Harry/Jeanne | ? | |||
1d | Review with GEMINI | Harry/Jeanne | ? | |||
1e | Review with GIMI | Harry/Jeanne | ? | |||
1f | ||||||
2 | Resolve design issues | |||||
2a | Issue 2.1: How are persistent accounts established for each user in iRODSs? | Shu, Jeanne | ||||
2b | Issue 2.2: How are storage capacity limits established and enforced for each IRODS user? Are older objects (artifacts) flagged for removal? | Shu | ||||
2c | Issue 2.3: How are archive capacity limits established and enforced for each IRODS user? Are older objects (artifacts) flagged for removal? | Shu | ||||
2d | Issue 5.1: Where is the proxy certificate created? How is the proxy certificate transferred to the service? | Ezra | ||||
2e | Issue 5.2: What happens if the proxy certificate expires? Is the user notified? How can they load an updated proxy certificate? | Ezra | ||||
2f | Issue 5.3: How is the target information transferred to the service? | Ezra | ||||
2g | Issue 5.4: How is the iticket transferred to the service? | Cong | ||||
2h | Issue 5.5: What happens if the iticket certificate expires? Is the user notified? How can they load an updated proxy certificate? | Cong | ||||
2i | Issue 5.6: How is all of this target information transferred to the service agent? | Cong | ||||
2j | Issue 6.1: Need to establish rules if there is a discrepancy in descriptors. | Shu | ||||
2k | Issue 6.2: Need to establish rules for changing or removing metadata.xml files. | Shu | ||||
2l | Issue 7.1: When using a browser in the Experiment Management Environment (or elsewhere) to view artifacts (files and directories) in the GSAS, how will the associated descriptors (metadata) will be displayed? | Shu | ||||
2m | Issue 8.1: After the bag and .tar file have been created and used, is there some cleanup that should be done? | Shu | ||||
2n | Issue 8.2: After changes have been made to directories and files, what is the process for recreating the bag and .tar file? | Shu | ||||
2o | Issue 10.1: Is there a way to derive the descriptors in an archive.xml file from descriptors in the other types of metadata.xml files, or at least an initial set of descriptors for the archive.xml file? | Giridhar | ||||
3 | Establish GENI policy on sharing research results | |||||
3a | DRAFT policy ( 2 pages) | Larry/Giridhar | 6/1/13 | |||
3b | Review with GPO | Larry/Giridhar | 6/1/13 | |||
3c | Review with community | Larry/Giridhar | GEC17 | |||
1 | ||||||
1 |
3) GENI ObjectDescriptor Schema
Schema:
Examples of metadata.xml files:
ArchiveofProject
ArchiveofExperiment
ArchiveofStep
ArchiveofArtifact
Versions:
Open issues:
Next steps:
4) DataCite Schema
When GENI research results are shared with the research community, they will use the DataCite metadata schema, which has been established for the research community.
References:
5) iRODS Design
Design:
iRODS fact sheet
iRODS overview
iRODS configuration
iRODS use cases
Plan:
Open issues:
Next steps:
6) iRODS Deployment
Design:
Versions:
Open issues:
Next steps:
7) Access to GSAS from UserWorkspace
Design:
Versions:
Open issues:
Next steps:
8) Access to GSAS from GEMINI I&M Tools
Design:
Versions:
Open issues:
Next steps:
9) Access to GSAS from GIMI I&M Tools
Design:
Versions:
Open issues:
Next steps:
10) Acceptance Tests
11) Tutorials and Experimenter Support
8) Key task list:
ID | Description | Who | Due | Status | Demos | Notes |
1 | ||||||
1 | ||||||
1 | ||||||
1 | ||||||
1 | ||||||
1 |
A) Technical References
iRODS fact sheet
iRODS overview
iRODS configuration
iRODS use cases
iRODS plan
I&M Tools: Basic Test/Tutorial/Experiment Storyboard
I&M Tools: Basic Test/Tutorial/Experiment Workflow
Attachments (39)
- 042613 v1.1_GENIObjectDescriptor Schema.docx (380.3 KB) - added by 12 years ago.
- GENIObject.xsd (9.2 KB) - added by 12 years ago.
- project1.xml (1.1 KB) - added by 12 years ago.
- Experiment1.xml (998 bytes) - added by 12 years ago.
- projectSerialized.xml (4.1 KB) - added by 12 years ago.
- Experiment1.2.xml (998 bytes) - added by 12 years ago.
- Artifact1.xml (900 bytes) - added by 12 years ago.
- ArchiveOfProject.xml (5.3 KB) - added by 12 years ago.
- ArchiveOfExperiment.xml (3.8 KB) - added by 12 years ago.
- ArchiveOfExperiment.2.xml (3.8 KB) - added by 12 years ago.
- ArchiveOfArtifact.xml (1.9 KB) - added by 12 years ago.
- Step1.xml (610 bytes) - added by 12 years ago.
- DataCite-MetadataKernel_v2.2.pdf (303.7 KB) - added by 12 years ago.
- DataCite-metadata.xsd (13.8 KB) - added by 12 years ago.
- gimi-irods-4-15-13b.docx (19.0 KB) - added by 12 years ago.
- 042913 v1.2_GENIObjectDescriptor Schema.docx (375.1 KB) - added by 12 years ago.
- 082112b_MDASrvc_Figures.vsd (3.2 MB) - added by 12 years ago.
- 091012d_GIMI_Figures.vsd (2.7 MB) - added by 12 years ago.
- 040313_GEMINI_Figures.vsd (2.3 MB) - added by 12 years ago.
- 042913_iRODS_Figures.vsd (1.4 MB) - added by 12 years ago.
- Visio-042913_iRODS_Figures.pdf (20.3 KB) - added by 12 years ago.
- 042913b v1.2_GENIObjectDescriptor Schema.docx (375.4 KB) - added by 12 years ago.
- gimi-irods-5-1-13b.docx (17.8 KB) - added by 12 years ago.
- Visio-050713_iRODS_Figures.pdf (20.1 KB) - added by 12 years ago.
- 050713_iRODS_Figures.vsd (1.4 MB) - added by 12 years ago.
- 051613_iRODS_Figures.vsd (1.4 MB) - added by 12 years ago.
- Visio-051613_iRODS_Figures.pdf (24.4 KB) - added by 12 years ago.
- 051613_GIMI_Figures.vsd (1.8 MB) - added by 12 years ago.
- Visio-051613_GIMI_Figures.pdf (137.8 KB) - added by 12 years ago.
- gimi-irods-5-9-13b.docx (16.7 KB) - added by 12 years ago.
- 042913c v1.2_GENIObjectDescriptor Schema.docx (376.1 KB) - added by 12 years ago.
- 060613 v1.3_GENIObjectDescriptor Schema.docx (386.1 KB) - added by 12 years ago.
- 060613schemaonly v1.3_GENIObjectDescriptor Schema.docx (29.4 KB) - added by 12 years ago.
- GENIObject-v1.3.xsd (13.9 KB) - added by 11 years ago.
- Artifact1.3.xml (1.1 KB) - added by 11 years ago.
- Experiment1.3.xml (925 bytes) - added by 11 years ago.
- Step1.3.xml (432 bytes) - added by 11 years ago.
- Project1.3.xml (913 bytes) - added by 11 years ago.
- ProjectSerialized1.3.xml (3.8 KB) - added by 11 years ago.