wiki:sol4/IDMS/ExpTutorial

Version 9 (modified by adorwat@umail.iu.edu, 5 years ago) (diff)

--

IDMS Experiment in GENI

The persistent IDMS experiment allows other users to attach at shared VLAN sites, granting access to the available storage resources. Users may then upload and download their experiment data using a set of client tools. IDMS provides:

  • A custom RSpec that configures a node to access the IDMS experiment
  • Nodes use DHCP to lease an address within the IDMS data plane subnet
  • Automatically installed client tools to upload/download data to available storage nodes

Base Experiment Topology

Shared VLANs at Stanford, Utah, Illinois, ig BBN and eg BBN

  • Stitching between the shared VLAN sites to create a cross-country data plane
  • Phoebus Gateways on ProtoGENI backbone nodes for external access to the experiment and to improve WAN transfer performance
  • DHCP server at illinois-ig - 10.10.150.0
Layer2 example topology

Prerequisite

In order to attach external user to base experiment. You need to reserve resources. You can use either Flack or Omni to do that. If you haven't used any of these tools before, please go through the introductory tutorials first.

Step 1: Create Slice

Create the slice in the project. Name the slice something like xxxxIdmsUser.

     omni createslice <slice_name> -r <project_name>

Step 2: Reserve Resources

  1. Download the rspec . For more information on rspec refer Inside rspec

  1. Base experiment can only be joined at shared VLAN nodes. Currently shared VLAN nodes are at
    • stanford-ig
    • utah-ig
    • illinois-ig
    • ig-bbn
  1. Select aggregate manager from list above.
       omni createsliver -a <aggregate_manager> <slice_name> <path_to_rspec_file> -r <project_name>
    
  1. Poll the resource status to make sure overall 'geni_status' is 'ready'
       omni sliverstatus -a <aggregate_manager> <slice_name> -r <project_name>
    
  1. If 'geni_status' is not 'ready' try using another aggregate manager

Step 3: login to Reserved Host

  1. Login to reserved host. If you are unaware of this, please follow instruction here
  1. Test configuration : DHCP server allocate ip to each host connected to base experiment. To ensure your host is connected to idms base experiment data plane, ping the DHCP server.
        ping 10.10.150.0
    
    To test connectivity with lbone server. telnet to lbone server on 6767 port. If communication is successful, you will get telnet prompt. Just hit Enter it will vanish.
       telnet monitor.incntre.iu.edu 6767
    
  1. To download / upload data Lors tool is already installed. Configure the Lors tool using following command. Make sure .xnd file is created in your 'Home' directory.
        lors_setup.sh
    

Step 4: Upload Data

  1. To upload data use lors_upload command.
       lors_upload –H monitor.incntre.iu.edu -f <file_to_be_uploaded> <options>
    
    .xnd file will be generated when you upload the file using lors_upload. This exnode file contains metadata about how your file has been distributed across available depots. If you wish to share you data across other host connected on IDMS data plane. Just share the .xnd file. More advanced options available to expore
   lors_upload --help
   Usage: lors_upload <filename>
     -f, --samename                 Specify the exNode filename by appending .xnd to the original filename.
     -i, --stdin                    Read from standard input rather than a specified filename.
     -o, --outputfile=STRING        Specify a specific name for the output exNode file.
     -v, --version                  Display Version information about this package.
     -V, --verbose=0                Modify the level or mode of Verbosity in output messages.
     -H, --lbone-host=STRING        Specify an L-Bone Server for resource discover and proxmity resolution.
     -P, --lbone-port=6767          Specify the port number when using an L-Bone Server on a non standard port.
     -l, --location="state= TN"     Specify a location hint to pass the L-Bone Query.
     -d, --duration=STRING          Specify the amount of time to allocate storage. The available modifiers are 'm' 'h' 'd' for 'minute' 'hour' and 'days' respectively. Default is seconds. (e.g. -d 1d or -d
                                 1h)
     -s, --soft                     Specify SOFT storage for storage allocations.
     -h, --hard                     Specify HARD storage for storage allocations.
     -m, --maxdepots=6              Specify the maximum number of depots returned by the 'location' hint.
     -b, -bs='5M'                   Specify the logical data blocksize of input file.
     -c, --copies=1                 Specify the number of copies.
     -F, --fragments=4              Rather than specifying the logical data blocksize, specify the number of blocks into which the file should be divided.
    --depot-list                   Only use depots specified in .xndrc file
     -t, --threads=INT              Specify the maximum number of threads to use to perform Upload.
     -T, --timeout='3m'             Specify the maximum time to allow for all operations.
     -M, --buffersize=STRING        Specify the maximum internal buffer to use on operations.
     -O, --offset=STRING            Specify an offset relative to the exNode from which download will begin
     -L, --length=STRING            Specify a length other than the logical extent of the exNode.
     -n, --none                     Turn off all e2e services.
     -e, --des                      Turn on des encryption.
     -a, --aes                      Turn on aes encryption.
     -z, --compress                 Turn on zlib compression.
     -x, --xor                      Turn on xor encryption.
     -k, --checksum                 Turn on checksum.
     -E, --e2e-blocksize=STRING     When specifying e2e conditioning, select an e2e bocksize which will evenly fit into your chosen Mapping Blocksize.
     -X, --xndrc=<filename>         You may specify an alternate source for the upload DepotList.
     -D, --demo                     Add the messages understood by the LoRS View visualization.
     --lodn-allocate[=STRING]       Turn on LoDN proxy allocation. The STRING value contains authentication credentials separated by an &.
     -n, --normal-allocate          Override LoDN proxy allocation if specified in the xndrc file.

   Help options:
     -?, --help                     Show this help message
     --usage                        Display brief usage message

Step 5: Download Data

To download data stashed on storage node use .xnd file generated during the upload.

  lors_download -f <xnd_file_location> -o <file_name>

More advanced options to explore

   lors_download --help
   Usage: lors_download <filename>
     -f, --samename              Specify the exNode filename by appending .xnd to the original filename.
     -o, --outputfile=STRING     Specify a specific name for the output exNode file.
     -v, --version               Display Version information about this package.
     -V, --verbose=0             Modify the level or mode of Verbosity in output messages.
     -a, --thread-depot=-1       Specify the maxmium connection to any single IBP depot.
     -r, --redundancy=1          Specify the maxmium number of threads working on a single download job.
     -R, --resume                Download will resume working on a partially downloaded file if one is detected.
     -p, --progress=0            The progress number specifies how many completed blocks are allowed before a download of a pending block is duplicated.
     -b, -bs='5M'                Specify the logical blocksize of transfers.
     -t, --threads=INT           Specify the maximum number of threads to use to perform Download.
     -T, --timeout='3m'          Specify the maximum time to allow for all operations.
     -q, --prebuffer=INT         Specify the number of blocks to prebuffer before outputting to file.
     -C, --cache=INT             Specify the number of block-sized memory buffers to use during download.
     -M, --buffersize=STRING     Specify the maximum internal buffer size to use on operations.
     -O, --offset=LONG           Specify an offset relative to the exNode from which download will begin
     -L, --length=LONG           Specify a length other than the logical extent of the exNode.
     -D, --demo                  Add the messages understood by the LoRS View visualization.
   Help options:
     -?, --help                  Show this help message
     --usage                     Display brief usage message

Inside rspec

Attachments (3)

Download all attachments as: .zip