Changes between Initial Version and Version 1 of PlasticSlices/BaselineEvaluation/Baseline8Details


Ignore:
Timestamp:
07/11/11 18:12:37 (10 years ago)
Author:
Josh Smift
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PlasticSlices/BaselineEvaluation/Baseline8Details

    v1 v1  
     1[[PageOutline]]
     2
     3Here are the details of [wiki:PlasticSlices/BaselineEvaluation#Baseline8 Baseline 8].
     4
     5This baseline didn't use the existing ten slices and experiments, but instead created many new slices, on the ProtoGENI slice authority pgeni.gpolab.bbn.com.
     6
     7We did three experiments, creating one slice per second in each: One for ten slices in ten seconds, one for a hundred slices in a hundred seconds, and one for a thousand slices in a thousand seconds.
     8
     9Since the second and third experiments revealed significant issues with the Slice Authority, we didn't attempt to use these slices to create slivers, but intend to experiment more with that after GEC 11 in the continuation phase of this project.
     10
     11The raw logs (output from the createslice commands) of each experiment are at http://www.gpolab.bbn.com/plastic-slices/baseline-logs/baseline-8/, with a subdirectory for each experiment.
     12
     13= Ten slices in ten seconds =
     14
     15== Commands run ==
     16
     17{{{
     18for i in {01..10} ; do omni createslice pb8-a$i > pb8-a$i.txt 2>&1 & sleep 1 ; done
     19}}}
     20
     21== Results ==
     22
     23All ten slices were created successfully.
     24
     25== Analysis ==
     26
     27Creating slices at a rate of one per second doesn't seem to pose any fundamental problem for the SA.
     28
     29= One hundred slices in one hundred seconds =
     30
     31== Commands run ==
     32
     33{{{
     34for i in {001..100} ; do omni createslice pb8-b$i > pb8-b$i.txt 2>&1 & sleep 1 ; done
     35}}}
     36
     37== Results ==
     38
     39About two thirds of the slices (62) were created successfully, with three types of failures for the others.
     40
     41Four slices failed in this way:
     42
     43{{{
     44ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b013: Error (code 2)
     45ERROR:omni:Create Slice Failed for slice name pb8-b013.
     46}}}
     47
     48Three slices failed in this way:
     49
     50{{{
     51ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
     52ERROR:omni.protogeni:Failed to resolve slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority
     53ERROR:omni:Create Slice Failed for slice name pb8-b045.
     54}}}
     55
     56Thirty-one slices failed in this way:
     57
     58{{{
     59ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
     60ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015
     61ERROR:omni:Create Slice Failed for slice name pb8-b015.
     62}}}
     63
     64One other note: An analysis of the PG MySQL database suggested that 79 slices were actually created. We didn't investigate the cause of the discrepancy.
     65
     66== Analysis ==
     67
     68The command above launched a createslice request every second, but the final request didn't complete until about eleven minutes after the first. There was a gap of about three minutes in the middle when no slices were created, due to load on the Slice Authority host.
     69
     70= One thousand slices in one thousand seconds =
     71
     72== Commands run ==
     73
     74{{{
     75for i in {0001..1000} ; do omni createslice pb8-c$i > pb8-c$i.txt 2>&1 & sleep 1 ; done
     76}}}
     77
     78== Results ==
     79
     80Only forty-six of the slices were created successfully, with a variety of failures for the others.
     81
     82Most of the failures (864 of them) mentioned failing to get a valid user credential. They all said:
     83
     84{{{
     85ERROR:omni.protogeni:Cannot create a PG slice without a valid user credential
     86ERROR:omni:Create Slice Failed for slice name pb8-c0544.
     87}}}
     88
     89and sometimes a few similar things. The most relevant differences in the errors, with counts of how many got that error, are:
     90
     91{{{
     92   1  ERROR:omni.protogeni:Received error message: Who are you?
     93   7  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: <Fault -32400: 'XMLRPC Server Error'>
     94  24  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: <Fault 503: 'CM temporarily offline; please try again later'>
     95  33  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno -3] Temporary failure in name resolution
     96 420  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno 104] Connection reset by peer
     97 337  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
     98  42  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
     99}}}
     100
     101Then there were these others:
     102
     103{{{
     104   8  ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0002: Error (code 2)
     105   1  ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0047 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
     106   2  ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0076 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
     107  48  ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0008 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
     108  31  ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0026 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
     109}}}
     110
     111We didn't analyze PG MySQL database after this run.
     112
     113== Analysis ==
     114
     115The command above launched a createslice request every second, but the final request didn't complete until about forty-five minutes after the first.
     116
     117The Slice Authority host was significantly overloaded, to the point that some normal Emulab / ProtoGENI processes died; we rebooted the server afterwards to bring everything back.