Changes between Initial Version and Version 1 of PlasticSlices/BaselineEvaluation/Baseline8Details

Timestamp:

07/11/11 18:12:37 (13 years ago)

Author:

Josh Smift

Comment:

--

PlasticSlices/BaselineEvaluation/Baseline8Details

@@ +1 @@
+[[PageOutline]]
+
+Here are the details of [wiki:PlasticSlices/BaselineEvaluation#Baseline8 Baseline 8].
+
+This baseline didn't use the existing ten slices and experiments, but instead created many new slices, on the ProtoGENI slice authority pgeni.gpolab.bbn.com.
+
+We did three experiments, creating one slice per second in each: One for ten slices in ten seconds, one for a hundred slices in a hundred seconds, and one for a thousand slices in a thousand seconds.
+
+Since the second and third experiments revealed significant issues with the Slice Authority, we didn't attempt to use these slices to create slivers, but intend to experiment more with that after GEC 11 in the continuation phase of this project.
+
+The raw logs (output from the createslice commands) of each experiment are at http://www.gpolab.bbn.com/plastic-slices/baseline-logs/baseline-8/, with a subdirectory for each experiment.
+
+= Ten slices in ten seconds =
+
+== Commands run ==
+
+{{{
+for i in {01..10} ; do omni createslice pb8-a$i > pb8-a$i.txt 2>&1 & sleep 1 ; done
+}}}
+
+== Results ==
+
+All ten slices were created successfully.
+
+== Analysis ==
+
+Creating slices at a rate of one per second doesn't seem to pose any fundamental problem for the SA.
+
+= One hundred slices in one hundred seconds =
+
+== Commands run ==
+
+{{{
+for i in {001..100} ; do omni createslice pb8-b$i > pb8-b$i.txt 2>&1 & sleep 1 ; done
+}}}
+
+== Results ==
+
+About two thirds of the slices (62) were created successfully, with three types of failures for the others.
+
+Four slices failed in this way:
+
+{{{
+ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b013: Error (code 2)
+ERROR:omni:Create Slice Failed for slice name pb8-b013.
+}}}
+
+Three slices failed in this way:
+
+{{{
+ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
+ERROR:omni.protogeni:Failed to resolve slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority
+ERROR:omni:Create Slice Failed for slice name pb8-b045.
+}}}
+
+Thirty-one slices failed in this way:
+
+{{{
+ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
+ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015
+ERROR:omni:Create Slice Failed for slice name pb8-b015.
+}}}
+
+One other note: An analysis of the PG MySQL database suggested that 79 slices were actually created. We didn't investigate the cause of the discrepancy.
+
+== Analysis ==
+
+The command above launched a createslice request every second, but the final request didn't complete until about eleven minutes after the first. There was a gap of about three minutes in the middle when no slices were created, due to load on the Slice Authority host.
+
+= One thousand slices in one thousand seconds =
+
+== Commands run ==
+
+{{{
+for i in {0001..1000} ; do omni createslice pb8-c$i > pb8-c$i.txt 2>&1 & sleep 1 ; done
+}}}
+
+== Results ==
+
+Only forty-six of the slices were created successfully, with a variety of failures for the others.
+
+Most of the failures (864 of them) mentioned failing to get a valid user credential. They all said:
+
+{{{
+ERROR:omni.protogeni:Cannot create a PG slice without a valid user credential
+ERROR:omni:Create Slice Failed for slice name pb8-c0544.
+}}}
+
+and sometimes a few similar things. The most relevant differences in the errors, with counts of how many got that error, are:
+
+{{{
+   1  ERROR:omni.protogeni:Received error message: Who are you?
+   7  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: <Fault -32400: 'XMLRPC Server Error'>
+  24  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: <Fault 503: 'CM temporarily offline; please try again later'>
+  33  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno -3] Temporary failure in name resolution
+ 420  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno 104] Connection reset by peer
+ 337  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
+  42  ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
+}}}
+
+Then there were these others:
+
+{{{
+   8  ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0002: Error (code 2)
+   1  ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0047 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
+   2  ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0076 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: 500 Internal Server Error>
+  48  ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0008 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
+  31  ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0026 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: <ProtocolError for boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa: -1 >
+}}}
+
+We didn't analyze PG MySQL database after this run.
+
+== Analysis ==
+
+The command above launched a createslice request every second, but the final request didn't complete until about forty-five minutes after the first.
+
+The Slice Authority host was significantly overloaded, to the point that some normal Emulab / ProtoGENI processes died; we rebooted the server afterwards to bring everything back.