[[PageOutline]] Here are the details of [wiki:PlasticSlices/BaselineEvaluation#Baseline8 Baseline 8]. This baseline didn't use the existing ten slices and experiments, but instead created many new slices, on the ProtoGENI slice authority pgeni.gpolab.bbn.com. We did three experiments, creating one slice per second in each: One for ten slices in ten seconds, one for a hundred slices in a hundred seconds, and one for a thousand slices in a thousand seconds. Since the second and third experiments revealed significant issues with the Slice Authority, we didn't attempt to use these slices to create slivers, but intend to experiment more with that after GEC 11 in the continuation phase of this project. The raw logs (output from the createslice commands) of each experiment are at http://www.gpolab.bbn.com/plastic-slices/baseline-logs/baseline-8/, with a subdirectory for each experiment. = Ten slices in ten seconds = == Commands run == {{{ for i in {01..10} ; do omni createslice pb8-a$i > pb8-a$i.txt 2>&1 & sleep 1 ; done }}} == Results == All ten slices were created successfully. == Analysis == Creating slices at a rate of one per second doesn't seem to pose any fundamental problem for the SA. = One hundred slices in one hundred seconds = == Commands run == {{{ for i in {001..100} ; do omni createslice pb8-b$i > pb8-b$i.txt 2>&1 & sleep 1 ; done }}} == Results == About two thirds of the slices (62) were created successfully, with three types of failures for the others. Four slices failed in this way: {{{ ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b013: Error (code 2) ERROR:omni:Create Slice Failed for slice name pb8-b013. }}} Three slices failed in this way: {{{ ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: ERROR:omni.protogeni:Failed to resolve slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b045 at PG slice authority ERROR:omni:Create Slice Failed for slice name pb8-b045. }}} Thirty-one slices failed in this way: {{{ ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-b015 ERROR:omni:Create Slice Failed for slice name pb8-b015. }}} One other note: An analysis of the PG MySQL database suggested that 79 slices were actually created. We didn't investigate the cause of the discrepancy. == Analysis == The command above launched a createslice request every second, but the final request didn't complete until about eleven minutes after the first. There was a gap of about three minutes in the middle when no slices were created, due to load on the Slice Authority host. = One thousand slices in one thousand seconds = == Commands run == {{{ for i in {0001..1000} ; do omni createslice pb8-c$i > pb8-c$i.txt 2>&1 & sleep 1 ; done }}} == Results == Only forty-six of the slices were created successfully, with a variety of failures for the others. Most of the failures (864 of them) mentioned failing to get a valid user credential. They all said: {{{ ERROR:omni.protogeni:Cannot create a PG slice without a valid user credential ERROR:omni:Create Slice Failed for slice name pb8-c0544. }}} and sometimes a few similar things. The most relevant differences in the errors, with counts of how many got that error, are: {{{ 1 ERROR:omni.protogeni:Received error message: Who are you? 7 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: 24 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed. Server says: 33 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno -3] Temporary failure in name resolution 420 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: Unknown socket error [Errno 104] Connection reset by peer 337 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: 42 ERROR:omni.protogeni:Call for Get PG user credential from SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa using cert /home/jbs/.gcf/jbs@pgeni.gpolab.bbn.com.pem failed.: ProtocolError: }}} Then there were these others: {{{ 8 ERROR:omni.protogeni:Failed to create new PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0002: Error (code 2) 1 ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0047 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: 2 ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0076 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: 48 ERROR:omni.protogeni:Call for Create PG slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0008 at SA https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: 31 ERROR:omni.protogeni:Call for Look up slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+pb8-c0026 at PG slice authority https://boss.pgeni.gpolab.bbn.com/protogeni/xmlrpc/sa failed.: ProtocolError: }}} We didn't analyze PG MySQL database after this run. == Analysis == The command above launched a createslice request every second, but the final request didn't complete until about forty-five minutes after the first. The Slice Authority host was significantly overloaded, to the point that some normal Emulab / ProtoGENI processes died; we rebooted the server afterwards to bring everything back.