Opened 9 years ago

Last modified 9 years ago

#134 new

"Error during join for unit" reported vy 11 nodes while setting up 10 Slivers w/10 VMs test

Reported by: lnevers@bbn.com Owned by: somebody
Priority: major Milestone:
Component: Experiment Version: SPIRAL5
Keywords: Cc:
Dependencies:

Description

=> Test 1: (10 slivers with 10 VMs with 1 vlan)

  1. Set up slivers 1-5 via the RCI SM. Reserved 50 nodes successfully. Checked listresources and found that 1 node was still available.
  1. Setup up sliver 6-10 via the ExoSM. Sliver 6-8 were successful, the 9th sliver failed with one node reporting "failed to join error", while the other 9 nodes were active. Since 13 additional nodes were available, setup the 10th slice, but all 10 VMs failed with the "failed to join error". After the 10th sliver, 3 nodes were still available according to listresources. (Overall total reserved 39)

Sliver failures for the 10th sliver at the ExoSM sliver:

{
  "geni_status": "failed", 
  "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln-10vm10", 
  "geni_resources": [
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-10", 
      "geni_error": "Reservation ec1f90bd-4dbd-40ec-9e2d-451a5b6004b0 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: EB9AE266 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-1", 
      "geni_error": "Reservation 506f4ed9-ae07-40c9-9c77-b1b64a901cec (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: E834858B [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-2", 
      "geni_error": "Reservation ac76aa6e-e929-4427-abce-d2f9d6c32b39 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: EDE5FB5 [1]: unable to 
create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-4", 
      "geni_error": "Reservation 3cdd9078-204b-42a1-afe9-22f2e9c802da (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 96024B84 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#Lan", 
      "geni_error": "", 
      "geni_status": "Active"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-3", 
      "geni_error": "Reservation bf860c1a-48a0-4b1d-b112-803f2c17d138 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: C647C367 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-6", 
      "geni_error": "Reservation 8be3072b-86fc-4dde-929f-d85689c92be6 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: AB19C185 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-5", 
      "geni_error": "Reservation 7102eef3-3f7c-46ab-8399-8f49b8825bf6 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 97DB3CFD [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-8", 
      "geni_error": "Reservation ae074dd5-184e-47c7-b3cc-0849d5c66264 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 5B15EE8D [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-7", 
      "geni_error": "Reservation edf31a03-3872-4495-8fb0-772c12cc7fc9 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: C6B0F0A0 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 
    {
      "orca_expires": "Wed Dec 19 20:17:29 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+a1b69c90-858c-4b19-868a-cac067cdbe06#VM-9", 
      "geni_error": "Reservation d2d918bd-0e61-4c94-af26-3416689fe222 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 2DD2CD35 [1]: unable to
 create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }
  ]
}

The failure in the 9th slice:

    {
      "orca_expires": "Wed Dec 19 20:07:34 EST 2012", 
      "geni_urn": "urn:publicid:IDN+exogeni.net:rcivmsite+sliver+378d4a07-47ad-4235-b923-67236416a5b3#VM-2", 
      "geni_error": "Reservation 041b8ec3-c2b7-407c-9b00-28550568be94 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice
+ln-10vm9) is in state [Failed,None], err=resources failed to join: Error during join for unit: 64F0AA0A [1]: unable to 
create instance: exit code 1, \n", 
      "geni_status": "Failed"
    }, 

Change History (2)

comment:1 Changed 9 years ago by lnevers@bbn.com

Ran another 100 node scenario, which also failed to get 100 VM and returned the same failures as test 1, so capturing results in this ticket.

=> Test 2: (2 slivers with 20 VMs and 1 sliver with 10 VMs)

  • Successfully set up 2 slivers with 20 VMs via RENCI SM.
  • Successfully set up 1 sliver with 10 VMs via RENCI SM.
  • Set up 2 slivers with 20 VMs via ExoSM. Sliver 1 ok, Sliver 2 failed with 2 nodes reporting "Error during join for unit"
  • Set up 1 sliver with 10 VMs via ExoSM, which failed all 10 nodes reporting "Error during join for unit". After this failure 3 nodes were still available via the ExoSM.

Following are the errors reported by the 8 nodes in the 10VM sliver:

"geni_error": "Reservation 36449ca9-295a-4d30-8bd2-bc50a1b34ca5 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 582047FF [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 199453c1-d598-4201-9921-fd5548b4518d (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 362C8176 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 8edcad00-a494-4824-ab9a-00f23cc5e887 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: AA11ABA2 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 4b981c9f-6cb4-4d8f-b4b4-c8a09761cdfe (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: CD3789FD [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 6887122e-57e9-407e-b0f4-29f4925d6ee2 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: D6198D9B [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 37958555-e63c-419f-8cce-a0e3a72bd9a9 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 61877728 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 85b6540a-abaa-4cb7-8258-f936f2c2b648 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 91DE53F6 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 227d8317-3c5c-455c-97b9-6aa12bce6c05 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 60212A6 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation aea4561b-ac2a-46e3-a02d-ef18981b038d (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: F9EFF9A3 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation b6106e67-094d-4b88-8e6e-26d70d1f0470 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln10vm1) is in state [Failed,None], err=resources failed to join: Error during join for unit: 7E9CA8A1 [1]: unable to create instance: exit code 1, \n", 

Following are the errors reported by the 2 nodes in the 20VM sliver:

"geni_error": "Reservation 59342fc4-5091-4f3f-9d87-fd9822a2f785 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln20vm2) is in state [Failed,None], err=resources failed to join: Error during join for unit: 9038FBE5 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation ef6aa37e-1e60-455f-866a-205fe88c7eec (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ln20vm2) is in state [Failed,None], err=resources failed to join: Error during join for unit: 672423CC [1]: unable to create instance: exit code 1, \n", 

comment:2 Changed 9 years ago by lnevers@bbn.com

Another test..

=> Test 3: (20 slivers with 5 VMs each)

  • Successfully created 10 slivers with 5 VMs via RENCI SM
  • Created 10 slivers with 5 VMs via ExoSM. Eight slivers were successful. Both the 9th sliver and 10th sliver had 5 nodes each that failed with "Error during join error".

Errors reported by sliver 9:

"geni_error": "Reservation 4aa38e94-08cc-4fb1-bf36-678bcf8f64ab (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice9) is in state [Failed,None], err=resources failed to join: Error during join for unit: 1357F87E [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation f57f3de5-9373-43ea-9318-5e534c3012a2 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice9) is in state [Failed,None], err=resources failed to join: Error during join for unit: 426C7E07 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation aa03a0b3-2ed0-4617-97bb-6a7e210be55d (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice9) is in state [Failed,None], err=resources failed to join: Error during join for unit: E3769554 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 38954d10-bf41-46ad-8f2e-badef217e3b1 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice9) is in state [Failed,None], err=resources failed to join: Error during join for unit: C8EE1D3F [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation d0083ca9-077f-4c04-a188-ee6ef16b4081 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice9) is in state [Failed,None], err=resources failed to join: Error during join for unit: BD6FB630 [1]: unable to create instance: exit code 1, \n", 

Error reported by sliver 10:

"geni_error": "Reservation 909f08bd-78eb-4532-9db9-6c665644dac3 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice10) is in state [Failed,None], err=resources failed to join: Error during join for unit: BBADA0AA [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 3a74b70e-9e21-4280-a3d9-2dcb87b1025b (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 187067FF [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation a5598d68-d375-4b8e-8ff0-7bbba0e5a20e (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice10) is in state [Failed,None], err=resources failed to join: Error during join for unit: B9F79B50 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation 92b2b384-dcff-42fe-b6d6-6c4762570746 (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice10) is in state [Failed,None], err=resources failed to join: Error during join for unit: 5F0535D1 [1]: unable to create instance: exit code 1, \n", 
"geni_error": "Reservation bb7ee881-e3ab-46c2-8621-5a67a1e78e5a (Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+5vmslice10) is in state [Failed,None], err=resources failed to join: Error during join for unit: DD179ED7 [1]: unable to create instance: exit code 1, \n", 
Note: See TracTickets for help on using tickets.