Opened 12 years ago

Closed 12 years ago

#58 closed (fixed)

Sliver status for monitoring sliver returns "Error during join for unit"

Reported by: lnevers@bbn.com Owned by: somebody
Priority: major Milestone: EG-MON-1
Component: Monitoring Version: SPIRAL4
Keywords: monitoring Cc:
Dependencies:

Description

The following sequence of event was reported by Tim for the slice tuptymon:

  1. Tim thought Ilia said that the meso-scale monitoring slice wouldn't be affected by the redeploy, but found that the pings started to fail, and did not recover.
  1. Tim tried to create a new sliver, that createsliver call hung, so he cancelled it.
  1. Tim then tried deleting the sliver just in case, and then created the sliver. The createsliver call hung again, so Tim cancelled it.
  1. Tim waited for a while and created the sliver again, and got sliverstatus "configuring" for a while for at least 30 minutes. Tim executed a sliverstatus which returned the following error: 
tupty@weentop-work:~$ omni sliverstatus -a
https://bbn-hn.exogeni.net:11443/orca/xmlrpc tuptymon
/usr/bin/expect
Enter PEM pass phrase:spawn /home/tupty/bin/latest_omni -c
/home/tupty/.gcf/omni_config sliverstatus -a
https://bbn-hn.exogeni.net:11443/orca/xmlrpc tuptymon
INFO:omni:Loading config file /home/tupty/.gcf/omni_config
INFO:omni:Using control framework pgeni
Enter PEM pass phrase:
Enter PEM pass phrase:
Enter PEM pass phrase:
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon
expires on 2012-09-15 00:00:00 UTC
INFO:omni:Status of Slice
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon:
Enter PEM pass phrase:
INFO:omni:Sliver status for Slice
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon at AM URL
https://bbn-hn.exogeni.net:11443/orca/xmlrpc
INFO:omni:{'geni_resources': [{'geni_error': "Reservation
0a339ca7-56fe-4547-af8f-633734d02457 (Slice
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon) is in state
[Failed,None], err=resources failed to join: Error during join for unit:
F901D267 [1]: unable to create instance: exit code 1, ip address must be
specified\n\nDisassociate a public IP address from an
instance.\n\neuca-disassociate-address [-h, --help] [--version]
[--debug] ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tIP address to
disassociate.\n\nOPTIONAL PARAMETERS\n\t\n\n\n-a, --access-key\t\tUser's
Access Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U,
--url\t\t\tURL of the Cloud to connect to.\n\n--config\t\t\tRead
credentials and cloud settings from the \n\t\t\t\tspecified config file
(defaults to $HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h,
--help\t\t\tDisplay this help message.\n\n--version \t\t\tDisplay the
version of this tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools
will use the environment variables EC2_URL, EC2_ACCESS_KEY,
EC2_SECRET_KEY, EC2_CERT, EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by
default.\n    \nip address must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \nip address
must be specified\n\nReleases a public IP
address.\n\neuca-release-address [-h, --help] [--version] [--debug]
ip\n\nREQUIRED PARAMETERS\n\nip\t\t\t\tThe public IP address to
release.\n\nOPTIONAL PARAMETERS\n\n\n\n-a, --access-key\t\tUser's Access
Key ID.\n\n-s, --secret-key\t\tUser's Secret Key.\n\n-U, --url\t\t\tURL
of the Cloud to connect to.\n\n--config\t\t\tRead credentials and cloud
settings from the \n\t\t\t\tspecified config file (defaults to
$HOME/.eucarc or /etc/euca2ools/eucarc).\n\n-h, --help\t\t\tDisplay this
help message.\n\n--version \t\t\tDisplay the version of this
tool.\n\n--debug \t\t\tTurn on debugging.\n\nEuca2ools will use the
environment variables EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT,
EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT by default.\n    \n",
                     'geni_status': 'Failed',
                     'geni_urn':
'urn:publicid:IDN+exogeni.net:bbnvmsite+sliver+bf6ba26f-bc8b-44f2-9203-d8f3a53c3be4#geni0',
                     'orca_expires': 'Fri Jun 29 18:57:07 UTC 2012'}],
 'geni_status': 'failed',
 'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon'}
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed sliverstatus:

  Options as run:
		aggregate: https://bbn-hn.exogeni.net:11443/orca/xmlrpc
		configfile: /home/tupty/.gcf/omni_config
		framework: pgeni
		native: True

  Args: sliverstatus tuptymon

  Result Summary: Slice
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon expires on
2012-09-15 00:00:00 UTC
Returned status of slivers on 1 of 1 possible aggregates.
INFO:omni: ============================================================

Attachments (1)

orca-bbn-exogeni-tuptymon.rspec​ (1.2 KB) - added by lnevers@bbn.com 12 years ago.

Download all attachments as: .zip

Change History (6)

Changed 12 years ago by lnevers@bbn.com

Attachment: orca-bbn-exogeni-tuptymon.rspec​ added

comment:1 Changed 12 years ago by lnevers@bbn.com

On 6/28/12 4:32 PM, Ilia Baldine wrote:

Try again.

BTW, Tim's slice was created on BBN-SM so it was affected by the redeploy.

-ilia

Since the sliver status was still reporting the error in the original problem description, Tim has deleted the existing sliver and will created the sliver again.

comment:2 Changed 12 years ago by lnevers@bbn.com

On 6/28/12 5:02 PM, Tim Upthegrove wrote:

On 06/28/2012 04:32 PM, Ilia Baldine wrote:

Try again.

BTW, Tim's slice was created on BBN-SM so it was affected by the redeploy.

Oops, right, I misread the subject of the email about the redeploy as "[exogeni-design] BBN SM restart (will not affect slices started via ExoSM, RCI SM or BBN SM)"

The VM is now up, and the mesoscale connectivity monitoring is working again.

On 6/28/12 5:07 PM, Ilia Baldine wrote:

This was a transient OpenStack? error. We will modify the driver code to provide a shorter error message.

-ilia

comment:3 Changed 12 years ago by lnevers@bbn.com

Ilia, we believe it may be difficult to reproduce this problem at will to verify the changes to the error report. Should we leave this ticket open as a place holder to remind you about the changes to the error reporting?

If not, we can close this ticket and will updated it if/when this problem occurs again with the error that is actually reported.

comment:4 Changed 12 years ago by ibaldin@renci.org

I suggest you close it. Ultimately it is a transient OpenStack? issue. There error message we would produce would be marginally more useful.

comment:5 Changed 12 years ago by lnevers@bbn.com

Resolution: → fixed
Status: new → closed

Agreed, closing ticket!

Note: See TracTickets for help on using tickets.