cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Old, Curtis" <Curtis....@neustar.biz>
Subject Re: Xenserver hypervisior will not reconnect after reboot
Date Thu, 30 May 2013 14:27:20 GMT
Not yet was hoping to avoid that....

Curtis Old 
Neustar, Inc. / Neustarlabs / Senior Research Engineer
46000 Center Oak Plaza Sterling, VA 20166
Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis.old@neustar.biz
/ www.neustar.biz <http://www.neustar.biz/>
________________________________________


The information contained in this e-mail message is intended only for the
use of the recipient(s) named above and may contain confidential and/or
privileged information. If you are not the intended recipient you have
received this e-mail message in error and any review, dissemination,
distribution, or copying of this message is strictly prohibited. If you
have received this communication in error, please notify us immediately
and delete the original message.





On 5/30/13 10:30 AM, "Ahmad Emneina" <aemneina@gmail.com> wrote:

>Did you reinstall the hypervisor before reconnecting it back to the pool?
>
>Ahmad
>
>On May 30, 2013, at 4:10 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>
>> Ok both are in an "Alert" state.  So I'M kinda at a lost for what to try
>> next.
>> 
>> 
>> On 5/30/13 6:55 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>> 
>>> Ok totally weird stuff now.
>>> 
>>> After trying the previous steps, both Xenserver nodes failed to
>>>reconnect
>>> to the cluster, so I decided to remove and re-add them.  I performed
>>>the
>>> following:
>>> 
>>>  1.  Unmanaged the cluster
>>>  2.  Removed the 2 nodes
>>>  3.  Managed the cluster, the 3 remaining nodes went into an Up state.
>>>  4.  Add 1 of the Xenserver nodes, and was going to wait to add the
>>>last
>>> one, if the 1st added successfully.
>>> 
>>> The result I see both of the 2 nodes trying to connect (one I didn't
>>>add
>>> yet), waiting to see if the complete or fail.
>>> 
>>> 
>>> 
>>> On 5/29/13 8:39 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>>> 
>>>> Same status after an hour.  Should I delete them and re-add them or ??
>>>> 
>>>> 
>>>> 
>>>> On 5/29/13 7:30 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> Well one lab-cloud-9 is in an "Alert" State the other, lab-cloud-10
>>>>>is
>>>>> "Disconnected", and after a refresh on the UI lab-cloud-9 is back to
>>>>> "Connecting"
>>>>> 
>>>>> 
>>>>> Suggestions?
>>>>> 
>>>>> 
>>>>> 
>>>>> On 5/29/13 7:17 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>>>>> 
>>>>>> Interesting the UUID's in the database where different then the
>>>>>> Xenserver
>>>>>> UUID's (there where 2 Xenservers hosts that had inode problems) But
>>>>>> then
>>>>>> again the other servers all had different UUID's as well <shrug>.
>>>>>> 
>>>>>> Mysql DB:
>>>>>> 
>>>>>>                  id: 21
>>>>>>                name: lab-cloud-10.neustarlab.biz
>>>>>>                uuid: 5c65a56a-5a71-44e8-b4ae-89bf81ef4d24
>>>>>> 
>>>>>> And on the Host:
>>>>>> 
>>>>>> 
>>>>>> uuid=dde1a6d7-8708-451f-a4a8-a9c5be79f57e
>>>>>> 
>>>>>> It has been 20 to 25 min and one lab-cloud-9 its state is
>>>>>>"Connecting"
>>>>>> lab-cloud-10 is in "Disconnected" and the other 3 hosts
>>>>>>(lab-cloud-6,
>>>>>> 7,
>>>>>> 8) are "Up"
>>>>>> 
>>>>>> Wait a bit longer?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 5/28/13 6:37 PM, "Kirk Kosinski" <kirkkosinski@gmail.com> wrote:
>>>>>> 
>>>>>>> Before hacking the db, I suggest trying one more thing.
>>>>>>> 
>>>>>>> 1. In the CS UI, unmanage the cluster with the affected host.
>>>>>>> 2. From the CLI of a host in the XS pool, clear the host tags of
>>>>>>>the
>>>>>>> affected host.
>>>>>>> xe host-param-clear param-name=tags uuid=<UUID of affected host>
>>>>>>> 3. In the CS UI, manage the cluster with the affected host.
>>>>>>> 
>>>>>>> This usually solves this type of problem (it might take a couple
>>>>>>> minutes).  Clearing host tags won't affect running VMs (if any).
>>>>>>> 
>>>>>>> Best regards,
>>>>>>> Kirk
>>>>>>> 
>>>>>>> On 05/28/2013 03:03 PM, Ahmad Emneina wrote:
>>>>>>>> first, I'd query to make sure its the right host, judging from the
>>>>>>>> logs
>>>>>>>> it
>>>>>>>> has an id of 20... execute the following
>>>>>>>> backup command from the cli that can access the db server:
>>>>>>>> 
>>>>>>>> mysqldump <credentials if necessary> --databases cloud > cloud.sql
>>>>>>>> 
>>>>>>>> Then start operating...
>>>>>>>> 
>>>>>>>> select * from host where id=20\G (hit the enter key)
>>>>>>>> 
>>>>>>>> if this is infact the misbehaving host, execute
>>>>>>>> 
>>>>>>>> update host set status='Up' where id=20; (again hit enter)
>>>>>>>> 
>>>>>>>> then try to launch a vm on that host (in the deployVirtualMachine
>>>>>>>> api
>>>>>>>> command you can/should be able to specify the exact host)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, May 28, 2013 at 2:46 PM, Old, Curtis
>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Oh god where do I find that sucker in the DB?  Sorry I'M an admin
>>>>>>>>> type
>>>>>>>>> that isn't a mysql guru or Ji
>>>>>>>>> 
>>>>>>>>> Curtis Old
>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>> ________________________________________
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The information contained in this e-mail message is intended only
>>>>>>>>> for
>>>>>>>>> the
>>>>>>>>> use of the recipient(s) named above and may contain confidential
>>>>>>>>> and/or
>>>>>>>>> privileged information. If you are not the intended recipient you
>>>>>>>>> have
>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>> dissemination,
>>>>>>>>> distribution, or copying of this message is strictly prohibited.
>>>>>>>>>If
>>>>>>>>> you
>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>> immediately
>>>>>>>>> and delete the original message.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 5/28/13 5:43 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> oh this looks like a bug. A potential workaround for it is
>>>>>>>>>>marking
>>>>>>>>>> the
>>>>>>>>>> host
>>>>>>>>>> as Up in the cloudstack db. I'd make a backup first then modify
>>>>>>>>>> the
>>>>>>>>>> db
>>>>>>>>>> entry.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis
>>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>>  Tailed last 3000 lines right after trying a forced reconnect:
>>>>>>>>>>> http://paste.cloudstack.org/hTOm/
>>>>>>>>>>> 
>>>>>>>>>>> Listed as Management log N*,
>>>>>>>>>>> 
>>>>>>>>>>>  Author Curtis Old
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>  Can we see the latest logs? paste.cloudstack.org
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis
>>>>>>>>>>> <Curtis.Old@neustar.biz>wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Tried that already (like 2 times)
>>>>>>>>>>>> 
>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> The information contained in this e-mail message is intended
>>>>>>>>>>>> only
>>>>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>>confidential
>>>>>>>>>>>> and/or
>>>>>>>>>>>> privileged information. If you are not the intended recipient
>>>>>>>>>>>> you
>>>>>>>>>>>> have
>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>> dissemination,
>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>>prohibited.
>>>>>>>>>>>> If
>>>>>>>>>>>> you
>>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>>> immediately
>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <aemneina@gmail.com>
>>>>>>>>>>>>wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> how about a reboot of the host, then force reconnect.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis
>>>>>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Bummer no goodÅ ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The information contained in this e-mail message is intended
>>>>>>>>>>>>>> only
>>>>>>>>>>>> for
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>>>> confidential
>>>>>>>>>>>> and/or
>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>recipient
>>>>>>>>>>>>>> you
>>>>>>>>>>>> have
>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>>>> prohibited.
>>>>>>>>>>>>>> If
>>>>>>>>>>>> you
>>>>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>>> immediately
>>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <aemneina@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> dont do that :D do a toolstack restart on the xenserver
>>>>>>>>>>>>>>>host,
>>>>>>>>>>>>>>> and a
>>>>>>>>>>>>>> force
>>>>>>>>>>>>>>> reconnect from cloudstack. see if that makes them play
>>>>>>>>>>>>>>>nice.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis
>>>>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>  Well the hypervisior looks fine from a Xenserver
>>>>>>>>>>>>>>>> prospective
>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> CloudStack won't "reconnect" it.  I thought about deleting
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>> CS
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> re-adding it, but that bugs me lol
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *Curtis Old**
>>>>>>>>>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research
>>>>>>>>>>>>>>>>Engineer*
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166**
>>>>>>>>>>>>>>>> **Office:** **+1.571.434.5384** **Mobile:
>>>>>>>>>>>>>>>> **+1.540.532.2230**
>>>>>>>>>>>> **/**
>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>> curtis.old@neustar.biz**  / **www.neustar.biz**     *
>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> *The information contained in this e-mail message is
>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>> only
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> the use of the recipient(s) named above and may contain
>>>>>>>>>>>> confidential
>>>>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>>> recipient
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>>>>>> prohibited.
>>>>>>>>>>>> If
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> have received this communication in error, please notify
>>>>>>>>>>>>>>>>us
>>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> delete the original message.*
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>  From: Ahmad Emneina <aemneina@gmail.com>
>>>>>>>>>>>>>>>> Reply-To: "aemneina@gmail.com" <aemneina@gmail.com>
>>>>>>>>>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM
>>>>>>>>>>>>>>>> To: "Old, Curtis" <curtis.old@neustar.biz>
>>>>>>>>>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect
>>>>>>>>>>>>>>>>after
>>>>>>>>>>>> reboot
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>  are you good to go again, after cleaning up some space?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis
>>>>>>>>>>>>>>>> <Curtis.Old@neustar.biz>wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Sorry the log entries are from the xenserver log
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The information contained in this e-mail message is
>>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>> only
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>>>>>>> confidential
>>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>>>> recipient
>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>> prohibited. If
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> have received this communication in error, please notify
>>>>>>>>>>>>>>>>>us
>>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis"
>>>>>>>>>>>>>>>>><Curtis.Old@neustar.biz>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The in innodes did fill, so after I cleaned them up I
>>>>>>>>>>>>>>>>>> rebooted
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>> the host was singing and dancing again in XenCenter, and
>>>>>>>>>>>> primary
>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>> was mounted.  What from the management log, if we can
>>>>>>>>>>>>>>>>>> narrow
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> down?
>>>>>>>>>>>>>>>>>> When I try the forced reconnect only this shows up:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622
>>>>>>>>>>>>>>>>> inet_rpc||http_critical]
>>>>>>>>>>>>>>>>>> Connection terminated
>>>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false)
>>>>>>>>>>>> connected
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>>> s_pid=23063
>>>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false)
>>>>>>>>>>>> connected
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>>> s_pid=23063
>>>>>>>>>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623
>>>>>>>>>>>>>>>>> inet_rpc||http_critical]
>>>>>>>>>>>>>>>>>> Connection terminated
>>>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false)
>>>>>>>>>>>> connected
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>>> s_pid=23068
>>>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false)
>>>>>>>>>>>> connected
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28
>>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>>> s_pid=23072
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The information contained in this e-mail message is
>>>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>> only
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>> confidential
>>>>>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>>>>> recipient
>>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>> prohibited.
>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> have received this communication in error, please notify
>>>>>>>>>>>>>>>>>> us
>>>>>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <aemneina@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> pretty odd... how does the disk space look on the
>>>>>>>>>>>>>>>>>>> xenserver
>>>>>>>>>>>> host?
>>>>>>>>>>>>>>>>> we'd
>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>> more from the management log, as well as the
>>>>>>>>>>>>>>>>>>> xensource.log
>>>>>>>>>>>>>>>>>>> off
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> hypervisor.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis
>>>>>>>>>>>>>> <Curtis.Old@neustar.biz
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get
>>>>>>>>>>>> "Command
>>>>>>>>>>>>>>>>> Failed
>>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>> to internal server Error" any ideas?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> management-server.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG
>>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details:
>>>>>>>>>>>> AsyncJobVO
>>>>>>>>>>>>>>>>>>>> {id:230,
>>>>>>>>>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null,
>>>>>>>>>>>>>>>>>>>>instanceType:
>>>>>>>>>>>> Host,
>>>>>>>>>>>>>>>>>>>> instanceId:
>>>>>>>>>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd,
>>>>>>>>>>>> cmdOriginator:
>>>>>>>>>>>>>>>>> null,
>>>>>>>>>>>>>>>>>>>> cmdInfo:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252b
>>>>>>>>>>>>>>>>>>>>c2
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> 6
>>>>>>>>>>>>>>>>>>>> 5
>>>>>>>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>>> ","
>>>>>>>>>>>>>>>>>>>> ses
>>>>>>>>>>>>>>>>>>>> si
>>>>>>>>>>>>>>>>>>>> onk
>>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2",
>>>>>>>>>>>>>>>>>>>>"_
>>>>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>>>>> 1
>>>>>>>>>>>>>>>>>>>> 369
>>>>>>>>>>>>>>>>>>>> 405
>>>>>>>>>>>>>>>>>>>> 98
>>>>>>>>>>>>>>>>>>>> 077
>>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"},
>>>>>>>>>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null,
>>>>>>>>>>>> status:
>>>>>>>>>>>>>> 0,
>>>>>>>>>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null,
>>>>>>>>>>>>>>>>>>>>initMsid:
>>>>>>>>>>>>>>>>> 112939138816,
>>>>>>>>>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled:
>>>>>>>>>>>>>>>>>>>>null,
>>>>>>>>>>>>>> created:
>>>>>>>>>>>>>>>>> null}
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG
>>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Executing
>>>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd
>>>>>>>>>>>>>>>>>>>> for job-230
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO
>>>>>>>>>>>> [agent.manager.AgentManagerImpl]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host
>>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>>> it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the correct state: host=20; Status=Disconnected
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN
>>>>>>>>>>>> [api.commands.ReconnectHostCmd]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Exception:
>>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(Reconne
>>>>>>>>>>>>>>>>>>>>ct
>>>>>>>>>>>>>>>>>>>> H
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>>> Cmd
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>>>> :10
>>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManag
>>>>>>>>>>>>>>>>>>>>er
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> m
>>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> va:
>>>>>>>>>>>>>>>>>>>> 43
>>>>>>>>>>>>>>>>>>>> 2)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Exe
>>>>>>>>>>>>>>>>>>>>cu
>>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> va:
>>>>>>>>>>>>>>>>>>>> 47
>>>>>>>>>>>>>>>>>>>> 1)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.jav
>>>>>>>>>>>>>>>a:
>>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.FutureTask.run(FutureTask.java:166
>>>>>>>>>>>>>>>>>>>>)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(Threa
>>>>>>>>>>>>>>>>>>>>dP
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>>>>>> xec
>>>>>>>>>>>>>>>>>>>> uto
>>>>>>>>>>>>>>>>>>>> r.
>>>>>>>>>>>>>>>>>>>> jav
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> :1110)
>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre
>>>>>>>>>>>>>>>>>>>>ad
>>>>>>>>>>>>>>>>>>>> P
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>>> Exe
>>>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>>> a:603)
>>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679)
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN
>>>>>>>>>>>>>>>>>>>>[cloud.api.ApiDispatcher]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) class
>>>>>>>>>>>>>> com.cloud.api.ServerApiException :
>>>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG
>>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230,
>>>>>>>>>>>> jobStatus: 2,
>>>>>>>>>>>>>>>>>>>> resultCode:
>>>>>>>>>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null
>>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG
>>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> catalina.out
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> INFO  [agent.manager.AgentManagerImpl]
>>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> disconnect host because it is not in the correct
>>>>>>>>>>>>>>>>>>>>state:
>>>>>>>>>>>> host=20;
>>>>>>>>>>>>>>>>>>>> Status=Disconnected
>>>>>>>>>>>>>>>>>>>> WARN  [api.commands.ReconnectHostCmd]
>>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>>>>>>>>>>> Exception:
>>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(Reconne
>>>>>>>>>>>>>>>>>>>>ct
>>>>>>>>>>>>>>>>>>>> H
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>>> Cmd
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>>>> :10
>>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:13
>>>>>>>>>>>>>>>>>8)
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> v
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>> 432
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors
>>>>>>>>>>>>>>.j
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> v
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> :
>>>>>>>>>>>>>> 471
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java
>>>>>>>>>>>>>>:3
>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(Threa
>>>>>>>>>>>>>>>>>>>>dP
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>>>>>> xec
>>>>>>>>>>>>>>>>>>>> uto
>>>>>>>>>>>>>>>>>>>> r.
>>>>>>>>>>>>>>>>>>>> jav
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> :1110)
>>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre
>>>>>>>>>>>>>>>>>>>>ad
>>>>>>>>>>>>>>>>>>>> P
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>>> Exe
>>>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>>> a:603)
>>>>>>>>>>>>>>>>>>>>        at java.lang.Thread.run(Thread.java:679)
>>>>>>>>>>>>>>>>>>>> WARN  [cloud.api.ApiDispatcher]
>>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException : null
>> 


Mime
View raw message