cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmad Emneina <aemne...@gmail.com>
Subject Re: Xenserver hypervisior will not reconnect after reboot
Date Thu, 30 May 2013 14:30:49 GMT
Did you reinstall the hypervisor before reconnecting it back to the pool?

Ahmad

On May 30, 2013, at 4:10 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:

> Ok both are in an "Alert" state.  So I'M kinda at a lost for what to try
> next.
> 
> 
> On 5/30/13 6:55 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
> 
>> Ok totally weird stuff now.
>> 
>> After trying the previous steps, both Xenserver nodes failed to reconnect
>> to the cluster, so I decided to remove and re-add them.  I performed the
>> following:
>> 
>>  1.  Unmanaged the cluster
>>  2.  Removed the 2 nodes
>>  3.  Managed the cluster, the 3 remaining nodes went into an Up state.
>>  4.  Add 1 of the Xenserver nodes, and was going to wait to add the last
>> one, if the 1st added successfully.
>> 
>> The result I see both of the 2 nodes trying to connect (one I didn't add
>> yet), waiting to see if the complete or fail.
>> 
>> 
>> 
>> On 5/29/13 8:39 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>> 
>>> Same status after an hour.  Should I delete them and re-add them or ??
>>> 
>>> 
>>> 
>>> On 5/29/13 7:30 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>>> 
>>>> 
>>>> 
>>>> Well one lab-cloud-9 is in an "Alert" State the other, lab-cloud-10 is
>>>> "Disconnected", and after a refresh on the UI lab-cloud-9 is back to
>>>> "Connecting"
>>>> 
>>>> 
>>>> Suggestions?
>>>> 
>>>> 
>>>> 
>>>> On 5/29/13 7:17 AM, "Old, Curtis" <Curtis.Old@neustar.biz> wrote:
>>>> 
>>>>> Interesting the UUID's in the database where different then the
>>>>> Xenserver
>>>>> UUID's (there where 2 Xenservers hosts that had inode problems) But
>>>>> then
>>>>> again the other servers all had different UUID's as well <shrug>.
>>>>> 
>>>>> Mysql DB:
>>>>> 
>>>>>                  id: 21
>>>>>                name: lab-cloud-10.neustarlab.biz
>>>>>                uuid: 5c65a56a-5a71-44e8-b4ae-89bf81ef4d24
>>>>> 
>>>>> And on the Host:
>>>>> 
>>>>> 
>>>>> uuid=dde1a6d7-8708-451f-a4a8-a9c5be79f57e
>>>>> 
>>>>> It has been 20 to 25 min and one lab-cloud-9 its state is "Connecting"
>>>>> lab-cloud-10 is in "Disconnected" and the other 3 hosts (lab-cloud-6,
>>>>> 7,
>>>>> 8) are "Up"
>>>>> 
>>>>> Wait a bit longer?
>>>>> 
>>>>> 
>>>>> 
>>>>> On 5/28/13 6:37 PM, "Kirk Kosinski" <kirkkosinski@gmail.com> wrote:
>>>>> 
>>>>>> Before hacking the db, I suggest trying one more thing.
>>>>>> 
>>>>>> 1. In the CS UI, unmanage the cluster with the affected host.
>>>>>> 2. From the CLI of a host in the XS pool, clear the host tags of the
>>>>>> affected host.
>>>>>> xe host-param-clear param-name=tags uuid=<UUID of affected host>
>>>>>> 3. In the CS UI, manage the cluster with the affected host.
>>>>>> 
>>>>>> This usually solves this type of problem (it might take a couple
>>>>>> minutes).  Clearing host tags won't affect running VMs (if any).
>>>>>> 
>>>>>> Best regards,
>>>>>> Kirk
>>>>>> 
>>>>>> On 05/28/2013 03:03 PM, Ahmad Emneina wrote:
>>>>>>> first, I'd query to make sure its the right host, judging from the
>>>>>>> logs
>>>>>>> it
>>>>>>> has an id of 20... execute the following
>>>>>>> backup command from the cli that can access the db server:
>>>>>>> 
>>>>>>> mysqldump <credentials if necessary> --databases cloud > cloud.sql
>>>>>>> 
>>>>>>> Then start operating...
>>>>>>> 
>>>>>>> select * from host where id=20\G (hit the enter key)
>>>>>>> 
>>>>>>> if this is infact the misbehaving host, execute
>>>>>>> 
>>>>>>> update host set status='Up' where id=20; (again hit enter)
>>>>>>> 
>>>>>>> then try to launch a vm on that host (in the deployVirtualMachine
>>>>>>> api
>>>>>>> command you can/should be able to specify the exact host)
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, May 28, 2013 at 2:46 PM, Old, Curtis
>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Oh god where do I find that sucker in the DB?  Sorry I'M an admin
>>>>>>>> type
>>>>>>>> that isn't a mysql guru or Ji
>>>>>>>> 
>>>>>>>> Curtis Old
>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>> curtis.old@neustar.biz
>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>> ________________________________________
>>>>>>>> 
>>>>>>>> 
>>>>>>>> The information contained in this e-mail message is intended only
>>>>>>>> for
>>>>>>>> the
>>>>>>>> use of the recipient(s) named above and may contain confidential
>>>>>>>> and/or
>>>>>>>> privileged information. If you are not the intended recipient you
>>>>>>>> have
>>>>>>>> received this e-mail message in error and any review,
>>>>>>>> dissemination,
>>>>>>>> distribution, or copying of this message is strictly prohibited. If
>>>>>>>> you
>>>>>>>> have received this communication in error, please notify us
>>>>>>>> immediately
>>>>>>>> and delete the original message.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 5/28/13 5:43 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> oh this looks like a bug. A potential workaround for it is marking
>>>>>>>>> the
>>>>>>>>> host
>>>>>>>>> as Up in the cloudstack db. I'd make a backup first then modify
>>>>>>>>> the
>>>>>>>>> db
>>>>>>>>> entry.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis
>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>>  Tailed last 3000 lines right after trying a forced reconnect:
>>>>>>>>>> http://paste.cloudstack.org/hTOm/
>>>>>>>>>> 
>>>>>>>>>> Listed as Management log N*,
>>>>>>>>>> 
>>>>>>>>>>  Author Curtis Old
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>  Can we see the latest logs? paste.cloudstack.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis
>>>>>>>>>> <Curtis.Old@neustar.biz>wrote:
>>>>>>>>>> 
>>>>>>>>>>> Tried that already (like 2 times)
>>>>>>>>>>> 
>>>>>>>>>>> Curtis Old
>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> The information contained in this e-mail message is intended
>>>>>>>>>>> only
>>>>>>>>>>> for
>>>>>>>>>>> the
>>>>>>>>>>> use of the recipient(s) named above and may contain confidential
>>>>>>>>>>> and/or
>>>>>>>>>>> privileged information. If you are not the intended recipient
>>>>>>>>>>> you
>>>>>>>>>>> have
>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>> dissemination,
>>>>>>>>>>> distribution, or copying of this message is strictly prohibited.
>>>>>>>>>>> If
>>>>>>>>>>> you
>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>> immediately
>>>>>>>>>>> and delete the original message.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> how about a reboot of the host, then force reconnect.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis
>>>>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Bummer no goodÅ ...
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The information contained in this e-mail message is intended
>>>>>>>>>>>>> only
>>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>>> confidential
>>>>>>>>>>> and/or
>>>>>>>>>>>>> privileged information. If you are not the intended recipient
>>>>>>>>>>>>> you
>>>>>>>>>>> have
>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>>> prohibited.
>>>>>>>>>>>>> If
>>>>>>>>>>> you
>>>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>> immediately
>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <aemneina@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> dont do that :D do a toolstack restart on the xenserver host,
>>>>>>>>>>>>>> and a
>>>>>>>>>>>>> force
>>>>>>>>>>>>>> reconnect from cloudstack. see if that makes them play nice.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis
>>>>>>>>>>> <Curtis.Old@neustar.biz>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>  Well the hypervisior looks fine from a Xenserver
>>>>>>>>>>>>>>> prospective
>>>>>>>>>>> but
>>>>>>>>>>>>>>> CloudStack won't "reconnect" it.  I thought about deleting
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> in
>>>>>>>>>>> CS
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> re-adding it, but that bugs me lol
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Curtis Old**
>>>>>>>>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166**
>>>>>>>>>>>>>>> **Office:** **+1.571.434.5384** **Mobile:
>>>>>>>>>>>>>>> **+1.540.532.2230**
>>>>>>>>>>> **/**
>>>>>>>>>>>>> **
>>>>>>>>>>>>>>> curtis.old@neustar.biz**  / **www.neustar.biz**     *
>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *The information contained in this e-mail message is
>>>>>>>>>>>>>>> intended
>>>>>>>>>>> only
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> the use of the recipient(s) named above and may contain
>>>>>>>>>>> confidential
>>>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>> recipient
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>>>>>> prohibited.
>>>>>>>>>>> If
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> delete the original message.*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>  From: Ahmad Emneina <aemneina@gmail.com>
>>>>>>>>>>>>>>> Reply-To: "aemneina@gmail.com" <aemneina@gmail.com>
>>>>>>>>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM
>>>>>>>>>>>>>>> To: "Old, Curtis" <curtis.old@neustar.biz>
>>>>>>>>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect after
>>>>>>>>>>> reboot
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>  are you good to go again, after cleaning up some space?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis
>>>>>>>>>>>>>>> <Curtis.Old@neustar.biz>wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sorry the log entries are from the xenserver log
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The information contained in this e-mail message is
>>>>>>>>>>>>>>>> intended
>>>>>>>>>>> only
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>>>>>>> confidential
>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>>> recipient
>>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>> prohibited. If
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> have received this communication in error, please notify us
>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis" <Curtis.Old@neustar.biz>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The in innodes did fill, so after I cleaned them up I
>>>>>>>>>>>>>>>>> rebooted
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>> the host was singing and dancing again in XenCenter, and
>>>>>>>>>>> primary
>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>> was mounted.  What from the management log, if we can
>>>>>>>>>>>>>>>>> narrow
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>> down?
>>>>>>>>>>>>>>>>> When I try the forced reconnect only this shows up:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622
>>>>>>>>>>>>>>>> inet_rpc||http_critical]
>>>>>>>>>>>>>>>>> Connection terminated
>>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false)
>>>>>>>>>>> connected
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>> s_pid=23063
>>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false)
>>>>>>>>>>> connected
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>> s_pid=23063
>>>>>>>>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623
>>>>>>>>>>>>>>>> inet_rpc||http_critical]
>>>>>>>>>>>>>>>>> Connection terminated
>>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false)
>>>>>>>>>>> connected
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>> s_pid=23068
>>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false)
>>>>>>>>>>> connected
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> 10.31.105.158:443
>>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28
>>>>>>>>>>> heartbeat|Heartbeat
>>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid
>>>>>>>>>>>>>>>>> task_opt=None
>>>>>>>>>>>>>>>> s_pid=23072
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Curtis Old
>>>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>>>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166
>>>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>>>>>>>>>>>>>>>> curtis.old@neustar.biz
>>>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The information contained in this e-mail message is
>>>>>>>>>>>>>>>>> intended
>>>>>>>>>>> only
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain
>>>>>>>>>>> confidential
>>>>>>>>>>>>>>>> and/or
>>>>>>>>>>>>>>>>> privileged information. If you are not the intended
>>>>>>>>>>>>>>>>> recipient
>>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> received this e-mail message in error and any review,
>>>>>>>>>>>>> dissemination,
>>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly
>>>>>>>>>>> prohibited.
>>>>>>>>>>> If
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> have received this communication in error, please notify
>>>>>>>>>>>>>>>>> us
>>>>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>>> and delete the original message.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <aemneina@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> pretty odd... how does the disk space look on the
>>>>>>>>>>>>>>>>>> xenserver
>>>>>>>>>>> host?
>>>>>>>>>>>>>>>> we'd
>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>> more from the management log, as well as the
>>>>>>>>>>>>>>>>>> xensource.log
>>>>>>>>>>>>>>>>>> off
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> hypervisor.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis
>>>>>>>>>>>>> <Curtis.Old@neustar.biz
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get
>>>>>>>>>>> "Command
>>>>>>>>>>>>>>>> Failed
>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>> to internal server Error" any ideas?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> management-server.log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG
>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details:
>>>>>>>>>>> AsyncJobVO
>>>>>>>>>>>>>>>>>>> {id:230,
>>>>>>>>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null, instanceType:
>>>>>>>>>>> Host,
>>>>>>>>>>>>>>>>>>> instanceId:
>>>>>>>>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd,
>>>>>>>>>>> cmdOriginator:
>>>>>>>>>>>>>>>> null,
>>>>>>>>>>>>>>>>>>> cmdInfo:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> {"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> 6
>>>>>>>>>>>>>>>>>>> 5
>>>>>>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>>>>>>> ","
>>>>>>>>>>>>>>>>>>> ses
>>>>>>>>>>>>>>>>>>> si
>>>>>>>>>>>>>>>>>>> onk
>>>>>>>>>>>>>>>>>>> e
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_
>>>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>>>> 1
>>>>>>>>>>>>>>>>>>> 369
>>>>>>>>>>>>>>>>>>> 405
>>>>>>>>>>>>>>>>>>> 98
>>>>>>>>>>>>>>>>>>> 077
>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"},
>>>>>>>>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null,
>>>>>>>>>>> status:
>>>>>>>>>>>>> 0,
>>>>>>>>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>>>>>>>>>>>>>> 112939138816,
>>>>>>>>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled: null,
>>>>>>>>>>>>> created:
>>>>>>>>>>>>>>>> null}
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG
>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Executing
>>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd
>>>>>>>>>>>>>>>>>>> for job-230
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO
>>>>>>>>>>> [agent.manager.AgentManagerImpl]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host
>>>>>>>>>>>>>>>>>>> because
>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> the correct state: host=20; Status=Disconnected
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN
>>>>>>>>>>> [api.commands.ReconnectHostCmd]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Exception:
>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd.execute(Reconnect
>>>>>>>>>>>>>>>>>>> H
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> Cmd
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>>> :10
>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManager
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> m
>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> va:
>>>>>>>>>>>>>>>>>>> 43
>>>>>>>>>>>>>>>>>>> 2)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Execu
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> r
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> va:
>>>>>>>>>>>>>>>>>>> 47
>>>>>>>>>>>>>>>>>>> 1)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:
>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>> 3
>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>>>>> xec
>>>>>>>>>>>>>>>>>>> uto
>>>>>>>>>>>>>>>>>>> r.
>>>>>>>>>>>>>>>>>>> jav
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> :1110)
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread
>>>>>>>>>>>>>>>>>>> P
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> Exe
>>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>> a:603)
>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679)
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN  [cloud.api.ApiDispatcher]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) class
>>>>>>>>>>>>> com.cloud.api.ServerApiException :
>>>>>>>>>>>>>>>> null
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG
>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230,
>>>>>>>>>>> jobStatus: 2,
>>>>>>>>>>>>>>>>>>> resultCode:
>>>>>>>>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null
>>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG
>>>>>>>>>>> [cloud.async.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> catalina.out
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> INFO  [agent.manager.AgentManagerImpl]
>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> disconnect host because it is not in the correct state:
>>>>>>>>>>> host=20;
>>>>>>>>>>>>>>>>>>> Status=Disconnected
>>>>>>>>>>>>>>>>>>> WARN  [api.commands.ReconnectHostCmd]
>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>>>>>>>>>> Exception:
>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd.execute(Reconnect
>>>>>>>>>>>>>>>>>>> H
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> s
>>>>>>>>>>>>>>>>>>> t
>>>>>>>>>>>>>>>>>>> Cmd
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>>> :10
>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.j
>>>>>>>>>>>>> a
>>>>>>>>>>>>> v
>>>>>>>>>>>>> a
>>>>>>>>>>>>> :
>>>>>>>>>>>>> 432
>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.j
>>>>>>>>>>>>> a
>>>>>>>>>>>>> v
>>>>>>>>>>>>> a
>>>>>>>>>>>>> :
>>>>>>>>>>>>> 471
>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>> 
>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:3
>>>>>>>>>>>>> 3
>>>>>>>>>>>>> 4
>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> E
>>>>>>>>>>>>>>>>>>> xec
>>>>>>>>>>>>>>>>>>> uto
>>>>>>>>>>>>>>>>>>> r.
>>>>>>>>>>>>>>>>>>> jav
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> :1110)
>>>>>>>>>>>>>>>>>>>        at
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread
>>>>>>>>>>>>>>>>>>> P
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> o
>>>>>>>>>>>>>>>>>>> l
>>>>>>>>>>>>>>>>>>> Exe
>>>>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> .ja
>>>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>>> a:603)
>>>>>>>>>>>>>>>>>>>        at java.lang.Thread.run(Thread.java:679)
>>>>>>>>>>>>>>>>>>> WARN  [cloud.api.ApiDispatcher]
>>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230)
>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException : null
> 

Mime
View raw message