cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmad Emneina <aemne...@gmail.com>
Subject Re: Xenserver hypervisior will not reconnect after reboot
Date Tue, 28 May 2013 21:43:40 GMT
oh this looks like a bug. A potential workaround for it is marking the host
as Up in the cloudstack db. I'd make a backup first then modify the db
entry.


On Tue, May 28, 2013 at 2:36 PM, Old, Curtis <Curtis.Old@neustar.biz> wrote:

>   Tailed last 3000 lines right after trying a forced reconnect:
> http://paste.cloudstack.org/hTOm/
>
>  Listed as Management log N*,
>
>   Author Curtis Old
>
>
>   Can we see the latest logs? paste.cloudstack.org
>
>
> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis <Curtis.Old@neustar.biz>wrote:
>
>> Tried that already (like 2 times)
>>
>> Curtis Old
>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>> 46000 Center Oak Plaza Sterling, VA 20166
>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis.old@neustar.biz
>> / www.neustar.biz <http://www.neustar.biz/>
>> ________________________________________
>>
>>
>> The information contained in this e-mail message is intended only for the
>> use of the recipient(s) named above and may contain confidential and/or
>> privileged information. If you are not the intended recipient you have
>> received this e-mail message in error and any review, dissemination,
>> distribution, or copying of this message is strictly prohibited. If you
>> have received this communication in error, please notify us immediately
>> and delete the original message.
>>
>>
>>
>>
>>
>>  On 5/28/13 5:12 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>>
>> >how about a reboot of the host, then force reconnect.
>> >
>> >
>> >On Tue, May 28, 2013 at 2:05 PM, Old, Curtis <Curtis.Old@neustar.biz>
>> >wrote:
>> >
>> >> Bummer no goodÅ ...
>> >>
>> >> Curtis Old
>> >> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>> >> 46000 Center Oak Plaza Sterling, VA 20166
>> >> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>> curtis.old@neustar.biz
>> >> / www.neustar.biz <http://www.neustar.biz/>
>> >> ________________________________________
>> >>
>> >>
>> >> The information contained in this e-mail message is intended only for
>> >>the
>> >> use of the recipient(s) named above and may contain confidential and/or
>> >> privileged information. If you are not the intended recipient you have
>> >> received this e-mail message in error and any review, dissemination,
>> >> distribution, or copying of this message is strictly prohibited. If you
>> >> have received this communication in error, please notify us immediately
>> >> and delete the original message.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 5/28/13 5:01 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>> >>
>> >> >dont do that :D do a toolstack restart on the xenserver host, and a
>> >>force
>> >> >reconnect from cloudstack. see if that makes them play nice.
>> >> >
>> >> >
>> >> >On Tue, May 28, 2013 at 1:58 PM, Old, Curtis <Curtis.Old@neustar.biz>
>> >> >wrote:
>> >> >
>> >> >>   Well the hypervisior looks fine from a Xenserver prospective
but
>> >> >> CloudStack won't "reconnect" it.  I thought about deleting it in
CS
>> >>and
>> >> >> re-adding it, but that bugs me lol
>> >> >>
>> >> >>  *Curtis Old**
>> >> >> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer*
>> >> >>
>> >> >> *46000 Center Oak Plaza Sterling, VA 20166**
>> >> >> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** **/**
>> >>**
>> >> >> curtis.old@neustar.biz**  / **www.neustar.biz**     *
>> >> >>  *
>> >> >> ------------------------------
>> >> >> *
>> >> >>
>> >> >> *The information contained in this e-mail message is intended only
>> >>for
>> >> >> the use of the recipient(s) named above and may contain confidential
>> >> >>and/or
>> >> >> privileged information. If you are not the intended recipient you
>> >>have
>> >> >> received this e-mail message in error and any review, dissemination,
>> >> >> distribution, or copying of this message is strictly prohibited.
If
>> >>you
>> >> >> have received this communication in error, please notify us
>> >>immediately
>> >> >>and
>> >> >> delete the original message.*
>> >> >>
>> >> >>   From: Ahmad Emneina <aemneina@gmail.com>
>> >> >> Reply-To: "aemneina@gmail.com" <aemneina@gmail.com>
>> >> >> Date: Tuesday, May 28, 2013 4:56 PM
>> >> >> To: "Old, Curtis" <curtis.old@neustar.biz>
>> >> >> Subject: Re: Xenserver hypervisior will not reconnect after reboot
>> >> >>
>> >> >>   are you good to go again, after cleaning up some space?
>> >> >>
>> >> >>
>> >> >> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis
>> >> >><Curtis.Old@neustar.biz>wrote:
>> >> >>
>> >> >>> Sorry the log entries are from the xenserver log
>> >> >>>
>> >> >>> Curtis Old
>> >> >>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
>> >> >>> 46000 Center Oak Plaza Sterling, VA 20166
>> >> >>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>> >> >>>curtis.old@neustar.biz
>> >> >>> / www.neustar.biz <http://www.neustar.biz/>
>> >> >>> ________________________________________
>> >> >>>
>> >> >>>
>> >> >>> The information contained in this e-mail message is intended
only
>> >>for
>> >> >>>the
>> >> >>> use of the recipient(s) named above and may contain confidential
>> >>and/or
>> >> >>> privileged information. If you are not the intended recipient
you
>> >>have
>> >> >>> received this e-mail message in error and any review,
>> dissemination,
>> >> >>> distribution, or copying of this message is strictly prohibited.
If
>> >>you
>> >> >>> have received this communication in error, please notify us
>> >>immediately
>> >> >>> and delete the original message.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>  On 5/28/13 4:20 PM, "Old, Curtis" <Curtis.Old@neustar.biz>
wrote:
>> >> >>>
>> >> >>> >The in innodes did fill, so after I cleaned them up I rebooted
and
>> >> >>>then
>> >> >>> >the host was singing and dancing again in XenCenter, and
primary
>> >> >>>storage
>> >> >>> >was mounted.  What from the management log, if we can narrow
it
>> >>down?
>> >> >>> >When I try the forced reconnect only this shows up:
>> >> >>> >
>> >> >>> >[20130528T20:17:39.118Z|debug|lab-cloud-9|8622
>> >> >>>inet_rpc||http_critical]
>> >> >>> >Connection terminated
>> >> >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false)
connected
>> >>to
>> >> >>> >10.31.105.158:443
>> >> >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
>> >> >>>s_pid=23063
>> >> >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false)
connected
>> >>to
>> >> >>> >10.31.105.158:443
>> >> >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
>> >> >>>s_pid=23063
>> >> >>> >[20130528T20:18:39.124Z|debug|lab-cloud-9|8623
>> >> >>>inet_rpc||http_critical]
>> >> >>> >Connection terminated
>> >> >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false)
connected
>> >>to
>> >> >>> >10.31.105.158:443
>> >> >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
>> >> >>>s_pid=23068
>> >> >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false)
connected
>> >>to
>> >> >>> >10.31.105.158:443
>> >> >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat
>> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
>> >> >>>s_pid=23072
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >Curtis Old
>> >> >>> >Neustar, Inc. / Neustarlabs / Senior Research Engineer
>> >> >>> >46000 Center Oak Plaza Sterling, VA 20166
>> >> >>> >Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
>> >> >>>curtis.old@neustar.biz
>> >> >>> >/ www.neustar.biz <http://www.neustar.biz/>
>> >> >>> >________________________________________
>> >> >>> >
>> >> >>> >
>> >> >>> >The information contained in this e-mail message is intended
only
>> >>for
>> >> >>>the
>> >> >>> >use of the recipient(s) named above and may contain confidential
>> >> >>>and/or
>> >> >>> >privileged information. If you are not the intended recipient
you
>> >>have
>> >> >>> >received this e-mail message in error and any review,
>> >>dissemination,
>> >> >>> >distribution, or copying of this message is strictly prohibited.
>> If
>> >> >>>you
>> >> >>> >have received this communication in error, please notify
us
>> >> >>>immediately
>> >> >>> >and delete the original message.
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >On 5/28/13 4:08 PM, "Ahmad Emneina" <aemneina@gmail.com>
wrote:
>> >> >>> >
>> >> >>> >>pretty odd... how does the disk space look on the xenserver
host?
>> >> >>>we'd
>> >> >>> >>need
>> >> >>> >>more from the management log, as well as the xensource.log
off
>> the
>> >> >>> >>hypervisor.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>On Tue, May 28, 2013 at 9:21 AM, Old, Curtis
>> >><Curtis.Old@neustar.biz
>> >> >
>> >> >>> >>wrote:
>> >> >>> >>
>> >> >>> >>>
>> >> >>> >>> Tried the force reconnect from CS 4.0.0 UI and
I get "Command
>> >> >>>Failed
>> >> >>> >>>due
>> >> >>> >>> to internal server Error" any ideas?
>> >> >>> >>>
>> >> >>> >>> management-server.log
>> >> >>> >>>
>> >> >>> >>> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >>> >>> (catalina-exec-7:null) submit async job-230, details:
>> AsyncJobVO
>> >> >>> >>>{id:230,
>> >> >>> >>> userId: 2, accountId: 2, sessionKey: null, instanceType:
Host,
>> >> >>> >>>instanceId:
>> >> >>> >>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd,
>> cmdOriginator:
>> >> >>>null,
>> >> >>> >>> cmdInfo:
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","ses
>> >>>>>>>>si
>> >> >>>>>>onk
>> >> >>> >>>e
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"1369405
>> >>>>>>>>98
>> >> >>>>>>077
>> >> >>> >>>8
>> >> >>> >>>","ctxAccountId":"2","ctxStartEventId":"1514"},
>> >> >>> >>> cmdVersion: 0, callbackType: 0, callbackAddress:
null, status:
>> >>0,
>> >> >>> >>> processStatus: 0, resultCode: 0, result: null,
initMsid:
>> >> >>>112939138816,
>> >> >>> >>> completeMsid: null, lastUpdated: null, lastPolled:
null,
>> >>created:
>> >> >>> null}
>> >> >>> >>> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >>> >>> (Job-Executor-16:job-230) Executing
>> >> >>> >>>com.cloud.api.commands.ReconnectHostCmd
>> >> >>> >>> for job-230
>> >> >>> >>> 2013-05-24 10:33:00,985 INFO  [agent.manager.AgentManagerImpl]
>> >> >>> >>> (Job-Executor-16:job-230) Unable to disconnect
host because it
>> >>is
>> >> >>>not
>> >> >>> >>>in
>> >> >>> >>> the correct state: host=20; Status=Disconnected
>> >> >>> >>> 2013-05-24 10:33:00,986 WARN  [api.commands.ReconnectHostCmd]
>> >> >>> >>> (Job-Executor-16:job-230) Exception:
>> >> >>> >>> com.cloud.api.ServerApiException
>> >> >>> >>> at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.ja
>> >>>>>>>>va
>> >> >>>>>>:10
>> >> >>> >>>8
>> >> >>> >>>)
>> >> >>> >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>> >> >>> >>> at
>> >> >>>
>> >>
>>
>> >>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:
>> >>>>>>>>43
>> >> >>>>>>2)
>> >> >>> >>> at
>> >> >>>
>> >>
>>
>> >>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
>> >>>>>>>>47
>> >> >>>>>>1)
>> >> >>> >>> at
>> >> >>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >> >>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >> >>> >>> at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
>> >>>>>>>>r.
>> >> >>>>>>jav
>> >> >>> >>>a
>> >> >>> >>>:1110)
>> >> >>> >>> at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
>> >>>>>>>>or
>> >> >>>>>>.ja
>> >> >>> >>>v
>> >> >>> >>>a:603)
>> >> >>> >>> at java.lang.Thread.run(Thread.java:679)
>> >> >>> >>> 2013-05-24 10:33:00,986 WARN  [cloud.api.ApiDispatcher]
>> >> >>> >>> (Job-Executor-16:job-230) class
>> >>com.cloud.api.ServerApiException :
>> >> >>> null
>> >> >>> >>> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >>> >>> (Job-Executor-16:job-230) Complete async job-230,
jobStatus: 2,
>> >> >>> >>>resultCode:
>> >> >>> >>> 530, result: Error Code: 534 Error text: null
>> >> >>> >>> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >>> >>> (catalina-exec-4:null) Async job-230 completed
>> >> >>> >>>
>> >> >>> >>> catalina.out
>> >> >>> >>>
>> >> >>> >>> INFO  [agent.manager.AgentManagerImpl]
>> (Job-Executor-16:job-230)
>> >> >>> Unable
>> >> >>> >>>to
>> >> >>> >>> disconnect host because it is not in the correct
state:
>> host=20;
>> >> >>> >>> Status=Disconnected
>> >> >>> >>> WARN  [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230)
>> >> >>> >>>Exception:
>> >> >>> >>> com.cloud.api.ServerApiException
>> >> >>> >>>         at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.ja
>> >>>>>>>>va
>> >> >>>>>>:10
>> >> >>> >>>8
>> >> >>> >>>)
>> >> >>> >>>         at
>> >> >>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>> >> >>> >>>         at
>> >> >>> >>>
>> >> >>>
>> >>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
>> >> >>> >>>         at
>> >> >>> >>>
>> >> >>>
>> >>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >> >>> >>>         at
>> >> >>> >>>
>> >>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >> >>> >>>         at
>> >>java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >> >>> >>>         at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
>> >>>>>>>>r.
>> >> >>>>>>jav
>> >> >>> >>>a
>> >> >>> >>>:1110)
>> >> >>> >>>         at
>> >> >>> >>>
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
>> >>>>>>>>or
>> >> >>>>>>.ja
>> >> >>> >>>v
>> >> >>>   >>>a:603)
>> >> >>> >>>         at java.lang.Thread.run(Thread.java:679)
>> >> >>> >>> WARN  [cloud.api.ApiDispatcher] (Job-Executor-16:job-230)
class
>> >> >>> >>> com.cloud.api.ServerApiException : null
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >
>> >> >>>
>> >> >>>
>> >> >>
>> >>
>> >>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message