cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmad Emneina <aemne...@gmail.com>
Subject Re: Xenserver hypervisior will not reconnect after reboot
Date Tue, 28 May 2013 21:18:46 GMT
Can we see the latest logs? paste.cloudstack.org


On Tue, May 28, 2013 at 2:15 PM, Old, Curtis <Curtis.Old@neustar.biz> wrote:

> Tried that already (like 2 times)
>
> Curtis Old
> Neustar, Inc. / Neustarlabs / Senior Research Engineer
> 46000 Center Oak Plaza Sterling, VA 20166
> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis.old@neustar.biz
> / www.neustar.biz <http://www.neustar.biz/>
> ________________________________________
>
>
> The information contained in this e-mail message is intended only for the
> use of the recipient(s) named above and may contain confidential and/or
> privileged information. If you are not the intended recipient you have
> received this e-mail message in error and any review, dissemination,
> distribution, or copying of this message is strictly prohibited. If you
> have received this communication in error, please notify us immediately
> and delete the original message.
>
>
>
>
>
> On 5/28/13 5:12 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
>
> >how about a reboot of the host, then force reconnect.
> >
> >
> >On Tue, May 28, 2013 at 2:05 PM, Old, Curtis <Curtis.Old@neustar.biz>
> >wrote:
> >
> >> Bummer no goodÅ ...
> >>
> >> Curtis Old
> >> Neustar, Inc. / Neustarlabs / Senior Research Engineer
> >> 46000 Center Oak Plaza Sterling, VA 20166
> >> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
> curtis.old@neustar.biz
> >> / www.neustar.biz <http://www.neustar.biz/>
> >> ________________________________________
> >>
> >>
> >> The information contained in this e-mail message is intended only for
> >>the
> >> use of the recipient(s) named above and may contain confidential and/or
> >> privileged information. If you are not the intended recipient you have
> >> received this e-mail message in error and any review, dissemination,
> >> distribution, or copying of this message is strictly prohibited. If you
> >> have received this communication in error, please notify us immediately
> >> and delete the original message.
> >>
> >>
> >>
> >>
> >>
> >> On 5/28/13 5:01 PM, "Ahmad Emneina" <aemneina@gmail.com> wrote:
> >>
> >> >dont do that :D do a toolstack restart on the xenserver host, and a
> >>force
> >> >reconnect from cloudstack. see if that makes them play nice.
> >> >
> >> >
> >> >On Tue, May 28, 2013 at 1:58 PM, Old, Curtis <Curtis.Old@neustar.biz>
> >> >wrote:
> >> >
> >> >>   Well the hypervisior looks fine from a Xenserver prospective but
> >> >> CloudStack won't "reconnect" it.  I thought about deleting it in CS
> >>and
> >> >> re-adding it, but that bugs me lol
> >> >>
> >> >>  *Curtis Old**
> >> >> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer*
> >> >>
> >> >> *46000 Center Oak Plaza Sterling, VA 20166**
> >> >> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** **/**
> >>**
> >> >> curtis.old@neustar.biz**  / **www.neustar.biz**     *
> >> >>  *
> >> >> ------------------------------
> >> >> *
> >> >>
> >> >> *The information contained in this e-mail message is intended only
> >>for
> >> >> the use of the recipient(s) named above and may contain confidential
> >> >>and/or
> >> >> privileged information. If you are not the intended recipient you
> >>have
> >> >> received this e-mail message in error and any review, dissemination,
> >> >> distribution, or copying of this message is strictly prohibited. If
> >>you
> >> >> have received this communication in error, please notify us
> >>immediately
> >> >>and
> >> >> delete the original message.*
> >> >>
> >> >>   From: Ahmad Emneina <aemneina@gmail.com>
> >> >> Reply-To: "aemneina@gmail.com" <aemneina@gmail.com>
> >> >> Date: Tuesday, May 28, 2013 4:56 PM
> >> >> To: "Old, Curtis" <curtis.old@neustar.biz>
> >> >> Subject: Re: Xenserver hypervisior will not reconnect after reboot
> >> >>
> >> >>   are you good to go again, after cleaning up some space?
> >> >>
> >> >>
> >> >> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis
> >> >><Curtis.Old@neustar.biz>wrote:
> >> >>
> >> >>> Sorry the log entries are from the xenserver log
> >> >>>
> >> >>> Curtis Old
> >> >>> Neustar, Inc. / Neustarlabs / Senior Research Engineer
> >> >>> 46000 Center Oak Plaza Sterling, VA 20166
> >> >>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
> >> >>>curtis.old@neustar.biz
> >> >>> / www.neustar.biz <http://www.neustar.biz/>
> >> >>> ________________________________________
> >> >>>
> >> >>>
> >> >>> The information contained in this e-mail message is intended only
> >>for
> >> >>>the
> >> >>> use of the recipient(s) named above and may contain confidential
> >>and/or
> >> >>> privileged information. If you are not the intended recipient you
> >>have
> >> >>> received this e-mail message in error and any review, dissemination,
> >> >>> distribution, or copying of this message is strictly prohibited.
If
> >>you
> >> >>> have received this communication in error, please notify us
> >>immediately
> >> >>> and delete the original message.
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>  On 5/28/13 4:20 PM, "Old, Curtis" <Curtis.Old@neustar.biz>
wrote:
> >> >>>
> >> >>> >The in innodes did fill, so after I cleaned them up I rebooted
and
> >> >>>then
> >> >>> >the host was singing and dancing again in XenCenter, and primary
> >> >>>storage
> >> >>> >was mounted.  What from the management log, if we can narrow
it
> >>down?
> >> >>> >When I try the forced reconnect only this shows up:
> >> >>> >
> >> >>> >[20130528T20:17:39.118Z|debug|lab-cloud-9|8622
> >> >>>inet_rpc||http_critical]
> >> >>> >Connection terminated
> >> >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) connected
> >>to
> >> >>> >10.31.105.158:443
> >> >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
> >> >>>s_pid=23063
> >> >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) connected
> >>to
> >> >>> >10.31.105.158:443
> >> >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
> >> >>>s_pid=23063
> >> >>> >[20130528T20:18:39.124Z|debug|lab-cloud-9|8623
> >> >>>inet_rpc||http_critical]
> >> >>> >Connection terminated
> >> >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) connected
> >>to
> >> >>> >10.31.105.158:443
> >> >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
> >> >>>s_pid=23068
> >> >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) connected
> >>to
> >> >>> >10.31.105.158:443
> >> >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat
> >> >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None
> >> >>>s_pid=23072
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >Curtis Old
> >> >>> >Neustar, Inc. / Neustarlabs / Senior Research Engineer
> >> >>> >46000 Center Oak Plaza Sterling, VA 20166
> >> >>> >Office: +1.571.434.5384 Mobile: +1.540.532.2230 /
> >> >>>curtis.old@neustar.biz
> >> >>> >/ www.neustar.biz <http://www.neustar.biz/>
> >> >>> >________________________________________
> >> >>> >
> >> >>> >
> >> >>> >The information contained in this e-mail message is intended
only
> >>for
> >> >>>the
> >> >>> >use of the recipient(s) named above and may contain confidential
> >> >>>and/or
> >> >>> >privileged information. If you are not the intended recipient
you
> >>have
> >> >>> >received this e-mail message in error and any review,
> >>dissemination,
> >> >>> >distribution, or copying of this message is strictly prohibited.
If
> >> >>>you
> >> >>> >have received this communication in error, please notify us
> >> >>>immediately
> >> >>> >and delete the original message.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >On 5/28/13 4:08 PM, "Ahmad Emneina" <aemneina@gmail.com>
wrote:
> >> >>> >
> >> >>> >>pretty odd... how does the disk space look on the xenserver
host?
> >> >>>we'd
> >> >>> >>need
> >> >>> >>more from the management log, as well as the xensource.log
off the
> >> >>> >>hypervisor.
> >> >>> >>
> >> >>> >>
> >> >>> >>On Tue, May 28, 2013 at 9:21 AM, Old, Curtis
> >><Curtis.Old@neustar.biz
> >> >
> >> >>> >>wrote:
> >> >>> >>
> >> >>> >>>
> >> >>> >>> Tried the force reconnect from CS 4.0.0 UI and I get
"Command
> >> >>>Failed
> >> >>> >>>due
> >> >>> >>> to internal server Error" any ideas?
> >> >>> >>>
> >> >>> >>> management-server.log
> >> >>> >>>
> >> >>> >>> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >>> >>> (catalina-exec-7:null) submit async job-230, details:
AsyncJobVO
> >> >>> >>>{id:230,
> >> >>> >>> userId: 2, accountId: 2, sessionKey: null, instanceType:
Host,
> >> >>> >>>instanceId:
> >> >>> >>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd,
cmdOriginator:
> >> >>>null,
> >> >>> >>> cmdInfo:
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","ses
> >>>>>>>>si
> >> >>>>>>onk
> >> >>> >>>e
> >> >>>
> >> >>>
> >>
> >>>>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"1369405
> >>>>>>>>98
> >> >>>>>>077
> >> >>> >>>8
> >> >>> >>>","ctxAccountId":"2","ctxStartEventId":"1514"},
> >> >>> >>> cmdVersion: 0, callbackType: 0, callbackAddress: null,
status:
> >>0,
> >> >>> >>> processStatus: 0, resultCode: 0, result: null, initMsid:
> >> >>>112939138816,
> >> >>> >>> completeMsid: null, lastUpdated: null, lastPolled:
null,
> >>created:
> >> >>> null}
> >> >>> >>> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >>> >>> (Job-Executor-16:job-230) Executing
> >> >>> >>>com.cloud.api.commands.ReconnectHostCmd
> >> >>> >>> for job-230
> >> >>> >>> 2013-05-24 10:33:00,985 INFO  [agent.manager.AgentManagerImpl]
> >> >>> >>> (Job-Executor-16:job-230) Unable to disconnect host
because it
> >>is
> >> >>>not
> >> >>> >>>in
> >> >>> >>> the correct state: host=20; Status=Disconnected
> >> >>> >>> 2013-05-24 10:33:00,986 WARN  [api.commands.ReconnectHostCmd]
> >> >>> >>> (Job-Executor-16:job-230) Exception:
> >> >>> >>> com.cloud.api.ServerApiException
> >> >>> >>> at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.ja
> >>>>>>>>va
> >> >>>>>>:10
> >> >>> >>>8
> >> >>> >>>)
> >> >>> >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> >> >>> >>> at
> >> >>>
> >>
> >>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:
> >>>>>>>>43
> >> >>>>>>2)
> >> >>> >>> at
> >> >>>
> >>
> >>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
> >>>>>>>>47
> >> >>>>>>1)
> >> >>> >>> at
> >> >>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> >>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> >>> >>> at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> >>>>>>>>r.
> >> >>>>>>jav
> >> >>> >>>a
> >> >>> >>>:1110)
> >> >>> >>> at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >>>>>>>>or
> >> >>>>>>.ja
> >> >>> >>>v
> >> >>> >>>a:603)
> >> >>> >>> at java.lang.Thread.run(Thread.java:679)
> >> >>> >>> 2013-05-24 10:33:00,986 WARN  [cloud.api.ApiDispatcher]
> >> >>> >>> (Job-Executor-16:job-230) class
> >>com.cloud.api.ServerApiException :
> >> >>> null
> >> >>> >>> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >>> >>> (Job-Executor-16:job-230) Complete async job-230,
jobStatus: 2,
> >> >>> >>>resultCode:
> >> >>> >>> 530, result: Error Code: 534 Error text: null
> >> >>> >>> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >>> >>> (catalina-exec-4:null) Async job-230 completed
> >> >>> >>>
> >> >>> >>> catalina.out
> >> >>> >>>
> >> >>> >>> INFO  [agent.manager.AgentManagerImpl] (Job-Executor-16:job-230)
> >> >>> Unable
> >> >>> >>>to
> >> >>> >>> disconnect host because it is not in the correct state:
host=20;
> >> >>> >>> Status=Disconnected
> >> >>> >>> WARN  [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230)
> >> >>> >>>Exception:
> >> >>> >>> com.cloud.api.ServerApiException
> >> >>> >>>         at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.ja
> >>>>>>>>va
> >> >>>>>>:10
> >> >>> >>>8
> >> >>> >>>)
> >> >>> >>>         at
> >> >>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> >> >>> >>>         at
> >> >>> >>>
> >> >>>
> >>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432)
> >> >>> >>>         at
> >> >>> >>>
> >> >>>
> >>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> >>> >>>         at
> >> >>> >>>
> >>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >> >>> >>>         at
> >>java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> >>> >>>         at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> >>>>>>>>r.
> >> >>>>>>jav
> >> >>> >>>a
> >> >>> >>>:1110)
> >> >>> >>>         at
> >> >>> >>>
> >> >>>
> >> >>>
> >>
> >>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >>>>>>>>or
> >> >>>>>>.ja
> >> >>> >>>v
> >> >>>   >>>a:603)
> >> >>> >>>         at java.lang.Thread.run(Thread.java:679)
> >> >>> >>> WARN  [cloud.api.ApiDispatcher] (Job-Executor-16:job-230)
class
> >> >>> >>> com.cloud.api.ServerApiException : null
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >
> >> >>>
> >> >>>
> >> >>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message