cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Luc Dion <pd...@cloudops.com>
Subject Re: SystemVM can't start
Date Fri, 31 Mar 2017 23:29:47 GMT
Can you look if the system vm start in PV mode? We come up with a situation
with xs7.1 where the systemvm fail to start because it was in HVM.


On Mar 27, 2017 10:29 AM, "Fabiano A. Souza" <f.souza@fdsouza.com.br> wrote:

> Hi Rafael,
>
> I just want to thank you for your help.
>
> I tried to use the hint of fixing state.db but it did not work. The XML
> validator return the message that no errors was found on file. I had to
> recreate the XenServer's Pool. As I had no-critical instances on this
> pool the downtime impact was less.
>
> Thanks again for your help.
>
> ---
> FABIANO A. SOUZA
> _Assistência e Consultoria em TI_
> f.souza@fdsouza.com.br
> http://www.fdsouza.com.br
>
> Em 15/03/2017 15:27, Rafael Weingärtner escreveu:
>
> > de nada ;)
> >
> > On Wed, Mar 15, 2017 at 2:26 PM, Fabiano A. Souza <
> f.souza@fdsouza.com.br>
> > wrote:
> >
> > I'm so sorry.
> >
> > Thanks
> >
> > ---
> > FABIANO A. SOUZA
> > _Assistência e Consultoria em TI_
> > f.souza@fdsouza.com.br
> > http://www.fdsouza.com.br
> >
> > Em 15/03/2017 15:21, Rafael Weingärtner escreveu:
> >
> > The answer is in my previous email, last line:
> >
> > You can check if there is something wrong in state.db by passing the
> > contents of it in an XML validator. There will probably be some broken
>  XML
>
> >> tags there.
> >
> > Last time I faced this problem, I put the host in maintenance, powered
>  off
>
> > or migrate host's VMs, then I opened the file with an XML validator,
>  found
>
> > the entries that were corrupted and removed them. Then  I replaced the
>  old
>
> > state.db file with this modified file. Afterward, reboot the server and
> > everything should be ok.
> >
> > No need to tell you to do a backup of things, check hypervisors
> > community...., right?
> > http://discussions.citrix.com/topic/241761-help-power-
>  failed-now-xapi-will-not-start/
>
> > <http://www.google.com/url?q=http%3A%2F%2Fdiscussions.
>  citrix.com%2Ftopic%2F241761-help-power-failed-now-xapi-
> will-not-start%2F&sa=D&sntz=1&usg=AFQjCNEMwpDSMLgdddVdD6iDcZ4--yxtkQ>
>
> > On Wed, Mar 15, 2017 at 2:05 PM, Fabiano A. Souza <
>  f.souza@fdsouza.com.br>
>
> > wrote:
> >
> > Very thanks Rafael. I was already imagining that this would be the
> > problem. Do you know any way to fix this problem in XenServer?
> >
> > By your experience do you believe that recreating the pool in XenServer
> > will correct this problem?
> >
> > Again thanks so much for your help.
> >
> > ---
> > FABIANO A. SOUZA
> > _Assistência e Consultoria em TI_
> > f.souza@fdsouza.com.br
> > http://www.fdsouza.com.br
> >
> > Em 15/03/2017 14:36, Rafael Weingärtner escreveu:
> >
> > Looking at the log from your XenServer, and as I said before the problem
>  is not in ACS.
>
> > This kind of log entries:
> >
> > Db_exn.DBCache_NotFound("missing row", "VBD",
> >
> > "OpaqueRef:e1136dfc-cdbc-ba21-3eb8-94838cc75030")
> > or
> >
> > D:424e469aca54 failed with exception Db_exn.DBCache_NotFound("missing
> > row", "task",
> > indicate a problem in the hypervisor metadata. In your case, XenServer
>   uses
>
> > a file called state.db. This file contains metadata in XML format.
>   Probably
>
> > your force reboot corrupted some entries there (I have already had
>   similar
>
> > problems before). You can check if there is something wrong in state.db
>   by
>
> > passing the contents of it in a XML validator. There will probably be
>   some
>
> > broken XML tags there.
> >
> > On Wed, Mar 15, 2017 at 12:15 PM, Fabiano A. Souza <
>   f.souza@fdsouza.com.br>
>
> > wrote:
> >
> >> Yes. I set up the DataCenter with the Advanced Network configuration.
> >>
> >> I'm attaching XenServer file log detailed sample.
> >>
> >> Thank you so much for your help.
> >> ---
> >>
> >> *Fabiano A. Souza*
> >> *Assistência e Consultoria em TI*
> >> f.souza@fdsouza.com.br
> >> http://www.fdsouza.com.br
> >>
> >> Em 15/03/2017 12:52, Rafael Weingärtner escreveu:
> >>
> >> I am not sure if this is a complete destroy and if ACS re-uses something
> >> from the old VMs.
> >>
> >> From your logs,
> >>
> >> VIF.unplug
> >> R:68394c4e8e7c failed with exception Db_exn.DBCache_NotFound("missing
> >>
> >> This probably happened because when the VM was being created and the VIF
> >> was not created in the hypervisor. There are probably other error log
> >> entries before this one. From your other log:
> >>
> >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
> >>
> >> It is clear that there is something wrong in your XenServer. If you
>   want us
>
> >> to help you, you need to provide a more log entries (not scattered
>   bits).
>
> >> Also, in the XenServer logs, there will be plenty of exceptions there
> >> regarding this problem.
> >> You said that user VMs are working. I am guessing you have an advanced
> >> networking set up. Maybe the Guest network was not affected, but the
> >> Management or Storage network configuration in the XenServer may be
> >> affected by some misconfiguration/corruption that happened during the
> >> force-reboot.
> >>
> >> On Wed, Mar 15, 2017 at 11:42 AM, Fabiano A. Souza
>   <f.souza@fdsouza.com.brwrote:
>
> >> The ACS is trying to do this itself.
> >>
> >> ACS create, trying start, after error stop and destroy the systemvms
> >> instance. The instance name or ID is being incremented at each attempt
> >> to create these instances (s-300-VM, s-301-VM...).
> >>
> >> ---
> >> FABIANO A. SOUZA
> >> _Assistência e Consultoria em TI_
> >> f.souza@fdsouza.com.br
> >> http://www.fdsouza.com.br
> >>
> >> Em 15/03/2017 12:27, Rafael Weingärtner escreveu:
> >>
> >> You said you forced a reboot. It looks like something got corrupted in
> >>
> >> your
> >>
> >> host.
> >>
> >> I would not do some drastic measure like destroying pool.
> >>
> >> Have you tried to destroy the system VMs and then let ACS re-create
>   them?
>
> >> On Wed, Mar 15, 2017 at 11:23 AM, Fabiano A. Souza <
> >>
> >> f.souza@fdsouza.com.br>
> >>
> >> wrote:
> >>
> >> Hi Rafael
> >>
> >> I have some other Guest (Windows/Linux) include VRs that works normally.
> >> Only systemvms has problem to deploy. I use one pool with two XenServer
> >> 7 host. In xenresource.log I see this error below.
> >>
> >> Mar 15 08:31:32 fl-mia-compute1 xapi: [error|fl-mia-compute1|605 INET
> >> :::80|dispatch:VIF.unplug D:6cce7f031461|backtrace] VIF.unplug
> >> R:68394c4e8e7c failed with exception Db_exn.DBCache_NotFound("missing
> >> row", "VIF", "OpaqueRef:20e49fbf-5c92-ef21-805b-13b1997e326e")
> >>
> >> Mar 15 08:31:32 fl-mia-compute1 xapi: [error|fl-mia-compute1|605 INET
> >> :::80|dispatch:VIF.unplug D:b37f4fd2de0c|backtrace] VIF.unplug
> >> R:fcca316e07d3 failed with exception Db_exn.DBCache_NotFound("missing
> >> row", "VIF", "OpaqueRef:4d7c3a36-982a-825e-e71f-8978108ec56b")
> >>
> >> I'm thinking remove the hosts from CloudStack and destroy pool. Create
> >> again the pool and re-attach them again on CloudStack configuration. Do
> >> you believe that can I have problem to start the guests instance after
> >> that?
> >>
> >> ---
> >> FABIANO A. SOUZA
> >> _Assistência e Consultoria em TI_
> >> f.souza@fdsouza.com.br
> >> http://www.fdsouza.com.br
> >>
> >> Em 15/03/2017 11:28, Rafael Weingärtner escreveu:
> >>
> >> This is not an Apache CloudStack (ACS) problem per se.
> >> "HOST_CANNOT_ATTACH_NETWOR"
> >> Something is wrong with your hosts. Are all of the networks of yours
> >>
> >> hosts working? is everything in place as ACS expects?
> >>
> >> On Wed, Mar 15, 2017 at 10:26 AM, Fabiano A. Souza <
> >>
> >> f.souza@fdsouza.com.br> wrote:
> >>
> >> After my host with XenServer 7 forced reboot the systemvms can't start
> >> anymore.
> >>
> >> Can you help me?
> >>
> >> 2017-03-14 09:42:10,724 WARN [c.c.h.x.r.CitrixResourceBase]
> >> (DirectAgent-184:ctx-f12cdd04) (logid:4552cd9d) Task failed! Task
> >> record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e
> >> nameLabel: Async.VM.start_on
> >> nameDescription:
> >> allowedOperations: []
> >> currentOperations: {}
> >> created: Tue Mar 14 09:42:46 BRT 2017
> >> finished: Tue Mar 14 09:42:46 BRT 2017
> >> status: failure
> >> residentOn: com.xensource.xenapi.Host@4e05ddb9
> >> progress: 1.0
> >> type: <none/>
> >> result:
> >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
> >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9,
> >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46]
> >> otherConfig: {}
> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> >> subtasks: []
> >>
> >> 2017-03-14 09:42:10,730 WARN [c.c.h.x.r.CitrixResourceBase]
> >> (DirectAgent-184:ctx-f12cdd04) (logid:4552cd9d) Unable to start
> >> VM(s-294-VM) on host(b8324267-2cbe-4aa9-9b26-215375cb5553) due to Task
> >> failed! Task record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e
> >> nameLabel: Async.VM.start_on
> >> nameDescription:
> >> allowedOperations: []
> >> currentOperations: {}
> >> created: Tue Mar 14 09:42:46 BRT 2017
> >> finished: Tue Mar 14 09:42:46 BRT 2017
> >> status: failure
> >> residentOn: com.xensource.xenapi.Host@4e05ddb9
> >> progress: 1.0
> >> type: <none/>
> >> result:
> >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
> >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9,
> >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46]
> >> otherConfig: {}
> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> >> subtasks: []
> >>
> >> Task failed! Task record: uuid: 84f3483d-1021-22fb-be39-2eb15366e03e
> >> nameLabel: Async.VM.start_on
> >> nameDescription:
> >> allowedOperations: []
> >> currentOperations: {}
> >> created: Tue Mar 14 09:42:46 BRT 2017
> >> finished: Tue Mar 14 09:42:46 BRT 2017
> >> status: failure
> >> residentOn: com.xensource.xenapi.Host@4e05ddb9
> >> progress: 1.0
> >> type: <none/>
> >> result:
> >> errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
> >> OpaqueRef:3269022e-9b3d-88ca-8fb1-b15f3948b3a9,
> >> OpaqueRef:cd7b3c3e-92fd-18a3-eaa7-099aa066bb46]
> >> otherConfig: {}
> >> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> >> subtasks: []
> >>
> >> at
> >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.
> >> checkForSuccess(CitrixResourceBase.java:471)
> >> at
> >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(
> >> CitrixResourceBase.java:4900)
> >> at
> >> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.
> >> CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:128)
> >> at
> >> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.
> >> CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
> >> at
> >> com.cloud.hypervisor.xenserver.resource.wrapper.
> >> xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >> at
> >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.
> >>
> >> executeRequest(
> >>
> >> CitrixResourceBase.java:1687)
> >> at
> >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(
> >> DirectAgentAttache.java:315)
> >> at
> >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(
> >> ManagedContextRunnable.java:49)
> >> at
> >> org.apache.cloudstack.managed.context.impl.
> >>
> >> DefaultManagedContext$1.call(
> >>
> >> DefaultManagedContext.java:56)
> >> at
> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> >> callWithContext(DefaultManagedContext.java:103)
> >> at
> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> >> runWithContext(DefaultManagedContext.java:53)
> >> at
> >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(
> >> ManagedContextRunnable.java:46)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$
> >> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$
> >>
> >> ScheduledFutureTask.run(
> >>
> >> ScheduledThreadPoolExecutor.java:293)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(
> >> ThreadPoolExecutor.java:1142)
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> >> ThreadPoolExecutor.java:617)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> --
> >> FABIANO A. SOUZA
> >> _Assistência e Consultoria em TI_
> >> f.souza@fdsouza.com.br
> >> http://www.fdsouza.com.br

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message