cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Disaster after maintenance
Date Tue, 19 Mar 2019 18:29:18 GMT
Share the network id and name as seen from GUI...

On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <j.zolotarjov@gmail.com>
wrote:

> >>> 1. Confirm please rolling restart is set to false please - double check
> Double checked - It is set to false
>
> >>>>2. If so - do you know the name of VR which you deleted ? Is it last
> one
> >>> ever created - if so we can find it easily...
> I dont know the name
> Is there a way to fetch it from DB?
>
> On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <andrija.panic@gmail.com>
> wrote:
>
> > Right...still complaining on missing running routers.
> >
> > 1. Confirm please rolling restart is set to false please - double check
> > 2. If so - do you know the name of VR which you deleted ? Is it last one
> > ever created - if so we can find it easily...
> >
> >
> >
> > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <j.zolotarjov@gmail.com>
> > wrote:
> >
> > > Here is management server log
> > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > >
> > >
> > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <andrija.panic@gmail.com
> >
> > > wrote:
> > >
> > > > I would also try to "undelete" VR in DB, but let's keep this as last
> > > step.
> > > >
> > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <andrija.panic@gmail.com>
> > > wrote:
> > > >
> > > > > Disclaimer: from mobile device.
> > > > >
> > > > > I don't see any special failure in agent log, except some long
> > running
> > > > > migration, timeout for graceful VM shutdown etc (and agent restart)
> > > > >
> > > > > You did not send mgmt log after last network restart failure ?
> > > > >
> > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > j.zolotarjov@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Under
> > > > >> Infrastructure / Hosts:
> > > > >> both hosts are enabled and green (Unsecure though)
> > > > >>
> > > > >> agent.log -
> > > > >>
> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > >>
> > > > >> I set network.rolling.restart to false now. Restarted management
> > > server
> > > > -
> > > > >> same problem - cannot restart not delete network
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > >> boris.stoyanov@shapeblue.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi, you shouldn’t be worried about data as long as you use a
> > > separate
> > > > >> > shared storage.
> > > > >> >
> > > > >> > What does the cloudstack console say about your host? Is it up?
> If
> > > > it’s
> > > > >> up
> > > > >> > then you should be able to deploy a VM on it.
> > > > >> >
> > > > >> > If not, you’ll need to investigate the reason cloudstack-agent
> is
> > > not
> > > > >> > connected, is the agent running? If not just start it and see if
> > it
> > > > >> > connects.
> > > > >> >
> > > > >> > If it’s running but still not connected please share the
> > agent.log.
> > > > >> >
> > > > >> > If nothing helps at all you can remove the host and add it
> again,
> > > but
> > > > >> > let’s try to troubleshoot first.
> > > > >> >
> > > > >> > Bobby.
> > > > >> >
> > > > >> >
> > > > >> > boris.stoyanov@shapeblue.com
> > > > >> > www.shapeblue.com
> > > > >> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > > >> > @shapeblue
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > > > j.zolotarjov@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > Guys, please help with it.
> > > > >> > > What can be done here?
> > > > >> > > There is too much valuable data.
> > > > >> > >
> > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> > > > >> > j.zolotarjov@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Tried that just now and got error:
> > > > >> > >> Resource [DataCenter:1] is unreachable: Can't find all
> > necessary
> > > > >> running
> > > > >> > >> routers!
> > > > >> > >>
> > > > >> > >> In the log I see:
> > > > >> > >> =============================================
> > > > >> > >>
> > > > >> > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > > > >> (logid:265a6099)
> > > > >> > >> Restarting network 204...
> > > > >> > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > > > >> (logid:265a6099)
> > > > >> > >> Performing rolling restart of routers of network
> > > Ntwk[204|Guest|6]
> > > > >> > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> > > > >> (logid:265a6099)
> > > > >> > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> > > > >> > >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099)
> > > > >> Unexpected
> > > > >> > >> exception while executing
> > > > >> > >>
> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > >> > >> com.cloud.exception.ResourceUnavailableException: Resource
> > > > >> > [DataCenter:1]
> > > > >> > >> is unreachable: Can't find all necessary running routers!
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > >> > >>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method)
> > > > >> > >>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > Source)
> > > > >> > >>        at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > >> > Source)
> > > > >> > >>        at java.lang.reflect.Method.invoke(Unknown Source)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > >> > >>        at com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > Source)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > >> > >>        at
> > > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > >> > >>        at
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > >> > >>        at
> > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > >> > >> Source)
> > > > >> > >>        at java.util.concurrent.FutureTask.run(Unknown Source)
> > > > >> > >>        at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > >> > >> Source)
> > > > >> > >>        at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > >> > >> Source)
> > > > >> > >>        at java.lang.Thread.run(Unknown Source)
> > > > >> > >> 2019-03-19 14:20:39,659 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099)
> > > > Complete
> > > > >> > async
> > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > >> > >> [DataCenter:1] is un
> > > > >> > >>
> > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > > > >> > andrija.panic@shapeblue.com>
> > > > >> > >> wrote:
> > > > >> > >>
> > > > >> > >>> ​​
> > > > >> > >>> Your network can't be deleted due to "Can't delete the
> > network,
> > > > not
> > > > >> all
> > > > >> > >>> user vms are expunged. Vm
> > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> > > > >> > >>>
> > > > >> > >>> You should be able to just start the user VM - but if you
> have
> > > > >> actually
> > > > >> > >>> delete the VR itself, then just do Network restart with
> > > "cleanup"
> > > > >> and
> > > > >> > it
> > > > >> > >>> will recreate a new VR, after which you should be able to
> > start
> > > > the
> > > > >> VM.
> > > > >> > >>>
> > > > >> > >>> Andrija
> > > > >> > >>>
> > > > >> > >>> andrija.panic@shapeblue.com
> > > > >> > >>> www.shapeblue.com
> > > > >> > >>> Amadeus House, Floral Street, London  WC2E 9DPUK
> > > > >> > >>> @shapeblue
> > > > >> > >>>
> > > > >> > >>>
> > > > >> > >>>
> > > > >> > >>>
> > > > >> > >>> -----Original Message-----
> > > > >> > >>> From: Jevgeni Zolotarjov <j.zolotarjov@gmail.com>
> > > > >> > >>> Sent: 19 March 2019 15:10
> > > > >> > >>> To: users@cloudstack.apache.org
> > > > >> > >>> Subject: Re: Disaster after maintenance
> > > > >> > >>>
> > > > >> > >>> I mean I cannot delete network: In the management server
> log I
> > > see
> > > > >> > >>>
> > > > >> > >>> ==========================================================
> > > > >> > >>> 019-03-19 14:06:36,316 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc)
> > > > >> Executing
> > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType:
> > > > Network,
> > > > >> > >>> instanceId: 204, cmd:
> > > > >> > >>>
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > >> > cmdInfo:
> > > > >> > >>>
> > > > >> > >>>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > > > >> > >>>
> > > > >> > >>>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> > > resultCode:
> > > > 0,
> > > > >> > >>> result: null, initMsid: 264216221068220, completeMsid: null,
> > > > >> > lastUpdated:
> > > > >> > >>> null, lastPolled: null, created: null}
> > > > >> > >>> 2019-03-19 14:06:36,351 WARN
> [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa)
> > > > >> > (logid:c734edfc)
> > > > >> > >>> Can't delete the network, not all user vms are expunged. Vm
> > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state
> > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG
> > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc)
> > > > Complete
> > > > >> > async
> > > > >> > >>> job-5090, jobStatus: FAILED, resultCode: 530, result:
> > > > >> > >>>
> > > > >> > >>>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > > > >> > >>> to delete network"}
> > > > >> > >>> ==========================================================
> > > > >> > >>>
> > > > >> > >>>
> > > > >> > >>> I deleted a router, expecting it to be recreated on deleting
> > > > >> network.
> > > > >> > But
> > > > >> > >>> I am unable to delete network because of above error
> > > > >> > >>>
> > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov <
> > > > >> > >>> j.zolotarjov@gmail.com>
> > > > >> > >>> wrote:
> > > > >> > >>>
> > > > >> > >>>> I've managed to make libvirtd running
> > > > >> > >>>> Now cloudstack console shows both hosts - running
> > > > >> > >>>>
> > > > >> > >>>> But now as I have removed network, VMs are unable to start.
> > > > >> > >>>>
> > > > >> > >>>> How can I recreate the network now?
> > > > >> > >>>>
> > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> > > > >> > >>>> <kudryavtsev_ia@bw-sw.com>
> > > > >> > >>>> wrote:
> > > > >> > >>>>
> > > > >> > >>>>> Jevgeniy, it may be a documentation bug. Take s look:
> > > > >> > >>>>>
> > > > https://github.com/apache/cloudstack-documentation/pull/27/files
> > > > >> > >>>>>
> > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov <
> > > > >> j.zolotarjov@gmail.com
> > > > >> > >:
> > > > >> > >>>>>
> > > > >> > >>>>>> That's it - libvirtd failed to start on second host.
> > > > >> > >>>>>> Tried restarting, but it does not start.
> > > > >> > >>>>>>
> > > > >> > >>>>>>
> > > > >> > >>>>>>>> Do you have some NUMA constraints or anything which
> > > requires
> > > > >> > >>>>> particular
> > > > >> > >>>>>> RAM configuration?
> > > > >> > >>>>>> No
> > > > >> > >>>>>>
> > > > >> > >>>>>> libvirtd.service - Virtualization daemon
> > > > >> > >>>>>>   Loaded: loaded
> (/usr/lib/systemd/system/libvirtd.service;
> > > > >> > >>>>>> enabled; vendor preset: enabled)
> > > > >> > >>>>>>   Active: failed (Result: start-limit) since Tue
> 2019-03-19
> > > > >> > >>>>>> 13:03:07
> > > > >> > >>>>> GMT;
> > > > >> > >>>>>> 12s ago
> > > > >> > >>>>>>     Docs: man:libvirtd(8)
> > > > >> > >>>>>>           https://libvirt.org
> > > > >> > >>>>>>  Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
> > > > >> > >>>>>> (code=exited,
> > > > >> > >>>>>> status=1/FAILURE)
> > > > >> > >>>>>> Main PID: 892 (code=exited, status=1/FAILURE)
> > > > >> > >>>>>>    Tasks: 19 (limit: 32768)
> > > > >> > >>>>>>   CGroup: /system.slice/libvirtd.service
> > > > >> > >>>>>>           ├─11338 /usr/sbin/libvirtd -d -l
> > > > >> > >>>>>>           ├─11909 /usr/sbin/dnsmasq
> > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > --leasefile-ro
> > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > >> > >>>>>>           └─11910 /usr/sbin/dnsmasq
> > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > --leasefile-ro
> > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > >> > >>>>>>
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > Failed
> > > > >> to
> > > > >> > >>>>>> start Virtualization daemon.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > Unit
> > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > > >> > >>>>> libvirtd.service
> > > > >> > >>>>>> failed.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > > >> > >>>>> libvirtd.service
> > > > >> > >>>>>> holdoff time over, scheduling restart.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > > Stopped
> > > > >> > >>>>>> Virtualization daemon.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > start
> > > > >> > >>>>>> request repeated too quickly for libvirtd.service Mar 19
> > > > 13:03:07
> > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
> > > > >> > >>>>>> Virtualization daemon.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > Unit
> > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> > > > >> > >>>>> libvirtd.service
> > > > >> > >>>>>> failed.
> > > > >> > >>>>>>
> > > > >> > >>>>>>
> > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus
> > > > >> > >>>>>> <paul.angus@shapeblue.com>
> > > > >> > >>>>>> wrote:
> > > > >> > >>>>>>
> > > > >> > >>>>>>> Can you check that the cloudstack agent is running on
> the
> > > host
> > > > >> > >>>>>>> and the agent logs (usual logs directory) Also worth
> > > checking
> > > > >> > >>>>>>> that libvirt has started ok.  Do you have some
> > > > >> > >>>>> NUMA
> > > > >> > >>>>>>> constraints or anything which requires particular RAM
> > > > >> > >>> configuration?
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> paul.angus@shapeblue.com
> > > > >> > >>>>>>> www.shapeblue.com
> > > > >> > >>>>>>> Amadeus House, Floral Street, London  WC2E 9DPUK
> > @shapeblue
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> -----Original Message-----
> > > > >> > >>>>>>> From: Jevgeni Zolotarjov <j.zolotarjov@gmail.com>
> > > > >> > >>>>>>> Sent: 19 March 2019 14:49
> > > > >> > >>>>>>> To: users@cloudstack.apache.org
> > > > >> > >>>>>>> Subject: Re: Disaster after maintenance
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Can you try migrating a VM to the server that you
> changed
> > > the
> > > > >> RAM
> > > > >> > >>>>> amount?
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Also:
> > > > >> > >>>>>>> What is the hypervisor version?
> > > > >> > >>>>>>> KVM
> > > > >> > >>>>>>> QEMU Version     : 2.0.0
> > > > >> > >>>>>>> Release     : 1.el7.6
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Host status in ACS?
> > > > >> > >>>>>>> 1st server: Unsecure
> > > > >> > >>>>>>> 2nd server: Disconnected
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Did you try to force a VM to start/deploy in this server
> > > where
> > > > >> you
> > > > >> > >>>>>> changed
> > > > >> > >>>>>>> the RAM?
> > > > >> > >>>>>>> Host status became disconnected. I don't know how to
> make
> > it
> > > > >> > >>>>> "connected"
> > > > >> > >>>>>>> again
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
> > > > >> > >>>>>>> rafaelweingartner@gmail.com> wrote:
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>> Can you try migrating a VM to the server that you
> changed
> > > the
> > > > >> RAM
> > > > >> > >>>>>> amount?
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>> Also:
> > > > >> > >>>>>>>> What is the hypervisor version?
> > > > >> > >>>>>>>> Host status in ACS?
> > > > >> > >>>>>>>> Did you try to force a VM to start/deploy in this
> server
> > > > where
> > > > >> > >>> you
> > > > >> > >>>>>>>> changed the RAM?
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
> > > > >> > >>>>>>>> <j.zolotarjov@gmail.com
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>> wrote:
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup running fine for few
> > > months
> > > > >> > >>> (>4)
> > > > >> > >>>>> The
> > > > >> > >>>>>>>>> setup is very simple: 2 hosts We decided to do a
> > > maintenance
> > > > >> to
> > > > >> > >>>>>>>>> increase RAM on both servers
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> For this we put first server to maintenance. All VMS
> > moved
> > > > to
> > > > >> > >>>>> second
> > > > >> > >>>>>>>>> host after a while.
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> Then first server was shutdown, RAM increased, server
> > > turned
> > > > >> > >>> ON.
> > > > >> > >>>>>>>>> Now nothing starts on first server.
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> Tried to delete network, but this fails as well
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> Please help !
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> Here is extract from log:
> > > > >> > >>>>>>>>> ======================================
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
> > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl]
> > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is
> > > ready
> > > > >> to
> > > > >> > >>>>>>>>> launch secondary storage VM
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG
> > > > [c.c.c.ConsoleProxyManagerImpl]
> > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1
> is
> > > > ready
> > > > >> > >>> to
> > > > >> > >>>>>>>>> launch console proxy
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be)
> > > ===START===
> > > > >> > >>>>>>>> 192.168.5.140
> > > > >> > >>>>>>>>> -- GET
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > >> > >>>>>>>> =json&_=1552998473154
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > (logid:cd8e30be)
> > > > >> > >>>>> CIDRs
> > > > >> > >>>>>>>>> from which account
> > > > >> > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> > > > >> > >>>>>>>> allowed
> > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO
> > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:f6751fa7)
> > > > >> Add
> > > > >> > >>>>>>>>> job-5081 into job monitoring
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > (logid:cd8e30be)
> > > > >> > >>>>> submit
> > > > >> > >>>>>>>> async
> > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO {id:5081, userId: 2,
> > > > accountId:
> > > > >> > >>> 2,
> > > > >> > >>>>>>>>> instanceType: Network, instanceId: 204, cmd:
> > > > >> > >>>>>>>>>
> > > > >> > >>>
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > >> > >>>>>>> cmdInfo:
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > >> > >>>>>>>> interface
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > >> > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> > > > >> > >>> resultCode:
> > > > >> > >>>>> 0,
> > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid:
> > > null,
> > > > >> > >>>>>>> lastUpdated:
> > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > (logid:cd8e30be)
> > > > >> > >>>>>>>>> ===END===
> > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > >> > >>>>>>>> =json&_=1552998473154
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2, accountId:
> 2,
> > > > >> > >>>>>>>>> instanceType: Network,
> > > > >> > >>>>>>>>> instanceId: 204, cmd:
> > > > >> > >>>>>>>>>
> > > > >> > >>>
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > >> > >>>>>>> cmdInfo:
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > >> > >>>>>>>> interface
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > >> > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> > > > >> > >>> resultCode:
> > > > >> > >>>>> 0,
> > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid:
> > > null,
> > > > >> > >>>>>>> lastUpdated:
> > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN
> > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081
> ctx-d5de7979)
> > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the network, not all
> user
> > > vms
> > > > >> are
> > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in Stopped state
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>>>> Complete
> > > > >> > >>>>>>>> async
> > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, resultCode: 530, result:
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed
> > > > >> > >>>>>>>>> to delete network"}
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>>>> Publish
> > > > >> > >>>>>>>>> async
> > > > >> > >>>>>>>>> job-5081 complete on message bus
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>> Wake
> > > > >> > >>>>> up
> > > > >> > >>>>>>>>> jobs related to job-5081
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>> Update
> > > > >> > >>>>>>>>> db status for job-5081
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>> Wake
> > > > >> > >>>>> up
> > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin all subjobs
> > created
> > > > from
> > > > >> > >>>>> job-
> > > > >> > >>>>>>>>> 5081
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>> Done
> > > > >> > >>>>>>>> executing
> > > > >> > >>>>>>>>>
> > > > >> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> > > > >> > >>>>> for
> > > > >> > >>>>>>>>> job-5081
> > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO
> > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > (logid:16897ea6)
> > > > >> > >>> Remove
> > > > >> > >>>>>>>> job-5081
> > > > >> > >>>>>>>>> from job monitoring
> > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet]
> > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75)
> > > ===START===
> > > > >> > >>>>>>>> 192.168.5.140
> > > > >> > >>>>>>>>> -- GET
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer]
> > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > (logid:7fe68f75)
> > > > >> > >>> CIDRs
> > > > >> > >>>>>>>>> from which account
> > > > >> > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> > > > >> > >>>>>>>> allowed
> > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet]
> > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > (logid:7fe68f75)
> > > > >> > >>>>>>>>> ===END===
> > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>
> > > > >> >
> > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> (logid:a605267a)
> > > > Begin
> > > > >> > >>>>>>>>> cleanup expired async-jobs
> > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO
> > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> (logid:a605267a)
> > > End
> > > > >> > >>>>> cleanup
> > > > >> > >>>>>>>> expired
> > > > >> > >>>>>>>>> async-jobs
> > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
> > > > >> > >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b)
> > > Found
> > > > 0
> > > > >> > >>>>>>>>> routers to update status.
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>> --
> > > > >> > >>>>>>>> Rafael Weingärtner
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>>
> > > > >> > >>>>>
> > > > >> > >>>>
> > > > >> > >>>
> > > > >> > >>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message