cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prashanth Manthena <prashanth.manth...@nuagenetworks.net>
Subject Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM
Date Tue, 18 Oct 2016 09:15:20 GMT
Hi All,

Raised the following issue on Jira to track this bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-9547

With regards,
Prashanth

On Mon, Oct 17, 2016 at 12:38 PM, Prashanth Manthena <
prashanth.manthena@nuagenetworks.net> wrote:

> Hi Paul,
>
> First of all, thank you for your time and investigation.
>
> I have tried with timeouts up to 180 seconds, and that didn't help either
> :)
>
> Yes, it doesn't explain the difference between 4.7 and 4.9, because I am
> using the same vcenter/vmware setup (i.e. no vmware tools installed).
>
> Let me know, if you want me to raise an issue on Jira for tracking this
> bug.
>
> On Sat, Oct 15, 2016 at 8:55 PM, Paul Angus <paul.angus@shapeblue.com>
> wrote:
>
>> I tried extending the timeout to 30 seconds, but that didn't help.
>> However, installing vmware tools (not easy in our built-in templates)
>> made the problem go away.  With the tools installed the removal was pretty
>> much instant.
>> Doesn't explain the difference 4.7 and 4.9 that you are seeing though.
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: Prashanth Manthena [mailto:prashanth.manthena@nuagenetworks.net]
>> Sent: 14 October 2016 13:24
>> To: dev@cloudstack.apache.org
>> Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
>> multi-nic VM
>>
>> Hi Paul,
>>
>> Thank you for trying it out.
>>
>> I am only hitting this issue for guest VMs (i.e. not with VPC VRs)
>> created in ACS 4.9 (i.e. not in ACS 4.7) with VMware setups.
>>
>> Moreover, I get the same error when I am trying to remove the NIC (i.e.
>> network adapter) directly from VMware's Vcenter.
>>
>> There is a possible workaround for this issue from VMware on Internet,
>> which doesn't work in this scenario both from CloudStack and VMware:
>> https://kb.vmware.com/selfservice/microsites/search.do?
>> language=en_US&cmd=displayKC&externalId=2081503
>>
>> Most likely, this issue has something to do with how we deploy
>> (multi-nic) guest VMs in ACS 4.9 with VMware setups.
>>
>> On Fri, Oct 14, 2016 at 1:00 PM, Paul Angus <paul.angus@shapeblue.com>
>> wrote:
>>
>> > Hi Prashanth,
>> >
>> > I've just tried that. I get the same error - The guest operating
>> > system did not respond to a hot-remove request for device ethernet1 in
>> > a timely manner.
>> >
>> > Kind regards,
>> >
>> > Paul Angus
>> >
>> > paul.angus@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Prashanth Manthena [mailto:prashanth.manthena@nuagenetworks.net]
>> > Sent: 14 October 2016 09:21
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
>> > multi-nic VM
>> >
>> > HI All,
>> >
>> > Does this issue ring a bell, and is anyone else hitting this issue ?
>> >
>> > Let me know, if it is a known issue.
>> >
>> > Thanking you in advance !!
>> >
>> > On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena <
>> prashanth.manthena@
>> > nuagenetworks.net> wrote:
>> >
>> > > Hi,
>> > >
>> > > I am hitting the following issue on an ACS 4.9 + VMware setup (steps
>> > > to
>> > > reproduce):
>> > >
>> > > 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM
>> > >
>> > > 2) Remove the non-default nic from the multi-nic VM, which fails with
>> > > the following error/exception in the management server log:
>> > >
>> > > 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
>> > > (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98)
>> > > ===END===  10.31.52.95 -- GET
>> > > command=queryAsyncJobResult&jobId=9ad66ce9-6e1b-4c25-bd2e-76
>> 3f4586dd86
>> > > &response=json&_=1475673245452
>> > > 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
>> > (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
>> > UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
>> > > java.lang.RuntimeException: The guest operating system did not respond
>> > to a hot-remove request for device ethernet1 in a timely manner.
>> > >         at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(
>> > VmwareClient.java:354)
>> > >         at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.
>> > configureVm(VirtualMachineMO.java:949)
>> > >         at com.cloud.hypervisor.vmware.resource.VmwareResource.
>> > execute(VmwareResource.java:1103)
>> > >         at com.cloud.hypervisor.vmware.resource.VmwareResource.
>> > executeRequest(VmwareResource.java:469)
>> > >         at com.cloud.agent.manager.Direct
>> AgentAttache$Task.runInContext(
>> > DirectAgentAttache.java:315)
>> > >         at org.apache.cloudstack.managed.context.
>> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > >         at org.apache.cloudstack.managed.context.
>> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > >         at java.util.concurrent.Executors$RunnableAdapter.
>> > call(Executors.java:471)
>> > >         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > >         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> > >         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> > >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1145)
>> > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:615)
>> > >         at java.lang.Thread.run(Thread.java:745)
>> > > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache]
>> > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
>> 4-1440588930805137508:
>> > Response Received:
>> > > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>> > > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
>> > > 4-1440588930805137508: MgmtId 275619427423488: Resp: Routing to peer
>> > > 2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache]
>> > > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
>> > > 4-1440588930805137508: No more commands found
>> > > 2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory]
>> > (API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f) (logid:9ad66ce9)
>> > Returning cached instance of singleton bean 'messageBus'
>> > > 2016-10-05 06:13:31,127 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> > > (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Unexpected
>> > > exception while executing
>> > > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin
>> > > com.cloud.utils.exception.CloudRuntimeException: Unable to remove
>> > Ntwk[205|Guest|16] from VM[User|i-2-3-VM]
>> > >         at com.cloud.vm.UserVmManagerImpl
>> .removeNicFromVirtualMachine(
>> > UserVmManagerImpl.java:1291)
>> > >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > >         at sun.reflect.NativeMethodAccessorImpl.invoke(
>> > NativeMethodAccessorImpl.java:57)
>> > >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> > DelegatingMethodAccessorImpl.java:43)
>> > >         at java.lang.reflect.Method.invoke(Method.java:606)
>> > > at org.springframework.aop.support.AopUtils.
>> > invokeJoinpointUsingReflection(AopUtils.java:317)
>> > >         at org.springframework.aop.framew
>> ork.ReflectiveMethodInvocation.
>> > invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> > >         at org.springframework.aop.framew
>> ork.ReflectiveMethodInvocation.
>> > proceed(ReflectiveMethodInvocation.java:150)
>> > >         at org.apache.cloudstack.network.contrail.management.
>> > EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>> > >         at org.springframework.aop.framew
>> ork.ReflectiveMethodInvocation.
>> > proceed(ReflectiveMethodInvocation.java:161)
>> > >         at com.cloud.event.ActionEventInterceptor.invoke(
>> > ActionEventInterceptor.java:51)
>> > >         at org.springframework.aop.framew
>> ork.ReflectiveMethodInvocation.
>> > proceed(ReflectiveMethodInvocation.java:161)
>> > >         at org.springframework.aop.interceptor.
>> > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> > >         at org.springframework.aop.framew
>> ork.ReflectiveMethodInvocation.
>> > proceed(ReflectiveMethodInvocation.java:172)
>> > >         at org.springframework.aop.framework.JdkDynamicAopProxy.
>> > invoke(JdkDynamicAopProxy.java:204)
>> > >         at com.sun.proxy.$Proxy198.remove
>> NicFromVirtualMachine(Unknown
>> > Source)
>> > >         at org.apache.cloudstack.api.command.admin.vm.
>> > RemoveNicFromVMCmdByAdmin.execute(RemoveNicFromVMCmdByAdmin.java:44)
>> > >         at com.cloud.api.ApiDispatcher.di
>> spatch(ApiDispatcher.java:150)
>> > >         at com.cloud.api.ApiAsyncJobDispatcher.runJob(
>> > ApiAsyncJobDispatcher.java:108)
>> > >         at org.apache.cloudstack.framework.jobs.impl.
>> > AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> > >         at org.apache.cloudstack.managed.context.
>> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > >         at org.apache.cloudstack.managed.context.impl.
>> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > >         at org.apache.cloudstack.managed.context.
>> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > >         at org.apache.cloudstack.framework.jobs.impl.
>> > AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> > >         at java.util.concurrent.Executors$RunnableAdapter.
>> > call(Executors.java:471)
>> > >         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1145)
>> > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:615)
>> > >         at java.lang.Thread.run(Thread.java:745)
>> > > 2016-10-05 06:13:31,129 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Complete
>> > > async job-171, jobStatus: FAILED, resultCode: 530, result:
>> > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":
>> > > [],"errorcode":530,"errortext":"Unable to remove Ntwk[205|Guest|16]
>> > > from VM[User|i-2-3-VM]"}
>> > >
>> > >
>> > > Here is the error in the VM's vmware.log:
>> > >
>> > > [root@csc-1 i-2-9-VM]# tail -f vmware.log 2016-10-06T04:34:33.214Z|
>> > > vcpu-0| I120: CDROM: Emulate GET CONFIGURATION RT 2 starting feature 0
>> > > 2016-10-06T04:34:33.214Z| vcpu-0| I120: CDROM: Emulate GET
>> > > CONFIGURATION RT 2 starting feature 0 2016-10-06T04:36:09.127Z| vmx|
>> > I120: GuestRpcSendTimedOut: message to toolbox timed out.
>> > > 2016-10-06T04:36:09.128Z| vmx| I120: Vix: [80405
>> > > guestCommands.c:1924]: Error VIX_E_TOOLS_NOT_RUNNING in
>> > VMAutomationTranslateGuestRpcError(): VMware Tools are not running in
>> the
>> > guest 2016-10-06T04:43:08.823Z| vmx| I120: Tools: No activity for 10
>> > minutes, resetting Tools version.
>> > > 2016-10-06T04:43:08.824Z| vmx| I120: ToolsSetVersionWork did nothing;
>> > > new tools version (0) matches old Tools version
>> > > 2016-10-06T04:43:23.152Z| vmx| W110: Requesting hot-remove of
>> > > ethernet1 2016-10-06T04:50:13.889Z| mks| I120: SOCKET 3 (121) recv
>> > detected client closed connection 2016-10-06T04:50:13.890Z| mks| I120:
>> > SOCKET 3 (121) VNC Remote Disconnect: socket closed.
>> > > 2016-10-06T04:53:25.699Z| vmx| W110: VigorHotRemovePoll: device
>> > ethernet1 failed to disappear.
>> > >
>> > >
>> > > Note: I am not hitting this issue on ACS 4.7 with same VMware setup.
>> > >
>> > >
>> > > With regards,
>> > >
>> > > Prashanth
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message