cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-4315) Vpc Network upgrade is failing on KVM due to the backend failure when plugNic "Plug Nic failed due to org.libvirt.LibvirtException: internal error unable to execute QEMU command 'device_add': Duplicate ID 'net2' for device"
Date Thu, 15 Aug 2013 00:00:48 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740451#comment-13740451
] 

ASF subversion and git services commented on CLOUDSTACK-4315:
-------------------------------------------------------------

Commit 90388ce77e8906ec70641fe5b9f8fe26bb9ccf2e in branch refs/heads/master from [~edison]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=90388ce ]

CLOUDSTACK-4315: add code to clean up libvirt data structure after plug/unplug, that's the
only thing I can do to fix this kind of libvirt issue

                
> Vpc Network upgrade is failing on KVM due to the backend failure when plugNic "Plug Nic
failed due to org.libvirt.LibvirtException: internal error unable to execute QEMU command
'device_add': Duplicate ID 'net2' for device"
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4315
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4315
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: KVM
>    Affects Versions: 4.2.0
>            Reporter: Prasanna Santhanam
>            Priority: Critical
>             Fix For: 4.2.0
>
>         Attachments: kvm.tar.bz2
>
>
> While running tests for vpc network upgrade following situation was
> encountered. Before upgrading the vpc network, the network is shutdown and it
> fails with an NPE
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,732 WARN
 [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) shutdownNetworkRules failed during the netw
> ork Ntwk[441|Guest|127] shutdown due to 
> Aug 13 03:17:59 java.lang.NullPointerException  
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createApplyLoadBalancingRulesCommands(VirtualNetworkApplianceManagerImpl.java:3313)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendLBRules(VirtualNetworkApplianceManagerImpl.java:3678)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$9.execute(VirtualNetworkApplianceManagerImpl.java:3671)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3657)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:393)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1549)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2089)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1586)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.lb.LoadBalancingRulesManagerImpl.revokeLoadBalancersForNetwork(LoadBalancingRulesManagerImpl.java:1519)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.shutdownNetworkResources(NetworkManagerImpl.java:4054)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.shutdownNetworkElementsAndResources(NetworkManagerImpl.java:2913)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.shutdownNetwork(NetworkManagerImpl.java:2859)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1937)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,742 DEBUG
[cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Sending network shutdown to VpcVirtualRoute
> r
> But subsequently the network does shutdown without issues:
> ug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,742 DEBUG
[cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Sending network shutdown to VpcVirtualRouter
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,750 DEBUG
[network.element.VpcVirtualRouterElement] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Router VM[DomainRouter|r-479-QA] is not a part the network Ntwk[441|Guest|127]
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,750 DEBUG
[cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Network id=441 is shutdown successfully, cleaning up corresponding resources now.
> And then as the upgrade is triggered:
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,766 DEBUG
[cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Lock is released for network id 441 as a part of network implement
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,766 WARN
 [cloud.network.NetworkServiceImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Failed to implement network Ntwk[441|Guest|127] elements and resources as a part of network
update due to 
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com com.cloud.utils.exception.CloudRuntimeException:
Failed to add VPC router VM[DomainRouter|r-479-QA] to guest network Ntwk[441|Guest|127]
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:178)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2003)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1908)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,773 ERROR
[cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com com.cloud.utils.exception.CloudRuntimeException:
Failed to implement network (with specified id) elements and resources as a part of network
update
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2244)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com     at java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,776 DEBUG
[cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230
]) Complete async job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ], jobStatus: 2, resultCode:
530, result: Error Code: 530 Error text: Failed to implement network (with specified id) elements
and resources as a part of network update
> This is consistenly seen on test runs on kvm environments, but doesn't look to
> be hypervisor specific
> Steps to reproduce:
> # 1. Create a VPC with cidr - 10.1.1.1/16
> # 2. Create a Network offering - NO1 with all supported services
> #    except PF services provided by VRVPC provider, conserve mode=OFF
> # 3.Create a Network offering - NO2 with all supported services
> #    including Pf services provided by VRVPC provider,conserve mode=OFF
> # 4. Add network1(10.1.1.1/24) using N01 to this VPC.
> # 5. Deploy vm1 and vm2 in network1.
> # 6. Create a Static Nat and LB rules for vms in network1.
> # 7. Make sure you are not allowed to create a PF rule for any Vm in
> #    network1 and the Static Nat and LB rules for vms work as expected
> # 8. Update network1 to NO2.
> Management server logs are attached.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message