Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F91710312 for ; Wed, 14 Aug 2013 08:20:55 +0000 (UTC) Received: (qmail 48014 invoked by uid 500); 14 Aug 2013 08:20:52 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 47961 invoked by uid 500); 14 Aug 2013 08:20:50 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 47929 invoked by uid 500); 14 Aug 2013 08:20:49 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 47913 invoked by uid 99); 14 Aug 2013 08:20:48 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Aug 2013 08:20:48 +0000 Date: Wed, 14 Aug 2013 08:20:48 +0000 (UTC) From: "Prasanna Santhanam (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-4315) Vpc Network upgrade is failing MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Santhanam updated CLOUDSTACK-4315: ------------------------------------------- Summary: Vpc Network upgrade is failing (was: If network shutdown fails, subsequent network upgrade appears to be failing) > Vpc Network upgrade is failing > ------------------------------ > > Key: CLOUDSTACK-4315 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4315 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Reporter: Prasanna Santhanam > Priority: Critical > > While running tests for vpc network upgrade following situation was > encountered. Before upgrading the vpc network, the network is shutdown and it > fails with an NPE > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,732 WARN [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) shutdownNetworkRules failed during the netw > ork Ntwk[441|Guest|127] shutdown due to > Aug 13 03:17:59 java.lang.NullPointerException > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createApplyLoadBalancingRulesCommands(VirtualNetworkApplianceManagerImpl.java:3313) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendLBRules(VirtualNetworkApplianceManagerImpl.java:3678) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$9.execute(VirtualNetworkApplianceManagerImpl.java:3671) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3657) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:393) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1549) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2089) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1586) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.lb.LoadBalancingRulesManagerImpl.revokeLoadBalancersForNetwork(LoadBalancingRulesManagerImpl.java:1519) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.shutdownNetworkResources(NetworkManagerImpl.java:4054) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.shutdownNetworkElementsAndResources(NetworkManagerImpl.java:2913) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.shutdownNetwork(NetworkManagerImpl.java:2859) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1937) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask.run(FutureTask.java:166) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.lang.Thread.run(Thread.java:679) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending network shutdown to VpcVirtualRoute > r > But subsequently the network does shutdown without issues: > ug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending network shutdown to VpcVirtualRouter > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,750 DEBUG [network.element.VpcVirtualRouterElement] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Router VM[DomainRouter|r-479-QA] is not a part the network Ntwk[441|Guest|127] > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,750 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Network id=441 is shutdown successfully, cleaning up corresponding resources now. > And then as the upgrade is triggered: > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,766 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Lock is released for network id 441 as a part of network implement > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,766 WARN [cloud.network.NetworkServiceImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Failed to implement network Ntwk[441|Guest|127] elements and resources as a part of network update due to > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com com.cloud.utils.exception.CloudRuntimeException: Failed to add VPC router VM[DomainRouter|r-479-QA] to guest network Ntwk[441|Guest|127] > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:178) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2003) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1908) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask.run(FutureTask.java:166) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.lang.Thread.run(Thread.java:679) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,773 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com com.cloud.utils.exception.CloudRuntimeException: Failed to implement network (with specified id) elements and resources as a part of network update > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2244) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.FutureTask.run(FutureTask.java:166) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at java.lang.Thread.run(Thread.java:679) > Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13 10:17:59,776 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Complete async job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to implement network (with specified id) elements and resources as a part of network update > This is consistenly seen on test runs on kvm environments, but doesn't look to > be hypervisor specific > Steps to reproduce: > # 1. Create a VPC with cidr - 10.1.1.1/16 > # 2. Create a Network offering - NO1 with all supported services > # except PF services provided by VRVPC provider, conserve mode=OFF > # 3.Create a Network offering - NO2 with all supported services > # including Pf services provided by VRVPC provider,conserve mode=OFF > # 4. Add network1(10.1.1.1/24) using N01 to this VPC. > # 5. Deploy vm1 and vm2 in network1. > # 6. Create a Static Nat and LB rules for vms in network1. > # 7. Make sure you are not allowed to create a PF rule for any Vm in > # network1 and the Static Nat and LB rules for vms work as expected > # 8. Update network1 to NO2. > Management server logs are attached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira