Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2969A10F50 for ; Sat, 11 Jan 2014 05:49:11 +0000 (UTC) Received: (qmail 62335 invoked by uid 500); 11 Jan 2014 05:48:55 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 62268 invoked by uid 500); 11 Jan 2014 05:48:51 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 62255 invoked by uid 500); 11 Jan 2014 05:48:50 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 62251 invoked by uid 99); 11 Jan 2014 05:48:50 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jan 2014 05:48:50 +0000 Date: Sat, 11 Jan 2014 05:48:50 +0000 (UTC) From: "Animesh Chaturvedi (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-5747) [Upgrade]Network restart failed after upgarding from 2.2.16 to 4.3 with External Firewall SRX added to CS. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-5747?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Animesh Chaturvedi updated CLOUDSTACK-5747: ------------------------------------------- Priority: Major (was: Critical) > [Upgrade]Network restart failed after upgarding from 2.2.16 to 4.3 with E= xternal Firewall SRX added to CS. > -------------------------------------------------------------------------= --------------------------------- > > Key: CLOUDSTACK-5747 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-574= 7 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Network Controller, Network Devices, Upgrade > Affects Versions: 4.3.0 > Environment: upgraded the CS2.2.16 with SRX to 4.3 > Reporter: manasaveloori > Assignee: Jayapal Reddy > Fix For: 4.3.0 > > Attachments: management-server.log.rar, mysqldump2216.dmp, mysqld= ump4.3.dmp > > > Steps: > 1. Deploy CS 2.2 X.16 using Xen5.6 sp2 HV. > 2. Add the External firewall SRX to CS. > 3. Set the GC parameter firewall.rule.ui.enabled to "true." > 4. Now acquire the IP and configure firewall and PF rules. > 5. Upgrade the CS to 4.3. > 6. Stop and start all the System VMs and router VMs so that the new templ= ate is upgraded. > 7. Now perform Network restart on which the firwall and PF rules are conf= igured. > Observation : > Observed the follwoing exceptions in Ms logs and Network restart failed. > 2014-01-03 17:43:32,329 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Added Egress firewall rule for guest network 965 > 2014-01-03 17:43:32,329 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Sending request: > 2014-01-03 17:43:33,966 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Checking response: mgd[edit security = policies from-zone trust to-zone untrust policy egress-trust-untrust-965]matchMissing mandatory statement:= 'source-address'commit failed: (missing statements) > 2014-01-03 17:43:33,966 ERROR [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Request failed due to: Missing mandatory statement: 'source= -address' > 2014-01-03 17:43:33,967 ERROR [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) com.cloud.utils.exception.ExecutionException: Failed to com= mit to global configuration. > 2014-01-03 17:43:33,967 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Sending request: > 2014-01-03 17:43:34,012 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Checking response: > 2014-01-03 17:43:34,012 DEBUG [c.c.n.r.JuniperSrxResource] (DirectAgent-7= 6:ctx-2128a4a2) Closed private configuration. > 2014-01-03 17:43:34,013 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-7= 6:ctx-2128a4a2) Seq 5-1877934113: Response Received: > 2014-01-03 17:43:34,014 DEBUG [c.c.a.t.Request] (DirectAgent-76:ctx-2128a= 4a2) Seq 5-1877934113: Processing: { Ans: , MgmtId: 7588401905746, via: 5,= Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"detail= s":"Exception: com.cloud.utils.exception.ExecutionException\nMessage: Faile= d to commit to global configuration.\nStack: com.cloud.utils.exception.Exec= utionException: Failed to commit to global configuration.\n\tat com.cloud.n= etwork.resource.JuniperSrxResource.commitConfiguration(JuniperSrxResource.j= ava:654)\n\tat com.cloud.network.resource.JuniperSrxResource.execute(Junipe= rSrxResource.java:881)\n\tat com.cloud.network.resource.JuniperSrxResource.= execute(JuniperSrxResource.java:912)\n\tat com.cloud.network.resource.Junip= erSrxResource.execute(JuniperSrxResource.java:912)\n\tat com.cloud.network.= resource.JuniperSrxResource.execute(JuniperSrxResource.java:912)\n\tat com.= cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:8= 30)\n\tat com.cloud.network.resource.JuniperSrxResource.executeRequest(Juni= perSrxResource.java:353)\n\tat com.cloud.agent.manager.DirectAgentAttache$T= ask.runInContext(DirectAgentAttache.java:216)\n\tat org.apache.cloudstack.m= anaged.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)= \n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c= all(DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.cont= ext.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:1= 03)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.= runWithContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.m= anaged.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n= \tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471= )\n\tat java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\= n\tat java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat java.u= til.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(S= cheduledThreadPoolExecutor.java:165)\n\tat java.util.concurrent.ScheduledTh= readPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:2= 66)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecu= tor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:636)\n","w= ait":0}}] } > 2014-01-03 17:43:34,014 DEBUG [c.c.a.t.Request] (Job-Executor-53:ctx-4d95= c752 ctx-99ce704c) Seq 5-1877934113: Received: { Ans: , MgmtId: 7588401905= 746, via: 5, Ver: v1, Flags: 10, { Answer } } > 2014-01-03 17:43:34,015 DEBUG [c.c.a.m.AgentManagerImpl] (Job-Executor-53= :ctx-4d95c752 ctx-99ce704c) Details from executing class com.cloud.agent.ap= i.routing.SetFirewallRulesCommand: Exception: com.cloud.utils.exception.Exe= cutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to = global configuration. > at com.cloud.network.resource.JuniperSrxResource.commitConfigurat= ion(JuniperSrxResource.java:654) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:881) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:830) > at com.cloud.network.resource.JuniperSrxResource.executeRequest(J= uniperSrxResource.java:353) > at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(D= irectAgentAttache.java:216) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2014-01-03 17:43:34,015 ERROR [c.c.n.ExternalFirewallDeviceManagerImpl] (= Job-Executor-53:ctx-4d95c752 ctx-99ce704c) External firewall was unable to = apply static nat rules to the SRX appliance in zone zonexen due to: Excepti= on: com.cloud.utils.exception.ExecutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to = global configuration. > at com.cloud.network.resource.JuniperSrxResource.commitConfigurat= ion(JuniperSrxResource.java:654) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:881) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:830) > at com.cloud.network.resource.JuniperSrxResource.executeRequest(J= uniperSrxResource.java:353) > at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(D= irectAgentAttache.java:216) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > . > 2014-01-03 17:43:34,015 WARN [c.c.n.f.FirewallManagerImpl] (Job-Executor= -53:ctx-4d95c752 ctx-99ce704c) Failed to apply firewall rules due to > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1]= is unreachable: External firewall was unable to apply static nat rules to = the SRX appliance in zone zonexen due to: Exception: com.cloud.utils.except= ion.ExecutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to = global configuration. > at com.cloud.network.resource.JuniperSrxResource.commitConfigurat= ion(JuniperSrxResource.java:654) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:881) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:912) > at com.cloud.network.resource.JuniperSrxResource.execute(JuniperS= rxResource.java:830) > at com.cloud.network.resource.JuniperSrxResource.executeRequest(J= uniperSrxResource.java:353) > at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(D= irectAgentAttache.java:216) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutu= reTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > . > at com.cloud.network.ExternalFirewallDeviceManagerImpl.sendFirewa= llRules(ExternalFirewallDeviceManagerImpl.java:616) > at com.cloud.network.ExternalFirewallDeviceManagerImpl.applyFirew= allRules(ExternalFirewallDeviceManagerImpl.java:573) > at com.cloud.network.element.JuniperSRXExternalFirewallElement.ap= plyFWRules(JuniperSRXExternalFirewallElement.java:233) > at com.cloud.network.firewall.FirewallManagerImpl.applyRules(Fire= wallManagerImpl.java:569) > at com.cloud.network.IpAddressManagerImpl.applyRules(IpAddressMan= agerImpl.java:502) > at com.cloud.network.firewall.FirewallManagerImpl.applyRules(Fire= wallManagerImpl.java:523) > at com.cloud.network.firewall.FirewallManagerImpl.applyFirewallRu= les(FirewallManagerImpl.java:643) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy174.applyFirewallRules(Unknown Source) > at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator= .reprogramNetworkRules(NetworkOrchestrator.java:1106) > at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator= .implementNetworkElementsAndResources(NetworkOrchestrator.java:1063) > at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator= .restartNetwork(NetworkOrchestrator.java:2370) > at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkSer= viceImpl.java:1839) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInter= ceptor.java:50) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:161) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy199.restartNetwork(Unknown Source) > at org.apache.cloudstack.api.command.user.network.RestartNetworkC= md.execute(RestartNetworkCmd.java:92) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJo= bDispatcher.java:109) > at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatche= r.java:66) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatch= er.java:63) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.runInContext(AsyncJobManagerImpl.java:522) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManage > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManaged > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:4 > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2014-01-03 17:43:34,021 WARN [o.a.c.e.o.NetworkOrchestrator] (Job-Execut= or-53:ctx-4d95c752 ctx-99ce704c) > 2014-01-03 17:43:34,055 DEBUG [c.c.a.t.Request] (Job-Executor-53:ctx-4d95= c752 ctx-99ce704c) Seq 5-1877934 = v1, Flags: 100011, [{"com.cloud.ag= ent.api.routing.SetPortForwardingRulesCommand":{"rules":[{"dstIp":"10.0 = = "protocol":"tcp","srcPortRange":[22,22],"revoked":false,"alreadyAdded"= :true,"purpose":"PortForwarding","d = VlanTag":"untagged","srcIp":"10= .147.47.6","protocol":"tcp","srcPortRange":[22,22],"revoked":false,"alread = = ":{},"wait":0}}] } > Attaching the DB dumps and MSlogs: -- This message was sent by Atlassian JIRA (v6.1.5#6160)