cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sheng Yang (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CLOUDSTACK-689) RVR: Stop pending flag is not cleared when user start the disconnected router from another host
Date Fri, 21 Dec 2012 02:53:15 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sheng Yang resolved CLOUDSTACK-689.
-----------------------------------

    Resolution: Fixed

Fixed at:

commit 53e6825b744252daeab7c4e694adae69c65cf0d3
Author: Sheng Yang <sheng.yang@citrix.com>
Date:   Thu Dec 20 17:01:12 2012 -0800

    CLOUDSTACK-689: Clean stop pending flags when VR start/stop

                
> RVR: Stop pending flag is not cleared when user start the disconnected router from another
host
> -----------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-689
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-689
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Network Controller
>    Affects Versions: 4.0.0
>            Reporter: Sheng Yang
>            Assignee: Sheng Yang
>             Fix For: 4.1.0
>
>
> Here is what's happened: 
> 2012-09-16 21:27:24,285 WARN [network.router.VirtualNetworkApplianceManagerImpl] (HA-Worker-0:work-37779)
Unable to apply dhcp entry on disconnected router r-38445-VM 
> com.cloud.exception.AgentUnavailableException: Resource [Host:610] is unreachable: Host
610: Host is not in the right state: Down 
> [...] 
> 2012-09-16 21:27:24,286 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (HA-Worker-0:work-37779)
About to stop the router r-38445-VM due to: Unable to apply dhcp entry on disconnected router

> At the above time, we set stop pending flag, due to host 610 is down, and we failed to
reach the router r-38445-VM. 
> Then: 
> 2012-09-16 21:51:54,067 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-28:null)
submit async job-296045, details: AsyncJobVO {id:296045, userId: 6217, accountId: 6178, sessionKey:
null, instanceType: DomainRouter, instanceId: 38445, cmd: com.cloud.api.commands.StartRouterCmd,
cmdOriginator: null, cmdInfo: {"response":"json","id":"f14313a7-5490-408f-a9d7-292303784f8f","sessionkey":"lByhfEpE7fygtOASvMH3i8xgEok\u003d","ctxUserId":"6217","_":"1347799961307","ctxAccountId":"6178","ctxStartEventId":"1533940"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 152687106451262, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null} 
> 2012-09-16 21:51:54,070 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-126:job-296045)
Executing com.cloud.api.commands.StartRouterCmd for job-296045 
> User start the router manually, and CS deploy it on host 606: 
> 2012-09-16 21:51:56,069 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-126:job-296045)
Deployment found - P0=VM[DomainRouter|r-38445-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone( 
> 2)-Pod(29)-Cluster(52)-Host(606)-Storage()] 
> 2012-09-16 21:51:56,090 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-126:job-296045)
VM state transitted from :Starting to Starting with event: OperationRetryvm's original host
id: 610 new host id: 606 host id before state transiti 
> on: null 
> We didn't cover this situation in our code. And this won't clear stop pending flag of
course.
> Also the stop pending flag can be set if the command sent to router is time out, but
host is not detected as Disconnected by CS. 
> See the mgmt log attached for r-5-VM: 
> 2012-09-29 22:20:55,413 DEBUG [agent.manager.AgentAttache] (Job-Executor-130:job-450)
Seq 1-1662189781: Cancelling. 
> 2012-09-29 22:20:55,413 WARN [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-130:job-450)
Timed Out 
> com.cloud.exception.OperationTimedoutException: Commands 1662189781 to Host 1 timed out
after 3600 
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:425) 
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:501) 
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1022)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2590)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendPortForwardingRules(VirtualNetworkApplianceManagerImpl.java:2725)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$5.execute(VirtualNetworkApplianceManagerImpl.java:2703)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:2809)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:2688)

>         at com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:636)

>         at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3245)

>         at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:374)

>         at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRulesForNetwork(RulesManagerImpl.java:714)

>         at com.cloud.network.NetworkManagerImpl.reprogramNetworkRules(NetworkManagerImpl.java:3516)

>         at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1812)

>         at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1754)

>         at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30) 
>         at com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:3436)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2285)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2255)

>         at com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:32)

>         at com.cloud.api.commands.StartRouterCmd.execute(StartRouterCmd.java:101) 
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134) 
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:427) 
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

>         at java.lang.Thread.run(Thread.java:636) 
> 2012-09-29 22:20:55,414 WARN [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-130:job-450)
Unable to apply firewall rules on disconnected router r-5-VM 
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host
1: Unable to send commands to virtual router 
>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2593)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendPortForwardingRules(VirtualNetworkApplianceManagerImpl.java:2725)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$5.execute(VirtualNetworkApplianceManagerImpl.java:2703)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:2809)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:2688)

>         at com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:636)

>         at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3245)

>         at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:374)

>         at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRulesForNetwork(RulesManagerImpl.java:714)

>         at com.cloud.network.NetworkManagerImpl.reprogramNetworkRules(NetworkManagerImpl.java:3516)

>         at com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1812)

>         at com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1754)

>         at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30) 
>         at com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:3436)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2285)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2255)

>         at com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:32)

>         at com.cloud.api.commands.StartRouterCmd.execute(StartRouterCmd.java:101) 
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134) 
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:427) 
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

>         at java.lang.Thread.run(Thread.java:636) 
> Caused by: com.cloud.exception.OperationTimedoutException: Commands 1662189781 to Host
1 timed out after 3600 
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:425) 
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:501) 
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1022)

>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2590)

>         ... 33 more 
> 2012-09-29 22:20:55,415 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-130:job-450)
About to stop the router r-5-VM due to: Unable to apply firewall rules on disconnected router

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message