Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Mon, 13 Jan 2014 11:21:55 +0000 (UTC)
From: "Murali Reddy (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12688053.1389268937161.86070.1389612115196@arcas>
In-Reply-To: <JIRA.12688053.1389268937161@arcas>
References: <JIRA.12688053.1389268937161@arcas>
Subject: [jira] [Comment Edited] (CLOUDSTACK-5849) Failed to shutdown the
 network when corresponding External LB provider gets Disabled while still
 in use by the network
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CLOUDSTACK-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869424#comment-13869424 ] 

Murali Reddy edited comment on CLOUDSTACK-5849 at 1/13/14 11:21 AM:
--------------------------------------------------------------------

Here is my analysis of the bug after looking at the logs.

Network failed to implement in 3.0.6 deployment, because of the error:

com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: There are no NetScaler load balancer devices with the free capacity for implementing this network                                           
at com.cloud.network.element.NetscalerElement.implement(NetscalerElement.java:176)

So there is NetScaler allocated for the network and subsequently there is no referance for the network in 'network_external_lb_device_map' as well. Network went into 'Shutdown' state as expected. There is know issue in 3.0.* releases where network in 'Shutdown' state can not be deleted which is why there is error:

2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75) Network is not not in the correct state to be destroyed: Shutdown. Opened 'CLOUDSTACK-5861' to enable force deleting a network.

Other problem reported in the bug is on upgrade network state changed from 'Shutdown' to 'Implemented', this is a upgrade issue which needs to be fixed.


was (Author: murali.reddy):
Here is my analysis of the bug after looking at the logs.

Network failed to implement in 3.0.6 deployment, because of the error:

com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: There are no NetScaler load balancer devices with the free capacity for implementing this network                                           
at com.cloud.network.element.NetscalerElement.implement(NetscalerElement.java:176)

So there is NetScaler allocated for the network and subsequently there is no referance for the network in 'network_external_lb_device_map' as well. Network went into 'Shutdown' state as expected. There is know issue in 3.0.* releases where network in 'Shutdown' state can not be deleted which is why there is error:

2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75) Network is not not in the correct state to be destroyed: Shutdown. Opened 'CLOUDSTACK-5861' to enable force deleting a network.

Other problem reported in the bug is on upgrade network state changed from 'Shutdown' to 'Implemented', this is a upgrade which needs to be fixed.


> Failed to shutdown the network when corresponding External LB provider gets Disabled while still in use by the network
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5849
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5849
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Network Controller, Network Devices
>    Affects Versions: 4.3.0
>         Environment: upgraded from 3.0.6 patch E to 4.3
>            Reporter: manasaveloori
>            Assignee: Murali Reddy
>             Fix For: 4.3.0
>
>         Attachments: management-server306.log.rar, management-server4.3.rar, mysqldump306PatchE.dmp, mysqldump4.3.dmp
>
>
> Steps:
> 1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
> 2. Created the service offering for netscaler device.
> 3. Added  incompatible   version of Netscaler. added NS10.1: Build 120.1316 to CS 3.0.6 build.
> it was success.
> 4. Now created the network using the service offering created in step 2.
> 5. Observed that the network went into shut down state as the netscaler failed to implement the network.
> 6. Removed the netscaler device from CS and disabled the service offering.
> 7. Now there were no VMs associated to that network.Tried to delete the network.But it failed as the network was not in correct state.
> Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown state.
> 2014-01-07 21:32:09,720 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null) Details from executing class com.cloud.agent.api.NetworkUsageCommand:
> 2014-01-07 21:32:09,720 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Recieved and Sent bytes are both 0. Not updating user_statistics
> 2014-01-07 21:32:11,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) Ping from 8
> 2014-01-07 21:32:11,917 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) Ping from 5
> 2014-01-07 21:32:12,164 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-19:null) submit async job-75, job: AsyncJobVO {id:75, userId: 2, accountId: 2, sessionKey: null, instanceType: null, instanceId: null, cmd: com.cloud.api.commands.DeleteNetworkCmd, cmdOriginator: null, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-07 21:32:12,169 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd for job-75
> 2014-01-07 21:32:12,179 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75) Sync job-75 execution on object network.216
> 2014-01-07 21:32:12,188 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75) job com.cloud.api.commands.DeleteNetworkCmd for job-75 was queued, processing the queue.
> 2014-01-07 21:32:12,197 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75) Executing sync queue item: SyncQueueItemVO {id:17, queueId: 17, contentType: AsyncJob, contentId: 75, lastProcessMsid: 6642334695485, lastprocessNumber: 1, lastProcessTime: Tue Jan 07 21:32:12 IST 2014, created: Tue Jan 07 21:32:12 IST 2014}
> 2014-01-07 21:32:12,199 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75) Schedule queued job-75
> 2014-01-07 21:32:12,207 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-69:job-75) There is a pending process in sync queue(id: 17)
> 2014-01-07 21:32:12,210 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-70:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd for job-75
> 2014-01-07 21:32:12,233 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75) Network is not implemented: Ntwk[216|Guest|17]
> 2014-01-07 21:32:12,233 DEBUG [db.Transaction.Transaction] (Job-Executor-70:job-75) Rolling back the transaction: Time = 1 Name =  -AsyncJobManagerImpl$1.run:396-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679; called by -Transaction.rollback:854-Transaction.removeUpTo:797-Transaction.close:621-DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-NetworkManagerImpl.destroyNetwork:3799-DatabaseCallback.intercept:30-NetworkManagerImpl.deleteNetwork:3656-ActionEventCallback.intercept:32-DeleteNetworkCmd.execute:65-ApiDispatcher.dispatch:263-AsyncJobManagerImpl$1.run:430
> 2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75) Network is not not in the correct state to be destroyed: Shutdown
> 2014-01-07 21:32:12,242 WARN  [cloud.api.ApiDispatcher] (Job-Executor-70:job-75) class com.cloud.api.ServerApiException : Failed to delete network
> 2014-01-07 21:32:12,243 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-70:job-75) Complete async job-75, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to delete network
> 8. Now upgraded the CS to 4.3.
> 9. The network state changed to implemented.
> 10.Tried to delete the network now which failed.Following Error is observed:
> 2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b) ===START===  10.252.192.34 -- GET  command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) submit async job-217, details: AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo: {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) ===END===  10.252.192.34 -- GET  command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,463 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-39:ctx-a33dc0eb) Add job-217 into job monitoring
> 2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb) Executing AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo: {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is acquired for network Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 port forwarding rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 static nat rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Public load balancing rules for network id=217
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules to forward to the network elements
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Internal load balancing rules for network id=217
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules to forward to the network elements
> 2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall ingress rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall egress rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) applying default firewall egress rules
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need to apply firewall rules on the backend; virtual router doesn't exist in the network 217
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Found 0 static nat rules to apply for network id 217
> 2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need to associate ip addresses on the backend; virtual router doesn't exist in the network 217
> 2014-01-09 22:51:18,583 WARN  [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Unable to complete shutdown of the network elements due to element: Netscaler either doesn't exist or not enabled in the physical network 200
> 2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to Netscaler
> 2014-01-09 22:51:18,588 WARN  [c.c.n.ExternalLoadBalancerDeviceManagerImpl] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network shutdwon requested on external load balancer element, which did not implement the network. Either network implement failed half way through or already network shutdown is completed. So just returning.
> 2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to VirtualRouter
> 2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is released for network Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network is not not in the correct state to be destroyed: Implemented
> 2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb) Complete async job-217, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to delete network"}
> 2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb) Done executing org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
> Attaching the MS ,DB dumps.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)