cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alena Prokharchyk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-5849) Unable to delete the network after upgrading from 3.0.6 to 4.3
Date Thu, 09 Jan 2014 20:57:50 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867040#comment-13867040
] 

Alena Prokharchyk commented on CLOUDSTACK-5849:
-----------------------------------------------

It has nothing to do with the Netscaler versioning. Netscaler serves as and LB provider for
certain networks:

mysql> select * from ntwk_service_map where provider not like '%VirtualRouter%' and service='Lb'
and network_id in (select id from networks where removed is null);
+----+------------+---------+-----------+---------------------+
| id | network_id | service | provider  | created             |
+----+------------+---------+-----------+---------------------+
| 76 |        216 | Lb      | Netscaler | 2014-01-07 15:58:21 |
| 83 |        217 | Lb      | Netscaler | 2014-01-07 15:59:54 |
+----+------------+---------+-----------+---------------------+
2 rows in set (0.00 sec)

Then some time along the way (may be in 3.0.6) the provider was disabled on the physical network:

mysql> select provider_name, state, removed from physical_network_service_providers where
provider_name='Netscaler';
+---------------+----------+---------+
| provider_name | state    | removed |
+---------------+----------+---------+
| Netscaler     | Disabled | NULL    |
| Netscaler     | Disabled | NULL    |
+---------------+----------+---------+
2 rows in set (0.00 sec)

Not sure if because of that, but the netscaler was removed from the netscaler/network references:

mysql> select * from network_external_lb_device_map;
Empty set (0.00 sec)


When the network gets shutdown, the call expects the reference to be there. When it can't
be found, it results in the Exception. 

This bug most likely has nothing to do with the upgrade. Murali, assigning it to you as you
wrote External Devices logic, and you know the expected behavior/original design better. 
>From my point of view, we shouldn't delete the references when there are active networks
still using it. If so, you should fix the upgrade code to add the references back to network_external_lb_device_map
table.



> Unable to delete the network after upgrading from 3.0.6 to 4.3
> --------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5849
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5849
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Network Controller, Network Devices
>    Affects Versions: 4.3.0
>         Environment: upgraded from 3.0.6 patch E to 4.3
>            Reporter: manasaveloori
>            Assignee: Alena Prokharchyk
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: management-server306.log.rar, management-server4.3.rar, mysqldump306PatchE.dmp,
mysqldump4.3.dmp
>
>
> Steps:
> 1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
> 2. Created the service offering for netscaler device.
> 3. Added  incompatible   version of Netscaler. added NS10.1: Build 120.1316 to CS 3.0.6
build.
> it was success.
> 4. Now created the network using the service offering created in step 2.
> 5. Observed that the network went into shut down state as the netscaler failed to implement
the network.
> 6. Removed the netscaler device from CS and disabled the service offering.
> 7. Now there were no VMs associated to that network.Tried to delete the network.But it
failed as the network was not in correct state.
> Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown state.
> 2014-01-07 21:32:09,720 DEBUG [agent.manager.AgentManagerImpl] (RouterMonitor-1:null)
Details from executing class com.cloud.agent.api.NetworkUsageCommand:
> 2014-01-07 21:32:09,720 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null)
Recieved and Sent bytes are both 0. Not updating user_statistics
> 2014-01-07 21:32:11,498 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null)
Ping from 8
> 2014-01-07 21:32:11,917 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null)
Ping from 5
> 2014-01-07 21:32:12,164 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-19:null)
submit async job-75, job: AsyncJobVO {id:75, userId: 2, accountId: 2, sessionKey: null, instanceType:
null, instanceId: null, cmd: com.cloud.api.commands.DeleteNetworkCmd, cmdOriginator: null,
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
> 2014-01-07 21:32:12,169 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75)
Executing com.cloud.api.commands.DeleteNetworkCmd for job-75
> 2014-01-07 21:32:12,179 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75)
Sync job-75 execution on object network.216
> 2014-01-07 21:32:12,188 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75)
job com.cloud.api.commands.DeleteNetworkCmd for job-75 was queued, processing the queue.
> 2014-01-07 21:32:12,197 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75)
Executing sync queue item: SyncQueueItemVO {id:17, queueId: 17, contentType: AsyncJob, contentId:
75, lastProcessMsid: 6642334695485, lastprocessNumber: 1, lastProcessTime: Tue Jan 07 21:32:12
IST 2014, created: Tue Jan 07 21:32:12 IST 2014}
> 2014-01-07 21:32:12,199 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-75)
Schedule queued job-75
> 2014-01-07 21:32:12,207 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-69:job-75)
There is a pending process in sync queue(id: 17)
> 2014-01-07 21:32:12,210 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-70:job-75)
Executing com.cloud.api.commands.DeleteNetworkCmd for job-75
> 2014-01-07 21:32:12,233 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75)
Network is not implemented: Ntwk[216|Guest|17]
> 2014-01-07 21:32:12,233 DEBUG [db.Transaction.Transaction] (Job-Executor-70:job-75) Rolling
back the transaction: Time = 1 Name =  -AsyncJobManagerImpl$1.run:396-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679;
called by -Transaction.rollback:854-Transaction.removeUpTo:797-Transaction.close:621-DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-NetworkManagerImpl.destroyNetwork:3799-DatabaseCallback.intercept:30-NetworkManagerImpl.deleteNetwork:3656-ActionEventCallback.intercept:32-DeleteNetworkCmd.execute:65-ApiDispatcher.dispatch:263-AsyncJobManagerImpl$1.run:430
> 2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-70:job-75)
Network is not not in the correct state to be destroyed: Shutdown
> 2014-01-07 21:32:12,242 WARN  [cloud.api.ApiDispatcher] (Job-Executor-70:job-75) class
com.cloud.api.ServerApiException : Failed to delete network
> 2014-01-07 21:32:12,243 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-70:job-75)
Complete async job-75, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text:
Failed to delete network
> 8. Now upgraded the CS to 4.3.
> 9. The network state changed to implemented.
> 10.Tried to delete the network now which failed.Following Error is observed:
> 2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b) ===START===
 10.252.192.34 -- GET  command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-25:ctx-20a7144b
ctx-bbdb1ebe) submit async job-217, details: AsyncJobVO {id:217, userId: 2, accountId: 2,
instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
cmdInfo: {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
6642334695485, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe)
===END===  10.252.192.34 -- GET  command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,463 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-39:ctx-a33dc0eb)
Add job-217 into job monitoring
> 2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb)
Executing AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId: null,
cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo: {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
6642334695485, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Lock is acquired for network Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Releasing 0 port forwarding rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Releasing 0 static nat rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Revoking 0 Public load balancing rules for network id=217
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no Load Balancing Rules to forward to the network elements
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Revoking 0 Internal load balancing rules for network id=217
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no Load Balancing Rules to forward to the network elements
> 2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Releasing 0 firewall ingress rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Releasing 0 firewall egress rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) applying default firewall egress rules
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Virtual router elemnt doesn't need to apply firewall rules on the backend; virtual
router doesn't exist in the network 217
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) There are no rules to forward to the network elements
> 2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Found 0 static nat rules to apply for network id 217
> 2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Virtual router elemnt doesn't need to associate ip addresses on the backend;
virtual router doesn't exist in the network 217
> 2014-01-09 22:51:18,583 WARN  [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Unable to complete shutdown of the network elements due to element: Netscaler
either doesn't exist or not enabled in the physical network 200
> 2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Sending network shutdown to Netscaler
> 2014-01-09 22:51:18,588 WARN  [c.c.n.ExternalLoadBalancerDeviceManagerImpl] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Network shutdwon requested on external load balancer element, which did not
implement the network. Either network implement failed half way through or already network
shutdown is completed. So just returning.
> 2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Sending network shutdown to VirtualRouter
> 2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Lock is released for network Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-39:ctx-a33dc0eb
ctx-bbdb1ebe) Network is not not in the correct state to be destroyed: Implemented
> 2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb)
Complete async job-217, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
> 2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-39:ctx-a33dc0eb)
Done executing org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
> Attaching the MS ,DB dumps.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message