cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangeetha Hariharan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-1007) Not able to delete Shared network because of not being able to stop the router.
Date Fri, 18 Jan 2013 18:52:17 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557471#comment-13557471
] 

Sangeetha Hariharan commented on CLOUDSTACK-1007:
-------------------------------------------------

Tested with latest build from "network-refactor" branch:

I am able to successfully delete the Shared network that has no Vms assocaiated with it.


We are able to stop the router successfully as part of destroying the router.

management server logs:

2013-01-18 13:32:46,785 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-2:job-58)
Attempting to destroy router 10
2013-01-18 13:32:46,803 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-2:job-58)
VM state transitted from :Running to Stopping with event: StopRequestedvm's original host
id: 1 new host id: 1 host id before state transition: 1
2013-01-18 13:32:46,821 DEBUG [agent.transport.Request] (Job-Executor-2:job-58) Seq 1-386072589:
Sending  { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}]
}
2013-01-18 13:32:51,672 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===START===  10.217.252.59
-- GET  command=queryAsyncJobResult&jobId=f38eac22-30aa-4f56-b886-8f9eb3d0a25b&response=json&sessionkey=bgGUZzPWiG2VDGjCsU3oUROHbms%3D&_=1358534197988
2013-01-18 13:32:51,710 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===END===  10.217.252.59
-- GET  command=queryAsyncJobResult&jobId=f38eac22-30aa-4f56-b886-8f9eb3d0a25b&response=json&sessionkey=bgGUZzPWiG2VDGjCsU3oUROHbms%3D&_=1358534197988
2013-01-18 13:32:53,398 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq
1-386072589: Processing:  { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}]
}
2013-01-18 13:32:53,399 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-13:null)
Seq 1-386072589: No more commands found
2013-01-18 13:32:53,399 DEBUG [agent.transport.Request] (Job-Executor-2:job-58) Seq 1-386072589:
Received:  { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, { StopAnswer } }
2013-01-18 13:32:53,409 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-2:job-58)
Successfully updated user statistics as a part of domR VM[DomainRouter|r-10-VM] reboot/stop
2013-01-18 13:32:53,445 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-2:job-58)
VM[DomainRouter|r-10-VM] is stopped on the host.  Proceeding to release resource held.

Agent.log:
2013-01-18 18:36:44,388 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq
1-386072589:  { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}]
}
2013-01-18 18:36:44,388 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing
command: com.cloud.agent.api.StopCommand
2013-01-18 18:36:44,489 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /usr/lib64/cloud/common/scripts/vm/network/security_group.py destroy_network_rules_for_vm
--vmname r-10-VM --vif vnet12
2013-01-18 18:36:44,708 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
2013-01-18 18:36:44,709 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Try to stop the vm at first
2013-01-18 18:36:50,956 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Seq 1-386072589:
 { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}]
}
2013-01-18 18:36:51,307 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Request:Seq
1-386072590:  { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"storage.DestroyCommand":{"vmName":"r-10-VM","volume":{"id":10,"name":"ROOT-10","mountPoint":"/export/home/sangeetha/asf/primary","path":"bde8865f-aed6-4072-8c2c-43025219cbd7","size":725811200,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"afcf9136-8f08-3361-8328-fac0f471b3e4","deviceId":0},"wait":0}}]
}
2013-01-18 18:36:51,307 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Processing
command: com.cloud.agent.api.storage.DestroyCommand
2013-01-18 18:36:51,663 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null)
requested delete disk /mnt/afcf9136-8f08-3361-8328-fac0f471b3e4/r-10-VM-patchdisk
2013-01-18 18:36:51,672 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Seq 1-386072590:
 { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"Answer":{"result":true,"details":"Success","wait":0}}]
}


                
> Not able to delete Shared network because of not being able to stop the router.
> -------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-1007
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1007
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.1.0
>         Environment: Build from "network-refactor"
>            Reporter: Sangeetha Hariharan
>            Assignee: Marcus Sorensen
>             Fix For: 4.1.0
>
>         Attachments: agent.log, agent.log.2013-01-17.gz, management-server.zip
>
>
> Steps to reproduce the problem:
> Set up  - Advanced zone with KVM host.
> Create a Zone wide Shared network.
> Deploy Vm using this network.
> Destroy this Vm.
> Wait for the Vm to get Expunged.
> Delete the network.
> Network deletion fails because we are not able to destroy the router.
> Management server logs:
> 2013-01-17 18:05:31,818 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-6:job-34)
Sending destroy to com.cloud.network.element.VirtualRouterElement$$EnhancerByCGLIB$$d659f5e7@447d4275
> 2013-01-17 18:05:31,820 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-6:job-34)
Attempting to destroy router 10
> 2013-01-17 18:05:31,840 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-6:job-34)
VM state transitted from :Running to Stopping with event: StopRequestedvm's original host
id: 1 new host id: 1 host id before state transition: 1
> 2013-01-17 18:05:31,850 DEBUG [agent.transport.Request] (Job-Executor-6:job-34) Seq 1-772669469:
Sending  { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}]
}
> 2013-01-17 18:05:33,284 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector
is running...
> 2013-01-17 18:05:33,883 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-772669470:
Received:  { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer
} }
> 2013-01-17 18:05:36,715 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===START===
 10.217.252.59 -- GET  command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464170203
> 2013-01-17 18:05:36,746 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END===
 10.217.252.59 -- GET  command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464170203
> 2013-01-17 18:05:38,232 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null)
Seq 1-772669469: Processing:  { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110,
[{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}]
}
> 2013-01-17 18:05:38,232 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null)
Seq 1-772669469: No more commands found
> 2013-01-17 18:05:38,232 DEBUG [agent.transport.Request] (Job-Executor-6:job-34) Seq 1-772669469:
Received:  { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, { Answer } }
> 2013-01-17 18:05:38,232 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34)
Unable to stop vm VM[DomainRouter|r-10-VM]
> 2013-01-17 18:05:38,299 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-6:job-34)
VM state transitted from :Stopping to Running with event: OperationFailedvm's original host
id: 1 new host id: 1 host id before state transition: 1
> 2013-01-17 18:05:38,300 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34)
Unable to stop the VM so we can't expunge it.
> 2013-01-17 18:05:38,300 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34)
Unable to destroy the vm because it is not in the correct state: VM[DomainRouter|r-10-VM]
> 2013-01-17 18:05:38,300 INFO  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34)
Did not expunge VM[DomainRouter|r-10-VM]
> 2013-01-17 18:05:38,300 WARN  [cloud.network.NetworkManagerImpl] (Job-Executor-6:job-34)
Unable to complete destroy of the network: failed to destroy network element VirtualRouter
> 2013-01-17 18:05:38,317 WARN  [cloud.api.ApiDispatcher] (Job-Executor-6:job-34) class
com.cloud.api.ServerApiException : Failed to delete network
> 2013-01-17 18:05:38,318 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-6:job-34)
Complete async job-34, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text:
Failed to delete network
> 2013-01-17 18:05:38,352 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-6:job-34)
Sync queue (10) is currently empty
> 2013-01-17 18:05:38,352 WARN  [cloud.async.AsyncJobManagerImpl] (Job-Executor-6:job-34)
Unable to unregister active job 34 from JMX monitoring
> 2013-01-17 18:05:41,714 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) ===START===
 10.217.252.59 -- GET  command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464175201
> 2013-01-17 18:05:41,728 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-8:null)
Async job-34 completed
> in agent.log:
> 013-01-17 23:09:38,641 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq
1-772669469:  { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}]
}
> 2013-01-17 23:09:38,641 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing
command: com.cloud.agent.api.StopCommand
> 2013-01-17 23:09:38,703 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null)
Executing: /usr/lib64/cloud/common/scripts/vm/network/security_group.py destroy_network_rules_for_vm
--vmname r-10-VM --vif vnet12
> 2013-01-17 23:09:38,857 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null)
Execution is successful.
> 2013-01-17 23:09:38,858 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null)
Try to stop the vm at first
> 2013-01-17 23:09:40,083 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing
command: com.cloud.agent.api.GetHostStatsCommand
> 2013-01-17 23:09:40,083 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /bin/bash -c idle=$(top -b -n 1|grep Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> 2013-01-17 23:09:40,612 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
> 2013-01-17 23:09:40,613 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /bin/bash -c freeMem=$(free|grep cache:|awk '{print $4}');echo $freeMem
> 2013-01-17 23:09:40,627 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
> 2013-01-17 23:09:40,627 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /bin/bash -c free|grep Mem:|awk '{print $2}'
> 2013-01-17 23:09:40,644 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
> 2013-01-17 23:09:40,645 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /bin/bash -c cat /sys/class/net/cloudbr0/statistics/rx_bytes
> 2013-01-17 23:09:40,656 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
> 2013-01-17 23:09:40,657 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Executing: /bin/bash -c cat /sys/class/net/cloudbr0/statistics/tx_bytes
> 2013-01-17 23:09:40,668 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
Execution is successful.
> 2013-01-17 23:09:45,014 WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught:
> java.lang.NullPointerException
>         at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)
>         at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)
>         at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)
>         at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)
>         at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)
>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
>         at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-01-17 23:09:45,017 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 1-772669469:
 { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}]
}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message