cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-4598) [Performance Testing] High delays during deployVM - both network delay and deployment planner delay
Date Wed, 04 Sep 2013 09:56:51 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757612#comment-13757612
] 

Sowmya Krishnan commented on CLOUDSTACK-4598:
---------------------------------------------

Sorry - that must've got ripped off during grep. Error is concerning the Network Lock:

2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f]
com.cloud.exception.ConcurrentOperationException: Unable to lock network 204
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.findOrDeployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1432)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1947)
        at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:221)
        at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2057)
        at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2191)
        at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)

                
> [Performance Testing] High delays during deployVM - both network delay and deployment
planner delay
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4598
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4598
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.2.0
>         Environment: Simulator environment with large scale set up
>            Reporter: Sowmya Krishnan
>             Fix For: 4.2.1
>
>         Attachments: deployVMjob_999.log.gz
>
>
> This is mostly similar to CLOUDSTACK-3441 and CLOUDSTACK-4179. Both these issues were
fixed and verified in comparatively smaller environment with 4K and 8K hosts and 12K VMs
> Now trying in much larger infrastructure with 20k hosts, 20K clusters and 2K Pods. This
is also a special case where we are trying to deploy one VM in each host. 
> I am seeing delay both while acquiring network lock and during deployment planning.
> (There was also an ERROR observed in the log during deployment)
> Log snippet:
> 2013-09-02 22:40:52,335 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Listing clusters in order of aggregate capacity,
that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
> 2013-09-02 22:40:57,544 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Removing from the clusterId list these clusters
from avoid set: []
> ..
> ..
> 2013-09-02 22:41:05,637 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number
>  of nics for network id=204 on 1
> 2013-09-02 22:41:05,690 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-0d530dd3-3f25-4fde-b1fb-9ff9188f89e6-172.4.211.191]
> 2013-09-02 22:51:04,680 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Failed to start instance VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f]
> 2013-09-02 22:51:04,702 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Cleaning up resources for the vm VM[User|414aa09b-a38c-4b30-bf9c-f1d9fe51134f]
in Starting state
> ..
> ..
> 2013-09-02 22:51:17,018 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Changing active number of nics for network id=204
on 1
> 2013-09-02 22:51:17,074 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Asking VirtualRouter to prepare for Nic[2246-1407-159bacce-8663-477e-ab37-2d1081c0630b-172.4.211.191]
> 2013-09-02 22:57:56,139 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is acquired for network id 204 as a part
of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT-->Pool(9749))]
> 2013-09-02 22:57:56,144 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-336:job-999
= [ f437e46a-dfa4-4cea-a518-7da2f5360a89 ]) Lock is released for network id 204 as a part
of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(975)-Cluster(9749)-Host(9750)-Storage(Volume(1407|ROOT-->Pool(9749))]
> ..
> ..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message