cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-3441) [Load Test] High delays between VM being allocated to Pod and network implementation causing delays in VM deployment
Date Fri, 16 Aug 2013 08:30:49 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742030#comment-13742030
] 

Sowmya Krishnan commented on CLOUDSTACK-3441:
---------------------------------------------

I still see this issue with basic zone, 20K Hosts at < 1K VMs. 

2013-08-16 00:15:17,184 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) RAM STATS after allocation: for host: 246, old
used: 0, old reserved: 0, total: 8589934592; new used: 6815744000, reserved: 0; requested
mem: 6815744000,alloc_from_last:false
2013-08-16 00:15:17,195 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) VM is being created in podId: 25
2013-08-16 00:19:37,516 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Lock is acquired for network id 204 as a part
of network implement
2013-08-16 00:19:37,585 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Network id=204 is already implemented
2013-08-16 00:19:37,587 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Lock is released for network id 204 as a part
of network implement

...
...

2013-08-16 00:25:48,958 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) CPU STATS after allocation: for host: 244, old
used
: 0, old reserved: 0, actual total: 32000, total with overprovisioning: 32000; new used:500,
reserved:0; requested cpu:500,alloc_from_last:false
2013-08-16 00:25:48,958 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) RAM STATS after allocation: for host: 244, old
used
: 0, old reserved: 0, total: 8589934592; new used: 134217728, reserved: 0; requested mem:
134217728,alloc_from_last:false
2013-08-16 00:25:49,011 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) VM is being created in podId: 25
2013-08-16 00:32:59,693 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Lock is acquired for network id 204 as a part
of netw
ork implement
2013-08-16 00:32:59,710 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Network id=204 is already implemented
2013-08-16 00:32:59,711 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-586:job-1792
= [ e83085d7-3ef2-44bf-bd53-0df49664f598 ]) Lock is released for network id 204 as a part
of netw
ork implement


mysql> select id, job_cmd, job_status, created, last_updated from async_job where id =
1792\G
*************************** 1. row ***************************
          id: 1792
     job_cmd: org.apache.cloudstack.api.command.user.vm.DeployVMCmd
  job_status: 1
     created: 2013-08-16 07:14:59
last_updated: 2013-08-16 07:33:25
1 row in set (0.00 sec)



Attached the job log for the deploy VM which took 18 mins to complete:job-1792.log

                
> [Load Test] High delays between VM being allocated to Pod and network implementation
causing delays in VM deployment
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-3441
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3441
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server, Network Controller
>    Affects Versions: 4.2.0
>         Environment: Load test, with simulator setup
>            Reporter: Sowmya Krishnan
>            Assignee: Koushik Das
>             Fix For: 4.2.0
>
>         Attachments: job-3014_5mins
>
>
> Scale Test  set up with 20K simulated hosts, 20K VMs, basic zone, 3 Management servers

> Also using security groups, ~5K VMs are deployed in every Security group.
> While deployment of simulator VMs, following is observed after around 4K VMs:
> There's delay between VM being allocated to Pod and network implementation causing delays
in VM deployment. Following are the logs: 
> 2013-06-10 15:51:01,215 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1004:job-3014)
VM is being created in podId: 1390 
> 2013-06-10 15:55:31,494 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1004:job-3014)
Lock is acquired for network id 204 as a part of network implement 
> This causes delay in VM deployment and the async job to complete. The delays get higher
and higher as more and more VMs are deployed causing an unacceptable delay for huge deployments,
viz, > 15-20K  VMs

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message