incubator-cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-1496) List API Performance: listAccounts failing with OOME for high values of pagesize (>1000 )
Date Mon, 04 Mar 2013 12:47:13 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592166#comment-13592166
] 

Sowmya Krishnan commented on CLOUDSTACK-1496:
---------------------------------------------

Logs:

2013-03-04 18:08:23,838 WARN  [cloud.resource.ResourceManagerImpl] (AgentTaskPool-2:null)
Unable to connect due to
com.cloud.utils.exception.CloudRuntimeException: Unable to connect 581
        at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:630)
        at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
        at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at sun.proxy.$Proxy112.handleDirectConnectAgent(Unknown Source)
        at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
        at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
        at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at sun.proxy.$Proxy122.createHostAndAgent(Unknown Source)
        at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Caught: com.mysql.jdbc.PreparedStatement@57a07056:
SELECT vm_instance.id, vm_instance.name, vm_instance.vnc_password, vm_instance.proxy_id, vm_instance.proxy_assign_time,
vm_instance.state, vm_instance.private_ip_address, vm_instance.instance_name, vm_instance.vm_template_id,
vm_instance.guest_os_id, vm_instance.host_id, vm_instance.last_host_id, vm_instance.pod_id,
vm_instance.private_mac_address, vm_instance.data_center_id, vm_instance.vm_type, vm_instance.ha_enabled,
vm_instance.limit_cpu_use, vm_instance.update_count, vm_instance.created, vm_instance.removed,
vm_instance.update_time, vm_instance.domain_id, vm_instance.account_id, vm_instance.service_offering_id,
vm_instance.reservation_id, vm_instance.hypervisor_type, vm_instance.uuid, vm_instance.disk_offering_id,
domain_router.element_id, domain_router.public_ip_address, domain_router.public_mac_address,
domain_router.public_netmask, domain_router.is_redundant_router, domain_router.priority, domain_router.is_priority_bumpup,
domain_router.redundant_state, domain_router.stop_pending, domain_router.role, domain_router.template_version,
domain_router.scripts_version, domain_router.vpc_id FROM domain_router INNER JOIN vm_instance
ON domain_router.id=vm_instance.id   INNER JOIN router_network_ref ON domain_router.id=router_network_ref.router_id
WHERE vm_instance.type='DomainRouter' AND vm_instance.host_id = 581  AND vm_instance.removed
IS NULL  AND  (router_network_ref.guest_type = 'Isolated' )
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:417)
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:350)
        at com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:902)
        at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:879)
        at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:892)
        at com.cloud.vm.dao.DomainRouterDaoImpl.listIsolatedByHostId(DomainRouterDaoImpl.java:217)
        at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at sun.proxy.$Proxy156.listIsolatedByHostId(Unknown Source)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.processConnect(VirtualNetworkApplianceManagerImpl.java:3515)
        at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
        ... 45 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        ... 71 more

                
> List API Performance: listAccounts failing with OOME for high values of pagesize (>1000
)
> -----------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-1496
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1496
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: API
>    Affects Versions: 4.1.0
>         Environment: CentOS release 6.3 
>            Reporter: Sowmya Krishnan
>            Assignee: Min Chen
>             Fix For: 4.1.0
>
>
> listAccounts with higher values of pagesize is resulting in OOME and in general, showing
low performance for smaller page size values too.
> Configuration for the performance set up is as follows:
> (Using simulator set up)
> (Advanced zone)
> Accounts: 2117
> Hosts: 1986
> Users: 2116
> Virtual machines: 3299
> Server configurations:
> Management Server :
> =================
> 8 proc Intel(R) Xeon(R) CPU E5620  @ 2.40GHz processor
> CentOS release 6.3 (Final)
> Database: 
> ========
> 8 proc Intel(R) Xeon(R) CPU E5620  @ 2.40GHz processor
> Red Hat Enterprise Linux Server release 6.2 (Santiago)
> MySQL-server-5.5.21-1.linux2.6.x86_64 (InnoDB engine)
> Following are the results for listAccounts for various pagesize values:
> ======================================================
> (APIs are fired on 8080 as admin from a different client than the MS)
> pagesize=10 : 0m30.944s
> pagesize=50: 1m57.292s
> pagesize=100: 2m59.700s
> pagesize=500: 12m22.807s
> In comparison to the results before API Optimization, for pagesize=100, the response
was received within 62 seconds before optimization whereas now, the result is 149 seconds
> One difference though, is that the performance run before API optimization was done on
a DB server with much higher configuration (Quad core, 8 processor). But still, we shouldn't
be seeing OOME.
> Following is the extract of the result for listAccounts&listAll=true (# of accounts:
2117)
> ...
>  <title>Error 500 Java heap space</title> </head> <body><h2>HTTP
ERROR 500</h2> <p>Problem accessing /client/api. Reason: <pre> Java heap
space</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError:
Java heap space </pre> <hr /><i><small>Powered by Jetty://</small>
> Maven Settings:
> ============
> [root@centos63 ~]# echo $MAVEN_OPTS
> -Xmx5120m -XX:MaxPermSize=1024m -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message