incubator-cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailaja Mada (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-1677) Management server failed to start with out of memory
Date Thu, 21 Mar 2013 04:39:15 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sailaja Mada updated CLOUDSTACK-1677:
-------------------------------------

    Attachment: management-server.log
                apilog.log
    
> Management server failed to start with out of memory 
> -----------------------------------------------------
>
>                 Key: CLOUDSTACK-1677
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1677
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.2.0
>            Reporter: Sailaja Mada
>            Assignee: Kelven Yang
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: access_log.2013-03-14.txt, apilog.log, apilog.log, management-server.log,
management-server.log
>
>
> Setup: Advanced Networking Zone with Xen 6.1 host 
> Observation:
> Management server is idle for more than 12 hrs.   Its not coming up with out of memory.
> [root@rhel63 management]# ps -ef| grep cloud
> root      4741  4621  0 11:04 pts/0    00:00:00 grep cloud
> cloud    24529     1  0 Mar13 ?        00:10:17 /usr/lib/jvm/jre/bin/java -Djava.awt.headless=true
-Dcom.sun.management.jmxremote.port=45219 -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/
-XX:PermSize=512M -XX:MaxPermSize=800m -classpath :::/etc/cloudstack/management:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management/bin/bootstrap.jar:/usr/share/cloudstack-management/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
-Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management
-Djava.endorsed.dirs= -Djava.io.tmpdir=/usr/share/cloudstack-management/temp -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap
start
> [root@rhel63 management]#
> Observed error log while trying to access Management server :
> 2013-03-14 11:04:34,042 ERROR [cloud.api.ApiServlet] (catalina-exec-16:null) unknown
exception writing api response
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@248a5cea:
SELECT domain.id, domain.parent, domain.name, domain.owner, domain.path, domain.level, domain.removed,
domain.child_count, domain.next_child_seq, domain.state, domain.network_domain, domain.uuid,
domain.region_id FROM domain WHERE domain.path = _binary'/'  AND domain.removed IS NULL  ORDER
BY RAND() LIMIT 1
>         at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:415)
>         at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:350)
>         at com.cloud.utils.db.GenericDaoBase.findOneIncludingRemovedBy(GenericDaoBase.java:860)
>         at com.cloud.utils.db.GenericDaoBase.findOneBy(GenericDaoBase.java:871)
>         at com.cloud.domain.dao.DomainDaoImpl.findDomainByPath(DomainDaoImpl.java:200)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:40)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at $Proxy49.findDomainByPath(Unknown Source)
>         at com.cloud.user.DomainManagerImpl.findDomainByPath(DomainManagerImpl.java:175)
>         at com.cloud.user.DomainManagerImpl.findDomainByPath(DomainManagerImpl.java:65)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:46)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>  at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at $Proxy134.findDomainByPath(Unknown Source)
>         at com.cloud.api.ApiServer.loginUser(ApiServer.java:750)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:46)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at $Proxy384.loginUser(Unknown Source)
>         at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:210)
>         at com.cloud.api.ApiServlet.doPost(ApiServlet.java:71)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>         at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>         at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
>         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully
received from the server was 63,725,617 milliseconds ago.  The last packet sent successfully
to the server was 63,725,617 milliseconds ago. is longer than the server configured value
of 'wait_timeout'. You should consider either expiring and/or testing connection validity
before use in your application, increasing the server configured values for client timeouts,
or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>         at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
>         at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3829)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2449)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>         at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
>         at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:409)
>         ... 86 more
> Caused by: java.net.SocketException: Broken pipe
>         ... 100 more
> 2013-03-14 11:04:34,047 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:null) ===END===
 10.144.6.39 -- POST  null
> 2013-03-14 11:04:35,577 DEBUG [cloud.api.ApiServlet] (catalina-exec-23:null) ===START===
 10.144.6.39 -- POST  null

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message