cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Phillips <mphilli7...@hotmail.com>
Subject RE: Interesting 4.2.1. Issue...
Date Fri, 04 Apr 2014 02:40:50 GMT
So did you try changing your version of tomcat?

> Subject: RE: Interesting 4.2.1. Issue...
> From: amin@opencloud.net.au
> To: dev@cloudstack.apache.org
> Date: Fri, 4 Apr 2014 10:35:42 +0800
> 
> cd /usr/share/tomcat6/bin/
> ./version.sh
> 
> The output should be 6.0.33 instead of 6.0.35
> 
> Using CATALINA_BASE:   /usr/share/tomcat6
> Using CATALINA_HOME:   /usr/share/tomcat6
> Using CATALINA_TMPDIR: /usr/share/tomcat6/temp
> Using JRE_HOME:        /usr
> Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
> Server version: Apache Tomcat/6.0.35
> Server built:   
> Server number:  6.0.35.0
> OS Name:        Linux
> OS Version:     3.11.0-18-generic
> Architecture:   amd64
> JVM Version:    1.6.0_30-b30
> JVM Vendor:     Sun Microsystems Inc.
> 
> 
> Kind Regards
> Amin 
> 
> 
> 
> -----Original Message-----
> From: Michael Phillips [mailto:mphilli7823@hotmail.com] 
> Sent: Friday, 4 April 2014 10:31 AM
> To: dev@cloudstack.apache.org
> Subject: RE: Interesting 4.2.1. Issue...
> 
> I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for the next few
days to see if we get the error again.
> Do you know any way way to verify the version of tomcat that's running?
> 
> > Subject: RE: Interesting 4.2.1. Issue...
> > From: amin@opencloud.net.au
> > To: dev@cloudstack.apache.org
> > Date: Thu, 3 Apr 2014 10:35:29 +0800
> > 
> > No we didn't, it wouldn't matter because the memory would still fill 
> > up, the problem is it opens a thread and it fails to close it so 
> > whatever you will increase soon or later the memory will fill up (if I 
> > understand right)
> > 
> > The error in catalina is as follows:
> > 
> > SEVERE: The web application [/client] created a ThreadLocal with key of type [java.lang.ThreadLocal]
(value [java.lang.ThreadLocal@1bd66d2d]) and a value of type [com.cloud.api.SerializationContext]
(value [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove it when the web
application was stopped. This is very likely to create a memory leak.
> > 
> > If someone could help with this error generated in the catalina log, that would
be much appreicated.
> > 
> > Kind Regards
> > Amin
> > 
> > 
> > 
> > 
> > -----Original Message-----
> > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > Sent: Thursday, 3 April 2014 9:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: Interesting 4.2.1. Issue...
> > 
> > A few other articles also mention setting the initial heap size "-Xms" to the same
value as the heap size, to go ahead and fully commit the heap. Have you tried that?
> > One other thing I am curious of is have you fiddled with the Perm Size "XX:Permsize"
setting?
> > 
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > From: amin@opencloud.net.au
> > > To: dev@cloudstack.apache.org
> > > Date: Thu, 3 Apr 2014 09:26:31 +0800
> > > 
> > > Believe or not!! We have tried setting them in both formats and 
> > > still the catalina log produces java heap space error
> > > 
> > > Kind Regards
> > > Amin
> > > 
> > > 
> > > 
> > > -----Original Message-----
> > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > Sent: Thursday, 3 April 2014 9:24 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > 
> > > This may be a silly question, but in all of the docs I am reading online in
regards to increasing the java heap size, they are specifying it as -Xmx"size""MB" example
-Xmx2048m vs -Xmx2g Is it possible that it's not reading the 2g variable as 2GB?
> > > 
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > From: amin@opencloud.net.au
> > > > To: dev@cloudstack.apache.org
> > > > Date: Thu, 3 Apr 2014 09:06:17 +0800
> > > > 
> > > > It is 6.0.35 and it still produces this error, even after increasing the
Xmx to 4g, we have installed tomcat 6.0.33 and each time we install the cloudstack it does
not sense the installed 6.0.33 and attempts to install 6.0.35 as it is dependent on it. Silly
solution is that we scheduled a daily restart @ 2PM through a cron job!!!! But first you have
to "killall jvsc" then restart the management server.
> > > > 
> > > > What we are considering is to migrate the management server to CentOS
6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted to restore the dump on a pilot
environment and it worked.
> > > > 
> > > > If someone else has a better solution than this would you please share
it?
> > > > 
> > > > Kind Regards
> > > > Amin
> > > > 
> > > > -----Original Message-----
> > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > Sent: Thursday, 3 April 2014 5:31 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > 
> > > > So I just checked my tomcat version and we are running 6.0.35 which must
be the default that comes with Ubuntu 12.04 out of the box. Our memory settings are as follows:
> > > > JAVA_OPTS="-Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Xmx4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M
-XX:MaxPermSize=800m"
> > > > So what version of tomcat are you running 6.0.35 or 6.0.33?
> > > > 
> > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > From: amin@opencloud.net.au
> > > > > To: dev@cloudstack.apache.org
> > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800
> > > > > 
> > > > > We have the same issue, after an upgrade from 2.2.14 to 4.2.1, 
> > > > > and during this upgrade we had upgrade from Ubuntu 10 LTS to 
> > > > > Ubuntu 12 LTS, it seems it related to tomcat 6.0.35, because it 
> > > > > is recommended to have tomcat 6.0.33 which doesn't come by 
> > > > > default with Ubuntu
> > > > > 12.04.4
> > > > > 
> > > > > Kind Regards
> > > > > Amin    
> > > > > 
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Marcus [mailto:shadowsor@gmail.com]
> > > > > Sent: Tuesday, 1 April 2014 6:22 AM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: Interesting 4.2.1. Issue...
> > > > > 
> > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues 
> > > > > like that. You can send along your memory settings... here's 
> > > > > what I'm
> > > > > running:
> > > > > 
> > > > > JAVA_OPTS="-Djava.awt.headless=true
> > > > > -Dcom.sun.management.jmxremote.port=45219
> > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g 
> > > > > -XX:+HeapDumpOnOutOfMemoryError 
> > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ 
> > > > > -XX:PermSize=512M -XX:MaxPermSize=800m
> > > > > 
> > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips <mphilli7823@hotmail.com>
wrote:
> > > > > > So I have a redundant pair of management servers running on
4.2.1. At least once a day one of the management servers crashes and the log gets filled with
the following messages:
> > > > > > java.lang.OutOfMemoryError: Java heap 
> > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by:
> > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused by:

> > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed 
> > > > > > connection during handshake and there are a few others. When

> > > > > > the one management server tanks, although the other management

> > > > > > server is up and active, it won't actually process any UI 
> > > > > > commands until the crashed server is restarted.  Example of

> > > > > > won't process any UI command is create a new instance, create

> > > > > > a new account, etc. After doing some searching I have found

> > > > > > that others have noticed java heap errors in 4.2.1, and the

> > > > > > suggested fix is to increase the heap size. I am planning on

> > > > > > increasing it from 2g to 4g, however if the problem is 
> > > > > > something like a memory leak, then increasing the heap size

> > > > > > will just delay the inevitable. Has anyone else fixed this 
> > > > > > issue by increasing the heap size? Or what is the recommended

> > > > > > value? **updated...as expected I increased the heap size from

> > > > > > 2g to 4g, and it just too
> > > > >  k longer for the problem to reoccur...
> > > > > > In my honest opinion of bigger concern is the fact that when
one management server crashes the other stops functioning as well. So this begs the question
of why even bother with a redundant pair of servers..Anybody else experience this issue? I
would love to hear any dev guys opinion on this....
> > > > > 
> > > > > 
> > > >  		 	   		  
> > > > 
> > >  		 	   		  
> > > 
> >  		 	   		  
> > 
>  		 	   		  
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message