cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Phillips <mphilli7...@hotmail.com>
Subject RE: Interesting 4.2.1. Issue...
Date Fri, 04 Apr 2014 03:13:24 GMT
No problem. CS is at least running on tomcat 6.0.33 so that's good news. :)

> Subject: RE: Interesting 4.2.1. Issue...
> From: amin@opencloud.net.au
> To: dev@cloudstack.apache.org
> Date: Fri, 4 Apr 2014 11:10:28 +0800
> 
> Thanks and thanks for sharing the steps
> 
> Kind Regards
> Amin 
> 
> -----Original Message-----
> From: Michael Phillips [mailto:mphilli7823@hotmail.com] 
> Sent: Friday, 4 April 2014 11:02 AM
> To: dev@cloudstack.apache.org
> Subject: RE: Interesting 4.2.1. Issue...
> 
> So I manually downloaded tomcat 6.0.33 herehttps://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+CloudStack+Development+Environment+on+Linux
> Then did the following1. extracted 6.0.33 to /usr/share/tomcat6.0.33. 2. Changed symlink
of /usr/share/cloudstack-managemet/bin  to /usr/share/tomcat6.0.33/bin3. Changed symlink of
/usr/share/cloudstack-management/lib to /usr/share/tomcat6.0.33/lib4. Verified cloudstack
was running tomcat 6.0.33 by creating a tomcat_version.jsp file in /usr/share/cloudstack-management/webapps/client
> code for tomcat_version.jsp can be found herehttp://stackoverflow.com/questions/14925073/how-to-find-out-running-tomcat-version
> I'll definitely let you know how it goes...
> 
> > Subject: RE: Interesting 4.2.1. Issue...
> > From: amin@opencloud.net.au
> > To: dev@cloudstack.apache.org
> > Date: Fri, 4 Apr 2014 10:43:35 +0800
> > 
> > I tried but I failed to do so, each time cloudstack attempts to install to go fetches
the 6.0.35 from the repo, maybe you have installed it after installing the cloudstack, if
you managed to have a running cloudstack version above the 6.0.33 feedback with the results.
> > 
> > Kind Regards
> > Amin
> > 
> > -----Original Message-----
> > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > Sent: Friday, 4 April 2014 10:41 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: Interesting 4.2.1. Issue...
> > 
> > So did you try changing your version of tomcat?
> > 
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > From: amin@opencloud.net.au
> > > To: dev@cloudstack.apache.org
> > > Date: Fri, 4 Apr 2014 10:35:42 +0800
> > > 
> > > cd /usr/share/tomcat6/bin/
> > > ./version.sh
> > > 
> > > The output should be 6.0.33 instead of 6.0.35
> > > 
> > > Using CATALINA_BASE:   /usr/share/tomcat6
> > > Using CATALINA_HOME:   /usr/share/tomcat6
> > > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp
> > > Using JRE_HOME:        /usr
> > > Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
> > > Server version: Apache Tomcat/6.0.35
> > > Server built:   
> > > Server number:  6.0.35.0
> > > OS Name:        Linux
> > > OS Version:     3.11.0-18-generic
> > > Architecture:   amd64
> > > JVM Version:    1.6.0_30-b30
> > > JVM Vendor:     Sun Microsystems Inc.
> > > 
> > > 
> > > Kind Regards
> > > Amin
> > > 
> > > 
> > > 
> > > -----Original Message-----
> > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > Sent: Friday, 4 April 2014 10:31 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > 
> > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for the
next few days to see if we get the error again.
> > > Do you know any way way to verify the version of tomcat that's running?
> > > 
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > From: amin@opencloud.net.au
> > > > To: dev@cloudstack.apache.org
> > > > Date: Thu, 3 Apr 2014 10:35:29 +0800
> > > > 
> > > > No we didn't, it wouldn't matter because the memory would still 
> > > > fill up, the problem is it opens a thread and it fails to close it 
> > > > so whatever you will increase soon or later the memory will fill 
> > > > up (if I understand right)
> > > > 
> > > > The error in catalina is as follows:
> > > > 
> > > > SEVERE: The web application [/client] created a ThreadLocal with key of
type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) and a value of type
[com.cloud.api.SerializationContext] (value [com.cloud.api.SerializationContext@2f6baed9])
but failed to remove it when the web application was stopped. This is very likely to create
a memory leak.
> > > > 
> > > > If someone could help with this error generated in the catalina log, that
would be much appreicated.
> > > > 
> > > > Kind Regards
> > > > Amin
> > > > 
> > > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > Sent: Thursday, 3 April 2014 9:34 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > 
> > > > A few other articles also mention setting the initial heap size "-Xms"
to the same value as the heap size, to go ahead and fully commit the heap. Have you tried
that?
> > > > One other thing I am curious of is have you fiddled with the Perm Size
"XX:Permsize" setting?
> > > > 
> > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > From: amin@opencloud.net.au
> > > > > To: dev@cloudstack.apache.org
> > > > > Date: Thu, 3 Apr 2014 09:26:31 +0800
> > > > > 
> > > > > Believe or not!! We have tried setting them in both formats and 
> > > > > still the catalina log produces java heap space error
> > > > > 
> > > > > Kind Regards
> > > > > Amin
> > > > > 
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > > Sent: Thursday, 3 April 2014 9:24 AM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > 
> > > > > This may be a silly question, but in all of the docs I am reading
online in regards to increasing the java heap size, they are specifying it as -Xmx"size""MB"
example -Xmx2048m vs -Xmx2g Is it possible that it's not reading the 2g variable as 2GB?
> > > > > 
> > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > From: amin@opencloud.net.au
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800
> > > > > > 
> > > > > > It is 6.0.35 and it still produces this error, even after increasing
the Xmx to 4g, we have installed tomcat 6.0.33 and each time we install the cloudstack it
does not sense the installed 6.0.33 and attempts to install 6.0.35 as it is dependent on it.
Silly solution is that we scheduled a daily restart @ 2PM through a cron job!!!! But first
you have to "killall jvsc" then restart the management server.
> > > > > > 
> > > > > > What we are considering is to migrate the management server
to CentOS 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted to restore the dump
on a pilot environment and it worked.
> > > > > > 
> > > > > > If someone else has a better solution than this would you please
share it?
> > > > > > 
> > > > > > Kind Regards
> > > > > > Amin
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > > > Sent: Thursday, 3 April 2014 5:31 AM
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > 
> > > > > > So I just checked my tomcat version and we are running 6.0.35
which must be the default that comes with Ubuntu 12.04 out of the box. Our memory settings
are as follows:
> > > > > > JAVA_OPTS="-Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Xmx4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M
-XX:MaxPermSize=800m"
> > > > > > So what version of tomcat are you running 6.0.35 or 6.0.33?
> > > > > > 
> > > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > > From: amin@opencloud.net.au
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800
> > > > > > > 
> > > > > > > We have the same issue, after an upgrade from 2.2.14 to

> > > > > > > 4.2.1, and during this upgrade we had upgrade from Ubuntu
10 
> > > > > > > LTS to Ubuntu 12 LTS, it seems it related to tomcat 6.0.35,

> > > > > > > because it is recommended to have tomcat 6.0.33 which 
> > > > > > > doesn't come by default with Ubuntu
> > > > > > > 12.04.4
> > > > > > > 
> > > > > > > Kind Regards
> > > > > > > Amin    
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > -----Original Message-----
> > > > > > > From: Marcus [mailto:shadowsor@gmail.com]
> > > > > > > Sent: Tuesday, 1 April 2014 6:22 AM
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: Re: Interesting 4.2.1. Issue...
> > > > > > > 
> > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues

> > > > > > > like that. You can send along your memory settings... here's

> > > > > > > what I'm
> > > > > > > running:
> > > > > > > 
> > > > > > > JAVA_OPTS="-Djava.awt.headless=true
> > > > > > > -Dcom.sun.management.jmxremote.port=45219
> > > > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g 
> > > > > > > -XX:+HeapDumpOnOutOfMemoryError 
> > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/
> > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m
> > > > > > > 
> > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips <mphilli7823@hotmail.com>
wrote:
> > > > > > > > So I have a redundant pair of management servers running
on 4.2.1. At least once a day one of the management servers crashes and the log gets filled
with the following messages:
> > > > > > > > java.lang.OutOfMemoryError: Java heap 
> > > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused
by:
> > > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused
by: 
> > > > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed

> > > > > > > > connection during handshake and there are a few others.

> > > > > > > > When the one management server tanks, although the
other 
> > > > > > > > management server is up and active, it won't actually

> > > > > > > > process any UI commands until the crashed server is

> > > > > > > > restarted.  Example of won't process any UI command
is 
> > > > > > > > create a new instance, create a new account, etc.
After 
> > > > > > > > doing some searching I have found that others have
noticed 
> > > > > > > > java heap errors in 4.2.1, and the suggested fix is
to 
> > > > > > > > increase the heap size. I am planning on increasing
it 
> > > > > > > > from 2g to 4g, however if the problem is something
like a 
> > > > > > > > memory leak, then increasing the heap size will just
delay 
> > > > > > > > the inevitable. Has anyone else fixed this issue by

> > > > > > > > increasing the heap size? Or what is the recommended
value?
> > > > > > > > **updated...as expected I increased the heap size
from 2g 
> > > > > > > > to 4g, and it just too
> > > > > > >  k longer for the problem to reoccur...
> > > > > > > > In my honest opinion of bigger concern is the fact
that when one management server crashes the other stops functioning as well. So this begs
the question of why even bother with a redundant pair of servers..Anybody else experience
this issue? I would love to hear any dev guys opinion on this....
> > > > > > > 
> > > > > > > 
> > > > > >  		 	   		  
> > > > > > 
> > > > >  		 	   		  
> > > > > 
> > > >  		 	   		  
> > > > 
> > >  		 	   		  
> > > 
> >  		 	   		  
> > 
>  		 	   		  
> 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message