cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amin Samir" <a...@opencloud.net.au>
Subject RE: Interesting 4.2.1. Issue...
Date Mon, 14 Apr 2014 02:57:06 GMT
Change your browser, try another browser that you never used to log on to any other cloudstack,
or clear the cache or any other history in your browser it will work.

Kind Regards
Amin    




-----Original Message-----
From: Michael Phillips [mailto:mphilli7823@hotmail.com] 
Sent: Monday, 14 April 2014 10:33 AM
To: dev@cloudstack.apache.org
Subject: RE: Interesting 4.2.1. Issue...

Excellent news!
Quick question.
I tried using centos 6.5 with 4.2.1, however after installing I could never log in with the
default username\password combo of admin\password.
Did you experience that?

> Subject: RE: Interesting 4.2.1. Issue...
> From: amin@opencloud.net.au
> To: dev@cloudstack.apache.org
> Date: Mon, 14 Apr 2014 10:08:16 +0800
> 
> Hi Michael,
> 
> For whole three days now we have been using CentOS 6.5 as OS platform for our management
server, and no signs of crashes nor did we need have to restart on a daily basis.
> 
> No signs of java heap space errors neither in the Catalina.out nor in management.log.
> 
> The environment in detail as follows:
> 1) Management server operating system CentOs 6.5 (updated as of 9th of April).
> 2) CloudStack Management version 4.2.1.
> 3) The default Tomcat version that comes with CentOs 6.5 is (6.0.24)
> 3) Primary & Secondary Storage server Ubuntu12.04 LTS (NFS Service)
> 4) Hypervisors Xen Server 6.1 update with the latest patches as of today 14th April.
> 
> If we come through any issues will sure keep you updated.
> 
> Kind Regards
> Amin
> 
> -----Original Message-----
> From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> Sent: Thursday, 10 April 2014 11:10 PM
> To: dev@cloudstack.apache.org
> Subject: RE: Interesting 4.2.1. Issue...
> 
> Amin,
>    Any luck on your project taking your 3 mgmt servers to 4.3 on Centos 6.5?
> 
> > Subject: RE: Interesting 4.2.1. Issue...
> > From: amin@opencloud.net.au
> > To: dev@cloudstack.apache.org
> > Date: Fri, 4 Apr 2014 11:10:28 +0800
> > 
> > Thanks and thanks for sharing the steps
> > 
> > Kind Regards
> > Amin
> > 
> > -----Original Message-----
> > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > Sent: Friday, 4 April 2014 11:02 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: Interesting 4.2.1. Issue...
> > 
> > So I manually downloaded tomcat 6.0.33 
> > herehttps://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+u
> > p+
> > CloudStack+Development+Environment+on+Linux
> > Then did the following1. extracted 6.0.33 to /usr/share/tomcat6.0.33. 
> > 2. Changed symlink of /usr/share/cloudstack-managemet/bin  to 
> > /usr/share/tomcat6.0.33/bin3. Changed symlink of 
> > /usr/share/cloudstack-management/lib to /usr/share/tomcat6.0.33/lib4.
> > Verified cloudstack was running tomcat 6.0.33 by creating a 
> > tomcat_version.jsp file in 
> > /usr/share/cloudstack-management/webapps/client
> > code for tomcat_version.jsp can be found 
> > herehttp://stackoverflow.com/questions/14925073/how-to-find-out-runn
> > in g-tomcat-version I'll definitely let you know how it goes...
> > 
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > From: amin@opencloud.net.au
> > > To: dev@cloudstack.apache.org
> > > Date: Fri, 4 Apr 2014 10:43:35 +0800
> > > 
> > > I tried but I failed to do so, each time cloudstack attempts to install to
go fetches the 6.0.35 from the repo, maybe you have installed it after installing the cloudstack,
if you managed to have a running cloudstack version above the 6.0.33 feedback with the results.
> > > 
> > > Kind Regards
> > > Amin
> > > 
> > > -----Original Message-----
> > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > Sent: Friday, 4 April 2014 10:41 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: Interesting 4.2.1. Issue...
> > > 
> > > So did you try changing your version of tomcat?
> > > 
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > From: amin@opencloud.net.au
> > > > To: dev@cloudstack.apache.org
> > > > Date: Fri, 4 Apr 2014 10:35:42 +0800
> > > > 
> > > > cd /usr/share/tomcat6/bin/
> > > > ./version.sh
> > > > 
> > > > The output should be 6.0.33 instead of 6.0.35
> > > > 
> > > > Using CATALINA_BASE:   /usr/share/tomcat6
> > > > Using CATALINA_HOME:   /usr/share/tomcat6
> > > > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp
> > > > Using JRE_HOME:        /usr
> > > > Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
> > > > Server version: Apache Tomcat/6.0.35
> > > > Server built:   
> > > > Server number:  6.0.35.0
> > > > OS Name:        Linux
> > > > OS Version:     3.11.0-18-generic
> > > > Architecture:   amd64
> > > > JVM Version:    1.6.0_30-b30
> > > > JVM Vendor:     Sun Microsystems Inc.
> > > > 
> > > > 
> > > > Kind Regards
> > > > Amin
> > > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > Sent: Friday, 4 April 2014 10:31 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > 
> > > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for
the next few days to see if we get the error again.
> > > > Do you know any way way to verify the version of tomcat that's running?
> > > > 
> > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > From: amin@opencloud.net.au
> > > > > To: dev@cloudstack.apache.org
> > > > > Date: Thu, 3 Apr 2014 10:35:29 +0800
> > > > > 
> > > > > No we didn't, it wouldn't matter because the memory would 
> > > > > still fill up, the problem is it opens a thread and it fails 
> > > > > to close it so whatever you will increase soon or later the 
> > > > > memory will fill up (if I understand right)
> > > > > 
> > > > > The error in catalina is as follows:
> > > > > 
> > > > > SEVERE: The web application [/client] created a ThreadLocal with
key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) and a value of
type [com.cloud.api.SerializationContext] (value [com.cloud.api.SerializationContext@2f6baed9])
but failed to remove it when the web application was stopped. This is very likely to create
a memory leak.
> > > > > 
> > > > > If someone could help with this error generated in the catalina log,
that would be much appreicated.
> > > > > 
> > > > > Kind Regards
> > > > > Amin
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > > Sent: Thursday, 3 April 2014 9:34 AM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > 
> > > > > A few other articles also mention setting the initial heap size "-Xms"
to the same value as the heap size, to go ahead and fully commit the heap. Have you tried
that?
> > > > > One other thing I am curious of is have you fiddled with the Perm
Size "XX:Permsize" setting?
> > > > > 
> > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > From: amin@opencloud.net.au
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Date: Thu, 3 Apr 2014 09:26:31 +0800
> > > > > > 
> > > > > > Believe or not!! We have tried setting them in both formats

> > > > > > and still the catalina log produces java heap space error
> > > > > > 
> > > > > > Kind Regards
> > > > > > Amin
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > > > Sent: Thursday, 3 April 2014 9:24 AM
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > 
> > > > > > This may be a silly question, but in all of the docs I am reading
online in regards to increasing the java heap size, they are specifying it as -Xmx"size""MB"
example -Xmx2048m vs -Xmx2g Is it possible that it's not reading the 2g variable as 2GB?
> > > > > > 
> > > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > > From: amin@opencloud.net.au
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800
> > > > > > > 
> > > > > > > It is 6.0.35 and it still produces this error, even after
increasing the Xmx to 4g, we have installed tomcat 6.0.33 and each time we install the cloudstack
it does not sense the installed 6.0.33 and attempts to install 6.0.35 as it is dependent on
it. Silly solution is that we scheduled a daily restart @ 2PM through a cron job!!!! But first
you have to "killall jvsc" then restart the management server.
> > > > > > > 
> > > > > > > What we are considering is to migrate the management server
to CentOS 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted to restore the dump
on a pilot environment and it worked.
> > > > > > > 
> > > > > > > If someone else has a better solution than this would you
please share it?
> > > > > > > 
> > > > > > > Kind Regards
> > > > > > > Amin
> > > > > > > 
> > > > > > > -----Original Message-----
> > > > > > > From: Michael Phillips [mailto:mphilli7823@hotmail.com]
> > > > > > > Sent: Thursday, 3 April 2014 5:31 AM
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > > 
> > > > > > > So I just checked my tomcat version and we are running
6.0.35 which must be the default that comes with Ubuntu 12.04 out of the box. Our memory settings
are as follows:
> > > > > > > JAVA_OPTS="-Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Xmx4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M
-XX:MaxPermSize=800m"
> > > > > > > So what version of tomcat are you running 6.0.35 or 6.0.33?
> > > > > > > 
> > > > > > > > Subject: RE: Interesting 4.2.1. Issue...
> > > > > > > > From: amin@opencloud.net.au
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800
> > > > > > > > 
> > > > > > > > We have the same issue, after an upgrade from 2.2.14
to 
> > > > > > > > 4.2.1, and during this upgrade we had upgrade from

> > > > > > > > Ubuntu
> > > > > > > > 10 LTS to Ubuntu 12 LTS, it seems it related to tomcat

> > > > > > > > 6.0.35, because it is recommended to have tomcat 6.0.33

> > > > > > > > which doesn't come by default with Ubuntu
> > > > > > > > 12.04.4
> > > > > > > > 
> > > > > > > > Kind Regards
> > > > > > > > Amin    
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > -----Original Message-----
> > > > > > > > From: Marcus [mailto:shadowsor@gmail.com]
> > > > > > > > Sent: Tuesday, 1 April 2014 6:22 AM
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Subject: Re: Interesting 4.2.1. Issue...
> > > > > > > > 
> > > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen
any 
> > > > > > > > issues like that. You can send along your memory 
> > > > > > > > settings... here's what I'm
> > > > > > > > running:
> > > > > > > > 
> > > > > > > > JAVA_OPTS="-Djava.awt.headless=true
> > > > > > > > -Dcom.sun.management.jmxremote.port=45219
> > > > > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g 
> > > > > > > > -XX:+HeapDumpOnOutOfMemoryError 
> > > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/
> > > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m
> > > > > > > > 
> > > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips
<mphilli7823@hotmail.com> wrote:
> > > > > > > > > So I have a redundant pair of management servers
running on 4.2.1. At least once a day one of the management servers crashes and the log gets
filled with the following messages:
> > > > > > > > > java.lang.OutOfMemoryError: Java heap 
> > > > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused
by:
> > > > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused
by: 
> > > > > > > > > javax.net.ssl.SSLHandshakeException: Remote host

> > > > > > > > > closed connection during handshake and there
are a few others.
> > > > > > > > > When the one management server tanks, although
the 
> > > > > > > > > other management server is up and active, it
won't 
> > > > > > > > > actually process any UI commands until the crashed

> > > > > > > > > server is restarted.  Example of won't process
any UI 
> > > > > > > > > command is create a new instance, create a new

> > > > > > > > > account, etc. After doing some searching I have
found 
> > > > > > > > > that others have noticed java heap errors in
4.2.1, 
> > > > > > > > > and the suggested fix is to increase the heap
size. I 
> > > > > > > > > am planning on increasing it from 2g to 4g, however
if 
> > > > > > > > > the problem is something like a memory leak,
then 
> > > > > > > > > increasing the heap size will just delay the

> > > > > > > > > inevitable. Has anyone else fixed this issue
by increasing the heap size? Or what is the recommended value?
> > > > > > > > > **updated...as expected I increased the heap
size from 
> > > > > > > > > 2g to 4g, and it just too
> > > > > > > >  k longer for the problem to reoccur...
> > > > > > > > > In my honest opinion of bigger concern is the
fact that when one management server crashes the other stops functioning as well. So this
begs the question of why even bother with a redundant pair of servers..Anybody else experience
this issue? I would love to hear any dev guys opinion on this....
> > > > > > > > 
> > > > > > > > 
> > > > > > >  		 	   		  
> > > > > > > 
> > > > > >  		 	   		  
> > > > > > 
> > > > >  		 	   		  
> > > > > 
> > > >  		 	   		  
> > > > 
> > >  		 	   		  
> > > 
> >  		 	   		  
> > 
>  		 	   		  
> 
 		 	   		  


Mime
View raw message