Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <47A7D07F.2070701@hanik.com>
Date: Mon, 04 Feb 2008 19:57:03 -0700
From: Filip Hanik - Dev Lists <devlists@hanik.com>
User-Agent: Thunderbird 2.0.0.9 (Windows/20071031)
MIME-Version: 1.0
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
References: <00c801c86788$1955dd90$072613ac@jomax.paholdings.com>
 <47A7AC36.1080708@hanik.com>
 <00df01c86797$1c0ff6b0$072613ac@jomax.paholdings.com>
In-Reply-To: <00df01c86797$1c0ff6b0$072613ac@jomax.paholdings.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

ok, so there is a leak in the Java Heap, one way to figure out what is 
going would be to add
-XX:+*HeapDumpOnOutOfMemoryError
to the command line options.
This will dump the heap to a file that you can analyze. You need some 
serious RAM on a box to analyze a thread dump like that, but it will 
tell you everything.
I use www.yourkit.com when I analyze heapdumps

Filip
*
Dan D. wrote:
> Following is the exception that is thrown, along with a few 
> surrounding lines from the catalina.out log file:
>
> [Full GC 1982271K->1982271K(1999424K), 7.5088700 secs]
> [Full GC 1982271K->1982269K(1999424K), 7.5053160 secs]
> java.lang.reflect.InvocationTargetException
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [Full GC 1982271K->1982271K(1999424K), 7.4678950 secs]
> [Full GC 1982271K->1982246K(1999424K), 9.5055930 secs]
>        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
>
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOf(Arrays.java:2882)
>        at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) 
>
>        at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>        at java.lang.StringBuffer.append(StringBuffer.java:224)
>        at java.io.StringWriter.write(StringWriter.java:95)
>        at java.io.PrintWriter.write(PrintWriter.java:412)
>        at java.io.PrintWriter.write(PrintWriter.java:429)
>        at java.io.PrintWriter.print(PrintWriter.java:559)
>        at java.io.PrintWriter.println(PrintWriter.java:695)
>        at java.lang.Throwable.printStackTrace(Throwable.java:512)
>        at 
> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
>        at java.util.logging.StreamHandler.publish(StreamHandler.java:179)
>        at 
> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
>        at java.util.logging.Logger.log(Logger.java:472)
>        at java.util.logging.Logger.doLog(Logger.java:494)
>        at java.util.logging.Logger.logp(Logger.java:694)
> [Full GC 1982271K->1982271K(1999424K), 7.5629480 secs]
> [Full GC 1982271K->1982247K(1999424K), 7.4791920 secs]
>        at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
>        at 
> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
>        at 
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:929) 
>
>        at 
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883) 
>
>        at 
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
>        at 
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138)
>        at 
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) 
>
>        at 
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) 
>
>        at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>        at 
> org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
>        at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
>        at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
>        at 
> org.apache.catalina.core.StandardService.start(StandardService.java:516)
>        at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
>        at org.apache.catalina.startup.Catalina.start(Catalina.java:566)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [Full GC 1982271K->1982271K(1999424K), 7.4982800 secs]
> [Full GC 1982271K->1982271K(1999424K), 7.4976100 secs]
> [Full GC 1982271K->1982244K(1999424K), 9.5026360 secs]
> [Full GC 1982272K->1982244K(1999424K), 7.4494750 secs]
> [Full GC 1982272K->1982272K(1999424K), 7.4763190 secs]
> [Full GC 1982272K->1982234K(1999424K), 9.3931880 secs]
> [Full GC[Unloading class 
> com.atlassian.confluence.event.events.plugin.PluginDisableEvent]
> [Unloading class org.apache.log4j.spi.RepositorySelector]
> [Unloading class org.apache.xerces.xni.grammars.Grammar]
> [Unloading class org.apache.xerces.dom.DocumentTypeImpl]
> [Unloading class 
> org.springframework.web.util.IntrospectorCleanupListener]
> [Unloading class org.apache.xerces.impl.io.UTF8Reader]
> [Unloading class org.apache.xerces.impl.dtd.XMLDTDDescription]
>
> <snip> (many more classes are unloaded here)
>
> [Unloading class org.springframework.web.context.ContextLoaderListener]
> [Unloading class com.atlassian.core.util.XMLUtils]
> [Unloading class 
> org.springframework.beans.factory.support.ReplaceOverride]
> 1982271K->1982271K(1999424K), 7.4622510 secs]
> [Full GC 1982271K->1981801K(1999424K), 8.9731420 secs]
>
> And the Full GC's continue for a little while, and eventually the 
> process just dies off.
>
>
> ----- Original Message ----- From: "Filip Hanik - Dev Lists" 
> <devlists@hanik.com>
> To: "Tomcat Users List" <users@tomcat.apache.org>
> Sent: Monday, February 04, 2008 5:22 PM
> Subject: Re: Out of Memory Errors when upgrading from 5.0.27 to 6.0.14
>
>
>> what's the message after
>>
>> java.lang.OutOfMemoryError
>>
>> as there can be many different types of errors.
>>
>> Filip
>>
>> Dan D. wrote:
>>> Hello all,
>>>
>>> I'm currently working on upgrading our systems from using Tomcat 
>>> 5.0.27 to 6.0.14.
>>>
>>> The upgrades are not being done in place (ie, we will have both 
>>> systems out there, and will be doing migrations to the new 
>>> version).  I have been able to resolve most of the issues we have 
>>> run into thus far, however, I am now somewhat stuck, and am hoping 
>>> someone here may be able to help.
>>>
>>> About the environment:
>>> Shared hosting, with approximately 300-600 users on a server, with 
>>> an average of ~1500 applications (many users have a few 
>>> applications).  Not all users are utilizing any java at all, 
>>> however.  Apache 1.3.33 is running and configured to serve static 
>>> content.
>>>
>>> The older servers with TC5 on them have ~6GB of available RAM, new 
>>> servers for TC6 have 8GB of RAM.
>>> Customer content is stored on a remote NAS server.
>>>
>>> While I understand that having the <Host> and <Context> declaration 
>>> for all users listed in server.xml, that is how we have it currently 
>>> configured.
>>> About the problem:
>>> When tomcat is started and running on our existing 5.0.27 server, it 
>>> uses approximately 1GB of RAM, and successfully runs all 
>>> applications without issue.
>>>
>>> When starting tomcat on the 6.0.14 server with the _exact_ same 
>>> content, however, it runs out of heap space.  This is the issue I'm 
>>> running into right now that I can't seem to find any answers about.  
>>> Attempting to increase the available heap results in an error 
>>> allocating memory upon startup.
>>>
>>>
>>> The output from jmap seems to show that the majority of the 
>>> heap/memory is being consumed by character arrays.
>>>
>>> When viewing the {tomcat.home}/work/{engine}/{application} 
>>> directory, it appears there is a fair difference on how these are 
>>> loaded into tomcat.
>>>
>>> On Tomcat5, this directory consists of only directories that contain 
>>> a WEB-INF directory in them, and each of those have a tldCache.ser 
>>> file.
>>>
>>> On Tomcat6, this directory consists of all directories located 
>>> inside a directory that contains a WEB-INF directory.
>>>
>>> For example:
>>> Tomcat 6:
>>> [root@servername ]# pwd ; find .
>>> /web/tomcat/work/hosting/sitename.com
>>> .
>>> ./_
>>> ./cgi
>>> ./stats
>>> ./php_uploads
>>> ./images
>>> ./ad
>>> ./Video
>>> ./Templates
>>> ./Flv
>>> ./Songs
>>> ./newwebsite
>>> ./old files
>>> ./manager
>>>
>>> Tomcat5:
>>> [root@servername ]# pwd; find .
>>> /web/tomcat/work/hosting/sitename.com
>>> .
>>> ./_
>>> ./_/tldCache.ser
>>>
>>> And the contents of the actual home directory are:
>>> [root@servername html]# find . -maxdepth 1 -type d
>>> .
>>> ./cgi
>>> ./stats
>>> ./php_uploads
>>> ./WEB-INF
>>> ./images
>>> ./ad
>>> ./Video
>>> ./Templates
>>> ./Flv
>>> ./Songs
>>> ./newwebsite
>>> ./old files
>>>
>>> I have read http://wiki.apache.org/tomcat/FAQ/Memory and attempted 
>>> all the suggestions there, to no avail.
>>>
>>> Thus, it appears that on Tomcat6, it is attempting to load all of 
>>> these paths into memory, and eventually runs out.  Tomcat5 did not 
>>> load all of the paths to directories without applications, and thus 
>>> uses less of the heap.
>>>
>>> Does it seem like I am on the right track here?  Have others 
>>> encountered this in the past?  Is there a way to prevent this 
>>> behavior within Tomcat 6?
>>>
>>> I thank you for any insight you can provide on this issue.
>>>
>>> Sincerely,
>>> Dan D.
>>>
>>>   
>>> ------------------------------------------------------------------------ 
>>>
>>>
>>> No virus found in this incoming message.
>>> Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 
>>> 269.19.19/1257 - Release Date: 2/3/2008 5:49 PM
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org