tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Konjarla <srikanth.konja...@gmail.com>
Subject Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
Date Wed, 15 Dec 2010 02:40:58 GMT
I could catch Axis threadlocals from the app to clean up. However, I
have a question wrt following tomcat's log at the time of undeploy of
app. The message suggests that it is removing the same threadlocal
twice. Is it because it was not removed in the first attempt?

---------------------------------------------------------------------
ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@14213040]) and a
value of type [org.apache.catalina.loader.WebappClassLoader] (value
[WebappClassLoader
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader@6ee3849c
]) but failed to remove it when the web application was stopped. To
prevent a memory leak, the ThreadLocal has been forcibly removed.
Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@14213040]) and a
value of type [org.apache.catalina.loader.WebappClassLoader] (value
[WebappClassLoader
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader@6ee3849c
]) but failed to remove it when the web application was stopped. To
prevent a memory leak, the ThreadLocal has been forcibly removed.
--------------------------------------------------------------------

On 12/13/10 4:23 AM, Pid wrote:
> On 12/12/2010 18:18, Srikanth Konjarla wrote:
>>
>>
>> On 12/12/10 8:28 AM, Pid * wrote:
>>> On 11 Dec 2010, at 21:39, Srikanth Konjarla <srikanth.konjarla@gmail.com>
wrote:
>>>
>>>>
>>>> On Dec 11, 2010, at 1:04 PM, "Pid *" <pid@pidster.com> wrote:
>>>>
>>>>> On 11 Dec 2010, at 20:02, Srikanth Konjarla <srikanth.konjarla@gmail.com>
wrote:
>>>>>
>>>>>> Pid,
>>>>>>
>>>>>> Thanks for your patience. Here is the output from catalina.out file
>>>>>> while the webapp is being undeployed. As you can see there are few
>>>>>> threadLocals that are cleaned up.
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader
>>>>>> clearThreadLocalMap
>>>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7ee75db3])
and a
>>>>>> value of type [org.apache.catalina.loader.WebappClassLoader] (value
>>>>>> [WebappClassLoader
>>>>>> delegate: false
>>>>>> repositories:
>>>>>>  /WEB-INF/classes/
>>>>>> ----------> Parent Classloader:
>>>>>> org.apache.catalina.loader.StandardClassLoader@1eb3319f
>>>>>> ]) but failed to remove it when the web application was stopped.
To
>>>>>> prevent a memory leak, the ThreadLocal has been forcibly removed.
>>>>>
>>>>> The above means that something is storing the WebappClassLoader in a
>>>>> ThreadLocal. Is your app doing this?
>>>>>
>>>>> The two below I know about and I believe are defects in Axis 1.4.
>>>> The only component that uses threadlocals in my app is Axis and I believe
it is not cleaning up after completely.
>>>
>>> I agree.
>>>
>>>> Originally, I have started on tracking down Axis threads that are responsible.
In this case, Axis is performing webservice client operations.
>>>
>>> Then you are also seeing an additional problem - namely that the
>>> classloader itself is being stored in a threadlocal. I haven't seen
>>> Axis do that so far.
>> You are correct. I think it is Spring framework (also used in the
>> application) that is storing classloader in threadlocal. 
> 
> Which version of Spring Framework are you using and how do you arrive at
> that conclusion?
> 
> Are you using any other libraries and if so, which exact versions are
> you using?
> 
> 
>> BTW, do you
>> have any suggestions on where/how to track Axis threads so that I can
>> clean them up from my app?
> 
> I'm still looking into it - but I'll post here when I've got an answer.
> 
> 
> p
> 
>> Thanks
>>
>> Srikanth
>>
>>>
>>> You may need to use a newer version of Axis - I don't think v1 is
>>> expected to do any more releases.
>>>
>>>
>>> p
>>>
>>>
>>>> Srikanth
>>>>>
>>>>>
>>>>> p
>>>>>
>>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader
>>>>>> clearThreadLocalMap
>>>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7ee75db3])
and a
>>>>>> value of type [org.apache.catalina.loader.WebappClassLoader] (value
>>>>>> [WebappClassLoader
>>>>>> delegate: false
>>>>>> repositories:
>>>>>>  /WEB-INF/classes/
>>>>>> ----------> Parent Classloader:
>>>>>> org.apache.catalina.loader.StandardClassLoader@1eb3319f
>>>>>> ]) but failed to remove it when the web application was stopped.
To
>>>>>> prevent a memory leak, the ThreadLocal has been forcibly removed.
>>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader
>>>>>> clearThreadLocalMap
>>>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>>>> [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value
>>>>>> [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1@775d1479])
>>>>>> and a value of type [byte[]] (value [[B@5faeed4]) but failed to remove
>>>>>> it when the web application was stopped. To prevent a memory leak,
the
>>>>>> ThreadLocal has been forcibly removed.
>>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader
>>>>>> clearThreadLocalMap
>>>>>> SEVERE: A web application created a ThreadLocal with key of type
>>>>>> [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value
>>>>>> [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@3a0ee334])
>>>>>> and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl]
(value
>>>>>> [org.apache.xerces.jaxp.DocumentBuilderImpl@61583db6]) but failed
to
>>>>>> remove it when the web application was stopped. To prevent a memory
>>>>>> leak, the ThreadLocal has been forcibly removed.
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> Srikanth
>>>>>>
>>>>>> On 12/11/10 2:26 AM, Pid wrote:
>>>>>>> On 12/11/10 9:25 AM, Srikanth Konjarla wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Dec 11, 2010, at 1:18 AM, "Pid *" <pid@pidster.com>
wrote:
>>>>>>>>
>>>>>>>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla <srikanth.konjarla@gmail.com>
wrote:
>>>>>>>>>
>>>>>>>>>> BTW, I see some similarities with the following.
>>>>>>>>>>
>>>>>>>>>> https://jira.springframework.org/browse/SWS-533
>>>>>>>>>
>>>>>>>>> Similarities in which sense?
>>>>>>>>> The issue was closed as invalid.
>>>>>>>> The similarity is that the thread is retained.
>>>>>>>
>>>>>>> This is not a problem.  The threads exist in a pool and are reused.
>>>>>>>
>>>>>>> Chuck explained why the WAIT status isn't a problem.
>>>>>>>
>>>>>>> However, other than the case is invalid it does not provide detail
why
>>>>>>> it is invalid and why the thread is in WAIT state.
>>>>>>>
>>>>>>> Yes, it does.  It also refers to JAXB, not Axis.  So it's unrelated
to
>>>>>>> your problem.
>>>>>>>
>>>>>>> If the thread is reused by tomcat, what happened to the references
from
>>>>>>> previous cycle.
>>>>>>>
>>>>>>> Which references?  The ThreadLocals?
>>>>>>>
>>>>>>> If a ThreadLocal is created by an application, but not cleaned
up, the
>>>>>>> thread still contains the reference in the internal map of ThreadLocals.
>>>>>>> If the reference is to a class from a webapp which is subsequently
>>>>>>> unloaded, then the classloader will not be garbage collected.
>>>>>>>
>>>>>>>
>>>>>>> I would  be interested in learning.
>>>>>>>
>>>>>>> PLEASE post the leak warnings from your logs.
>>>>>>>
>>>>>>>
>>>>>>> p
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> You're using Axis 1.4 which I've been looking at recently,
because I
>>>>>>>>> believe it causes a leak.
>>>>>>>> I believe the same but wanted to make sure that leaking threads
are captured and cleaned during the undeploy process. Additionally, I was wondering how I
can detect and locate runaway Axis threads with thread locals.
>>>>>>>>>
>>>>>>>>> Please post the warnings from your logs so I can see
of it's the same problem.
>>>>>>>> Will do.
>>>>>>>>
>>>>>>>> Srikanth
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> p
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Srikanth
>>>>>>>>>>
>>>>>>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote:
>>>>>>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konjarla@gmail.com]
>>>>>>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State:
WAITING (on object monitor)
>>>>>>>>>>>
>>>>>>>>>>>> as part of the thread local cleanup process
at the time of undeploy,
>>>>>>>>>>>> tomcat removes few threadLocals but misses
the threadLocals for the
>>>>>>>>>>>> thread in question. I am wondering about
the tomcat thread still being
>>>>>>>>>>>> in "WAIT" mode and been cleaned up.
>>>>>>>>>>>
>>>>>>>>>>> The WAIT mode is normal - the thread is sitting
in the pool, waiting for a request to show up.  I believe the ThreadLocal objects are discarded
by letting such infected threads exit and creating new ones.  That's an area still undergoing
refinement, so you might want to try moving up to 6.0.29 and see if it makes any difference.
 Or try sending in enough concurrent requests to get all the threads busy and see if the references
disappear after that.
>>>>>>>>>>>
>>>>>>>>>>> - Chuck
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR
OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail and its attachments
from all computers.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message