Return-Path: Delivered-To: apmail-tomcat-users-archive@www.apache.org Received: (qmail 72460 invoked from network); 12 Dec 2010 16:27:44 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 12 Dec 2010 16:27:44 -0000 Received: (qmail 49482 invoked by uid 500); 12 Dec 2010 16:27:40 -0000 Delivered-To: apmail-tomcat-users-archive@tomcat.apache.org Received: (qmail 49348 invoked by uid 500); 12 Dec 2010 16:27:40 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 49339 invoked by uid 99); 12 Dec 2010 16:27:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Dec 2010 16:27:39 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of pid@pidster.com designates 74.125.82.173 as permitted sender) Received: from [74.125.82.173] (HELO mail-wy0-f173.google.com) (74.125.82.173) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Dec 2010 16:27:34 +0000 Received: by wyg36 with SMTP id 36so5957381wyg.18 for ; Sun, 12 Dec 2010 08:27:13 -0800 (PST) Received: by 10.227.145.134 with SMTP id d6mr1079309wbv.81.1292171233048; Sun, 12 Dec 2010 08:27:13 -0800 (PST) References: <4D029948.6000701@gmail.com> <99C8B2929B39C24493377AC7A121E21F9A09C39050@USEA-EXCH8.na.uis.unisys.com> <4D02ADD2.2040002@gmail.com> <99C8B2929B39C24493377AC7A121E21F9A09C390BF@USEA-EXCH8.na.uis.unisys.com> <4D02CEC3.8040602@gmail.com> <5965382455244512346@unknownmsgid> <4D0351B9.8060108@pidster.com> <4D03D8B7.7020600@gmail.com> <-7208931519977724218@unknownmsgid> From: "Pid *" In-Reply-To: Mime-Version: 1.0 (iPhone Mail 8B117) Date: Sun, 12 Dec 2010 16:28:06 +0000 Message-ID: <-2379727414238134208@unknownmsgid> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) To: Tomcat Users List Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 11 Dec 2010, at 21:39, Srikanth Konjarla w= rote: > > On Dec 11, 2010, at 1:04 PM, "Pid *" wrote: > >> On 11 Dec 2010, at 20:02, Srikanth Konjarla wrote: >> >>> Pid, >>> >>> Thanks for your patience. Here is the output from catalina.out file >>> while the webapp is being undeployed. As you can see there are few >>> threadLocals that are cleaned up. >>> >>> ---------------------------------------------------------------------- >>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7ee75db3]) and a >>> value of type [org.apache.catalina.loader.WebappClassLoader] (value >>> [WebappClassLoader >>> delegate: false >>> repositories: >>> /WEB-INF/classes/ >>> ----------> Parent Classloader: >>> org.apache.catalina.loader.StandardClassLoader@1eb3319f >>> ]) but failed to remove it when the web application was stopped. To >>> prevent a memory leak, the ThreadLocal has been forcibly removed. >> >> The above means that something is storing the WebappClassLoader in a >> ThreadLocal. Is your app doing this? >> >> The two below I know about and I believe are defects in Axis 1.4. > The only component that uses threadlocals in my app is Axis and I believe= it is not cleaning up after completely. I agree. > Originally, I have started on tracking down Axis threads that are respons= ible. In this case, Axis is performing webservice client operations. Then you are also seeing an additional problem - namely that the classloader itself is being stored in a threadlocal. I haven't seen Axis do that so far. You may need to use a newer version of Axis - I don't think v1 is expected to do any more releases. p > Srikanth >> >> >> p >> >>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7ee75db3]) and a >>> value of type [org.apache.catalina.loader.WebappClassLoader] (value >>> [WebappClassLoader >>> delegate: false >>> repositories: >>> /WEB-INF/classes/ >>> ----------> Parent Classloader: >>> org.apache.catalina.loader.StandardClassLoader@1eb3319f >>> ]) but failed to remove it when the web application was stopped. To >>> prevent a memory leak, the ThreadLocal has been forcibly removed. >>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value >>> [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1@775d1479]) >>> and a value of type [byte[]] (value [[B@5faeed4]) but failed to remove >>> it when the web application was stopped. To prevent a memory leak, the >>> ThreadLocal has been forcibly removed. >>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value >>> [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@3a0ee334]) >>> and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value >>> [org.apache.xerces.jaxp.DocumentBuilderImpl@61583db6]) but failed to >>> remove it when the web application was stopped. To prevent a memory >>> leak, the ThreadLocal has been forcibly removed. >>> -----------------------------------------------------------------------= - >>> >>> Srikanth >>> >>> On 12/11/10 2:26 AM, Pid wrote: >>>> On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >>>>> >>>>> >>>>> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >>>>> >>>>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla wrote: >>>>>> >>>>>>> BTW, I see some similarities with the following. >>>>>>> >>>>>>> https://jira.springframework.org/browse/SWS-533 >>>>>> >>>>>> Similarities in which sense? >>>>>> The issue was closed as invalid. >>>>> The similarity is that the thread is retained. >>>> >>>> This is not a problem. The threads exist in a pool and are reused. >>>> >>>> Chuck explained why the WAIT status isn't a problem. >>>> >>>> However, other than the case is invalid it does not provide detail why >>>> it is invalid and why the thread is in WAIT state. >>>> >>>> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >>>> your problem. >>>> >>>> If the thread is reused by tomcat, what happened to the references fro= m >>>> previous cycle. >>>> >>>> Which references? The ThreadLocals? >>>> >>>> If a ThreadLocal is created by an application, but not cleaned up, the >>>> thread still contains the reference in the internal map of ThreadLocal= s. >>>> If the reference is to a class from a webapp which is subsequently >>>> unloaded, then the classloader will not be garbage collected. >>>> >>>> >>>> I would be interested in learning. >>>> >>>> PLEASE post the leak warnings from your logs. >>>> >>>> >>>> p >>>> >>>> >>>> >>>>>> You're using Axis 1.4 which I've been looking at recently, because I >>>>>> believe it causes a leak. >>>>> I believe the same but wanted to make sure that leaking threads are c= aptured and cleaned during the undeploy process. Additionally, I was wonder= ing how I can detect and locate runaway Axis threads with thread locals. >>>>>> >>>>>> Please post the warnings from your logs so I can see of it's the sam= e problem. >>>>> Will do. >>>>> >>>>> Srikanth >>>>>> >>>>>> >>>>>> p >>>>>> >>>>>>> >>>>>>> Srikanth >>>>>>> >>>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konjarla@gmail.com] >>>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object mo= nitor) >>>>>>>> >>>>>>>>> as part of the thread local cleanup process at the time of undepl= oy, >>>>>>>>> tomcat removes few threadLocals but misses the threadLocals for t= he >>>>>>>>> thread in question. I am wondering about the tomcat thread still = being >>>>>>>>> in "WAIT" mode and been cleaned up. >>>>>>>> >>>>>>>> The WAIT mode is normal - the thread is sitting in the pool, waiti= ng for a request to show up. I believe the ThreadLocal objects are discard= ed by letting such infected threads exit and creating new ones. That's an = area still undergoing refinement, so you might want to try moving up to 6.0= .29 and see if it makes any difference. Or try sending in enough concurren= t requests to get all the threads busy and see if the references disappear = after that. >>>>>>>> >>>>>>>> - Chuck >>>>>>>> >>>>>>>> >>>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPR= IETARY MATERIAL and is thus for use only by the intended recipient. If you = received this in error, please contact the sender and delete the e-mail and= its attachments from all computers. >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------= --- >>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------= -- >>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------= - >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>> For additional commands, e-mail: users-help@tomcat.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >> For additional commands, e-mail: users-help@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org > For additional commands, e-mail: users-help@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org