tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool
Date Tue, 07 May 2013 22:06:01 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Michael,

On 5/7/13 4:33 PM, Michael-O wrote:
> Am 2013-05-07 17:20, schrieb Christopher Schultz:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>> 
>> Michael,
>> 
>> On 5/7/13 6:43 AM, Michael-O wrote:
>>>> Von: "Mark Thomas" <markt@apache.org> On 07/05/2013 10:25, 
>>>> Michael-O wrote:
>>>>> 
>>>>>> Von: "Mark Thomas" <markt@apache.org> On 07/05/2013
>>>>>> 09:16, Michael-O wrote:
>>>>>>> Hi folks,
>>>>>>> 
>>>>>>> I recently enabled a QueryTimeoutInterceptor with 
>>>>>>> queryTimeout of 60 seconds in a JDBC Pool data source 
>>>>>>> (7.0.37). When the app was shut down, Tomcat said: "The
>>>>>>> web application [/...] appears to have started a thread
>>>>>>> named [OracleTimeoutPollingThread] but has failed to
>>>>>>> stop it..."
>>>>>>> 
>>>>>>> We are using Oracle 11.2g with 11.2.0.3 JDBC drivers.
>>>>>>> I have figured out that this thread is spawned by the
>>>>>>> driver itself. According to this Stackoverflow answer
>>>>>>> [1] this is a long-living thread, same says the JDBC
>>>>>>> FAQ [2] of Oracle.
>>>>>>> 
>>>>>>> The thread seems to work like Pool's PoolCleaner
>>>>>>> thread. A few month a ago I reported the same issue
>>>>>>> with the PoolCleaner thread and Filip fixed the class
>>>>>>> loader orders.
>>>>>>> 
>>>>>>> Can this be a false-positive by the memory leak
>>>>>>> detector since this thread lives only once in the
>>>>>>> entire VM?
>>>>>> 
>>>>>> No. It is a memory leak and either or bug in your
>>>>>> application or in the JDBC driver.
>>>>>> 
>>>>>> Where is the Oracle JDBC driver? CATALINA_[BASE|HOME]/lib
>>>>>> or WEB-INF/lib
>>>>> 
>>>>> The driver is in the $CATALINA_HOME/lib only where 
>>>>> $CATALINA_BASE != $CATALINA_HOME. This was done for a
>>>>> single webapp for testing purposes.
>>>>> 
>>>>> Does this make a difference?
>>>> 
>>>> The important thing is that it isn't in WEB-INF/lib.
>>>> 
>>>>> How do you know that this is not a false-positive?
>>>> 
>>>> Experience, a lot of research into memory leaks and I wrote 
>>>> Tomcat's memory leak detection code.
>>>> 
>>>>> If you really know for sure, I can open a service request
>>>>> with Oracle Support.
>>>> 
>>>> Good luck with that.
>>>> 
>>>> The problem is that when the driver creates the thread it
>>>> does so when the current class loader is the web application
>>>> class loader. That means that the Thread will be created with
>>>> a context class loader set to the web application class
>>>> loader. That will trigger a memory leak when the web
>>>> application is stopped because a reference is retained to the
>>>> web application's class loader.
>>>> 
>>>> What the driver should do is, after it creates the thread,
>>>> set the thread's context class loader to the class loader
>>>> that loaded the driver.
>>>> 
>>>> What you are seeing is a variation of the leak described on
>>>> page 15 of [1].
>>> 
>>> After reading the slides and your explanation this makes
>>> sense. It's the same issue as "Pool cleaner thread should be
>>> created using the classloader that loaded the pool, not the
>>> context loader (fhanik)" fixed in 7.0.27.
>>> 
>>> I will file SR and let you know.
>> 
>> Note that you might be able to write your own code to mitigate
>> this problem, depending on exactly when that thread is created.
>> If the timeout thread isn't created until you actually try to
>> issue a query with a timeout, try something like this in a
>> ServletContextListener's contextInitialized method:
>> 
>> // NOTE: No resource management is being done in this example
>> 
>> // Get the current ClassLoader -- should be WebappClassLoader 
>> ClassLoader cl = Thread.currentThread().getContextClassLoader();
>> 
>> // WebappClassLoader.getParent should be "common" loader 
>> Thread.currentThread().setContextClassLoader(cl.getParent());
>> 
>> try { Connection conn = ...; // However you get a connection 
>> Statement s = conn.createStatement(); s.setQueryTimeout(5000); //
>> Doesn't really matter what TO is s.execute("SELECT 1 FROM dual",
>> ); } .... finally { // Pop back to the original ClassLoader 
>> Thread.currentThread().setContextClassLoader(cl); }
>> 
>> This is the exact technique that Tomcat's 
>> JreMemoryLeakPreventionListener uses to prevent ClassLoader
>> leaks.
> 
> Yes, this looks like the way JDBC Pool does with the Pool Cleaner.
> I would go for this as a last resort.
> 
>> A couple of notes:
>> 
>> 1. This won't work under a SecurityManager. If you need to
>> operate under a SecurityManager, have a look at the 
>> JreMemoryLeakPreventionListener code and adapt it to the above.
>> 
>> 2. If the Oracle driver launches the thread when the DataSource
>> is created, it might happen too early for a
>> ServletContextListener to intervene. In that case, simply modify
>> the JreMEmoryLeakPreventionListener code directly. Patches are
>> always welcome.
> 
> Not, it is not. I have attached VisualVM to Tomcat VM and have seen
> that the thread is forked when the first query is executed.

That's good news: it means that if you need it fixed *right away*, you
don't need to hack-and-recompile Tomcat: you can just do it in your
own ServletContextListener.

>> 3. If Oracle fixes this bug, Tomcat should not prevent against
>> it. If Oracle refuses to acknowledge/fix/etc. this bug, then it
>> may make sense to include such code in
>> JreMemoryLeakPreventionListener, but it should probably be done
>> in such a way that a) it's not enabled by default and b) the
>> query used for triggering the Thread to be created is
>> user-selectable with maybe a reasonable default (like "SELECT 1 
>> FROM dual", as that tends to be valid in most RDBMSs).
> 
> I am already in contact with an Oracle engineer who has received a
> demo WAR file to reproduce this issue. If Oracle won't, maybe some
> generic approach would be advisable but in the in 
> JreMemoryLeakPreventionListener but rather in Tomcat JDBC Pool
> IMHO.

I think this might be a rather esoteric thing to exercise, but one
could certainly add it as an interceptor that is not enabled by default.

> I do not know how other RDBMS vendors implement the timeout
> function.

MySQL has taken some time to fix similar/related bugs, but is
(eventually) willing to fix them:

http://bugs.mysql.com/bug.php?id=65909
http://bugs.mysql.com/bug.php?id=36565

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRiXrJAAoJEBzwKT+lPKRYOqIP/1VeVgEvA82+uAn6x9kSKoQK
HN5zOIUwUG/Mr5zWJjR7XeZoOSKaiYHb6xZOjM9zG7hROrHdfrvoj6YLO+GNIe73
wPw9Tx9HGww6NskBYReG/TtS1DKLODsxEd5xkngckDH+xrTr3P0lvhSSpg5pS5c6
gmMqw/PMP6fjvN2JeOtETShxzVYXVX3fh9tYLATPiJuZnMT1v3p0Jo7AZHX7ucRr
LnoJm/+VbszQkXq0Bve+FRyE7y4yXgc7J90/FOiJ/B5DJ7a4LQB8jIp88DGFBEKv
+s2Yrlf2R39mAtGA1Vj6KEVxvaEW9zIR0wPDuriR6zdxys3kkkYyN51TTkUGOsHA
FyFtAlJ0XqAe5oHxeNzW+P+4cn2wzBAz48A6Snhtf8aZoCzRQqJpz9GZfbtF/3YM
XLXD9zxpq4r6m61eT8Ai8wh0tz0BDCPDshjh7zD8F99Ov+NGtjI101+Fkkz2J+tM
ZDKKWrhfKPXjN47BCYTCw7bM2J9HexJc/Xp9mSr7uI7JaHPcoFPIyH7PJkJLOclZ
n4b2SZphYO3WEmyToIWimTupqJIBMYt+jrCC35qOzrSLXUVknA4oWI2IwHjjxNue
eoO8Ao6kMlv6+SGKoZGdW+nZCG40tZ7KVPOTPsClZwes/eeOCQrZHnWJCZqZ95qO
l9oE3KNfNv1Hu1bjFcyj
=oD8m
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message