felix-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Egli <e...@adobe.com>
Subject Re: [java6][deadlock] with parallel classloading, and how to deal with it
Date Wed, 19 Mar 2014 14:47:58 GMT
Hi,

I wrote a test which reproduces the deadlock on Java 6 (MacOS). It's
attached to FELIX-4462 ([0]). Unfortunately I only managed to get it fail
when running from Eclipse, not via 'mvn clean test'. I had to use byteman
to inject a delay in the BundleClassLoader.findClass - and if that inject
is not properly applied, the deadlock is not triggered. So there seems to
be a difference with byteman-injection under maven vs standalone.

Re a possible fix: I've also added a trivial patch suggesting to
synchronized findClass - but fear that this is too heavy, as it would
avoid parallel classloading for Java 7++. At least, when synchronizing
findClass, my test succeeds :)

Cheers,
Stefan
--
[0] https://issues.apache.org/jira/browse/FELIX-4462

On 3/17/14 4:22 PM, "Richard S. Hall" <heavy@ungoverned.org> wrote:

>On 3/17/14, 09:07 , Stefan Egli wrote:
>> Hi Richard,
>>
>> (see below)
>>
>> On 3/17/14 1:34 PM, "Richard S. Hall" <heavy@ungoverned.org> wrote:
>>
>>> On 3/14/14, 05:26 , Stefan Egli wrote:
>>>> Hi,
>>>>
>>>> I just ran into the famous deadlock with Java 6 where the same class
>>>>is
>>>> being loaded from two different bundles (also see FELIX-3953). This
>>>> happened on a startup of a CQ instance.
>>>>
>>>> I'll add more analysis details but my question is: what is the
>>>> suggested best practice to handle this issue?
>>>>    * Use '-XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass' as
>>>> suggested in [0] and [1]? But then again [1] says "You must treat it
>>>>as
>>>> unstable, experimental, temporary, unsupported".
>>> This is your only option if you are not able to use Java 7.
>>>
>>>>    * Or would it be worth while to reconsider fixing FELIX-3953?
>>> Unless we modify the JRE, there is no way to fix it. The Felix
>>>framework
>>> doesn't hold locks while class loading.
>> I obviously don't know the details of the BundleClassLoader, so I can
>>only
>> speculate. But it seems the problem is its 'class locking'
>>(m_classLocks).
>> The assumption there seems, that BundleClassLoader.findClass() is always
>> called unsynchronized. But this is clearly not the case in Java 6.
>
>The underlying issue of eliminating class loader cycle deadlocks is not
>possible.
>
>The m_classLocks structure was put in place to support Java 7 parallel
>class loaders. So, I guess the question is, does this approach make
>deadlock more likely on older JVMs? I don't know the answer to that nor
>do I have the time to look into it, but it would be worthwhile to
>investigate/understand.
>
>>
>> So maybe Java 6 would need a special class loader which is
>> synchronize-aware.. (in which case you could for example synchronize the
>> BundleClassLoaderJava6.findClass() - given that the synchronization
>>would
>> be done by the VM anyway).
>>
>> Or introduce a 'proxy class loader' such that the VM would always only
>>(be
>> able to) synchronize the proxy.. (if that's feasible)
>
>Again, the underlying issue is not resolvable. Before we did
>m_classLocks, we were actually locking on "this" (i.e., the class
>loader) before calling findLoadedClass() and that was the only time we
>held a lock and we were still getting into deadlocks because of the
>JVM-held locks.
>
>As I state above, the only real issue in my mind is if we were to, for
>example, effectively go back to locking on "this" for pre-Java7 JVMs,
>would this solve the issue in your specific use case?
>
>What we were trying to do was to keep a single approach for class
>loading, but one that would automatically gain the parallel benefits on
>Java 7. It could be possible that it interferes somehow.
>
>If you want to do a test, modify the class loader code so lock on "this"
>instead of m_classLocks when accessing the m_classLocks data structure,
>I think that is all that would be necessary to simulate it. If it turns
>out to help your situation, then it might be an easy patch. We just have
>to modify the class loader to detect which lock it should use.
>
>-> richard
>
>
>-> richard
>
>>
>> Cheers,
>> Stefan
>>
>>> -> richard
>>>> Here's more details (see thread-dumps below):
>>>>    * Thread 158 (on behalf on bundle X) wants to load class A which is
>>>> part of bundle Y - hence goes via getClassByDelegation - then does a
>>>> BundleWiringImpl$BundleClassLoader.findClass
>>>>    * before Thread 158 continues VM threading decides to give Thread
>>>>156
>>>> some execution time:
>>>>    * Thread 156 now wants to load same class A, but this time directly
>>>> on behalf of bundle Y.
>>>>    * Due to [1] Thread 156 internally does a synchronized
>>>> loadClassInternal, thus holds the lock on 6f025d710 (the
>>>> BundleClassLoader of bundle Y) (I was able to verify this by hooking
>>>>the
>>>> deadlocked-VM to a debugger, which indicated exactly this)
>>>>    * Thread 156 now continues into
>>>> BundleClassLoader.findClass(BundleWiringImpl.java:2115) where it has
>>>>to
>>>> do a m_classLocks.wait() - since Thread 158 earlier marked the
>>>> BundleClassLoader 'in use/locked for class A'.
>>>>    * Thread 158 on the other hand continues the findClass execution,
>>>> eventually walks into checkCerts, which is synchronized(this), which
>>>> Thread 156 holds
>>>>    * and we have the famous classloading deadlock.
>>>>
>>>> Cheers,
>>>> Stefan
>>>> --
>>>> [0] https://issues.apache.org/jira/browse/FELIX-3953
>>>> [1]
>>>> 
>>>>http://underlap.blogspot.de/2006/11/experimental-fix-for-sunbug-4670071
>>>>.h
>>>> tml
>>>>
>>>> PS: Used org.apache.felix.framework 4.3.0.R1558704 - which is a
>>>> snapshot-release as of SVN revision 1558704
>>>>
>>>> "Thread-158" daemon prio=5 tid=7f83a4cbd800 nid=0x25b73e000 waiting
>>>>for
>>>> monitor entry [25b73c000]
>>>>      java.lang.Thread.State: BLOCKED (on object monitor)
>>>>           at java.lang.ClassLoader.checkCerts(ClassLoader.java:788)
>>>>           - waiting to lock <6f025d710> (a
>>>> org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5)
>>>>           at 
>>>>java.lang.ClassLoader.preDefineClass(ClassLoader.java:493)
>>>>           at 
>>>>java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>>           at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass
>>>>(B
>>>> undleWiringImpl.java:2297)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega
>>>>ti
>>>> on(BundleWiringImpl.java:1519)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl
>>>>.j
>>>> ava:77)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass
>>>>(B
>>>> undleWiringImpl.java:1973)
>>>>           at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(Bundle
>>>>Wi
>>>> ringImpl.java:1392)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.searchImports(BundleWiringI
>>>>mp
>>>> l.java:1571)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega
>>>>ti
>>>> on(BundleWiringImpl.java:1502)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl
>>>>.j
>>>> ava:77)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass
>>>>(B
>>>> undleWiringImpl.java:1973)
>>>>           at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>> [class A being loaded from a class of bundle X]
>>>>
>>>> "Thread-156" daemon prio=5 tid=7f83a57ac800 nid=0x25b36b000 in
>>>> Object.wait() [25b369000]
>>>>      java.lang.Thread.State: WAITING (on object monitor)
>>>>           at java.lang.Object.wait(Native Method)
>>>>           at java.lang.Object.wait(Object.java:485)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass
>>>>(B
>>>> undleWiringImpl.java:2115)
>>>>           - locked <6f788bf40> (a java.util.HashMap)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega
>>>>ti
>>>> on(BundleWiringImpl.java:1519)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl
>>>>.j
>>>> ava:77)
>>>>           at
>>>> 
>>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass
>>>>(B
>>>> undleWiringImpl.java:1973)
>>>>           at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>> [class A being loaded from a class of bundle Y]
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>>> For additional commands, e-mail: users-help@felix.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>> For additional commands, e-mail: users-help@felix.apache.org
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>For additional commands, e-mail: users-help@felix.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org


Mime
View raw message