geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevan Miller <kevan.mil...@gmail.com>
Subject Re: jetty assembly deadlock (trunk only)
Date Mon, 14 Jul 2008 14:30:39 GMT

On Jul 10, 2008, at 10:48 AM, Jarek Gawor wrote:

> It seems that once per day one of the automatic trunk builds deadlocks
> in the following way (this only happens with the Jetty assembly):
>
> Is anybody else seeing this too? We've had similar problems like  
> this before...

I've not seen a deadlock like this, before. The previous ClassLoader  
deadlocks that I recall have involved TransformerCollections and the  
Finalizer thread (these should be fixed).

It looks like AspectJ is breaking the acyclic nature of our DAG  
ClassLoader structure -- causing monitors to be obtained out-of-order  
(making deadlocks a possibility). I haven't located the source for  
org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:73), which appears  
to be the source of the problem. So, don't yet fully understand the  
problem, or how to avoid...

--kevan

>
>
> Found one Java-level deadlock:
> =============================
> "RMI TCP Connection(4)-9.42.75.229":
>  waiting to lock monitor 0x0808c7fc (object 0xd1418d38, a
> org.apache.geronimo.kernel.config.MultiParentClassLoader),
>  which is held by "main"
> "main":
>  waiting to lock monitor 0x0808c83c (object 0xd16967d8, a
> org.apache.geronimo.kernel.config.MultiParentClassLoader),
>  which is held by "RMI TCP Connection(4)-9.42.75.229"
>
> Java stack information for the threads listed above:
> ===================================================
> "RMI TCP Connection(4)-9.42.75.229":
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadClassInternal(MultiParentClassLoader.java:449)
>        - waiting to lock <0xd1418d38> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
>        - locked <0xd169c118> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadClassInternal(MultiParentClassLoader.java:459)
>        - locked <0xd169c118> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
>        - locked <0xd16967d8> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadClassInternal(MultiParentClassLoader.java:459)
>        - locked <0xd16967d8> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
>        - locked <0xd1530c78> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadOptimizedClass(MultiParentClassLoader.java:410)
>        - locked <0xd1530c78> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
> 281)
>        - locked <0xd1530c78> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
> 319)
>        - locked <0xd1530c78> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo.jmxremoting.Authenticator.authenticate(Authenticator.java: 
> 62)
>        at  
> javax 
> .management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java: 
> 214)
>        at  
> javax 
> .management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:181)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at  
> sun 
> .reflect 
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at  
> sun 
> .reflect 
> .DelegatingMethodAccessorImpl 
> .invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at  
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
>        at sun.rmi.transport.Transport$1.run(Transport.java:153)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
>        at  
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java: 
> 466)
>        at sun.rmi.transport.tcp.TCPTransport 
> $ConnectionHandler.run(TCPTransport.java:707)
>        at java.lang.Thread.run(Thread.java:595)
> "main":
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadClassInternal(MultiParentClassLoader.java:449)
>        - waiting to lock <0xd16967d8> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
>        - locked <0xd16aaa28> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadClassInternal(MultiParentClassLoader.java:459)
>        - locked <0xd16aaa28> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadOptimizedClass(MultiParentClassLoader.java:410)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
> 281)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
> 319)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at java.lang.ClassLoader.defineClass1(Native Method)
>        at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>        at  
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 
> 124)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
>        at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadOptimizedClass(MultiParentClassLoader.java:425)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
> 281)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
> 319)
>        - locked <0xd16a1348> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at org.aspectj.weaver.loadtime.Aj 
> $WeaverContainer.getWeaver(Aj.java:115)
>        - locked <0xd171a720> (a java.util.WeakHashMap)
>        at org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:73)
>        - locked <0xd1418d38> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .aspectj 
> .weaver 
> .loadtime 
> .ClassPreProcessorAgentAdapter 
> .transform(ClassPreProcessorAgentAdapter.java:52)
>        at  
> org 
> .apache 
> .geronimo 
> .transformer 
> .TransformerCollection.transform(TransformerCollection.java:43)
>        at  
> sun.instrument.TransformerManager.transform(TransformerManager.java: 
> 122)
>        at  
> sun 
> .instrument.InstrumentationImpl.transform(InstrumentationImpl.java: 
> 155)
>        at java.lang.ClassLoader.defineClass1(Native Method)
>        at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>        at  
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 
> 124)
>        at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
>        at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
>        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .MultiParentClassLoader 
> .loadOptimizedClass(MultiParentClassLoader.java:425)
>        - locked <0xd1418d38> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
> 281)
>        - locked <0xd1418d38> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
> 319)
>        - locked <0xd1418d38> (a
> org.apache.geronimo.kernel.config.MultiParentClassLoader)
>        at  
> org 
> .apache 
> .geronimo 
> .common 
> .propertyeditor.PropertyEditors.findEditor(PropertyEditors.java:79)
>        at  
> org 
> .apache 
> .geronimo 
> .common 
> .propertyeditor.PropertyEditors.findEditor(PropertyEditors.java:120)
>        at  
> org 
> .apache 
> .geronimo 
> .system 
> .configuration.GBeanOverride.loadPropertyEditor(GBeanOverride.java: 
> 399)
>        at  
> org 
> .apache 
> .geronimo 
> .system.configuration.GBeanOverride.getValue(GBeanOverride.java:388)
>        - locked <0xd161f8b0> (a
> org.apache.geronimo.system.configuration.GBeanOverride)
>        at  
> org 
> .apache 
> .geronimo 
> .system 
> .configuration.GBeanOverride.applyOverrides(GBeanOverride.java:348)
>        at  
> org 
> .apache 
> .geronimo 
> .system 
> .configuration 
> .LocalAttributeManager.setAttributes(LocalAttributeManager.java:228)
>        - locked <0xd1430c00> (a
> org.apache.geronimo.system.configuration.LocalAttributeManager)
>        at  
> org 
> .apache 
> .geronimo 
> .system 
> .configuration 
> .LocalAttributeManager.applyOverrides(LocalAttributeManager.java:197)
>        - locked <0xd1430c00> (a
> org.apache.geronimo.system.configuration.LocalAttributeManager)
>        at  
> org 
> .apache 
> .geronimo.kernel.config.Configuration.<init>(Configuration.java:281)
>        at  
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at  
> sun 
> .reflect 
> .NativeConstructorAccessorImpl 
> .newInstance(NativeConstructorAccessorImpl.java:39)
>        at  
> sun 
> .reflect 
> .DelegatingConstructorAccessorImpl 
> .newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java: 
> 494)
>        at org.apache.xbean.recipe.ReflectionUtil 
> $ConstructorFactory.create(ReflectionUtil.java:882)
>        at  
> org 
> .apache.xbean.recipe.ObjectRecipe.internalCreate(ObjectRecipe.java: 
> 272)
>        at  
> org.apache.xbean.recipe.AbstractRecipe.create(AbstractRecipe.java:96)
>        at  
> org.apache.xbean.recipe.AbstractRecipe.create(AbstractRecipe.java:61)
>        at  
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:910)
>        at  
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java: 
> 269)
>        at  
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:103)
>        at  
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:524)
>        at  
> org 
> .apache 
> .geronimo.kernel.basic.BasicKernel.startGBean(BasicKernel.java:359)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .KernelConfigurationManager.load(KernelConfigurationManager.java:163)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .SimpleConfigurationManager 
> .loadConfiguration(SimpleConfigurationManager.java:312)
>        - locked <0xd1677580> (a
> org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .SimpleConfigurationManager 
> .loadConfiguration(SimpleConfigurationManager.java:280)
>        - locked <0xd1677580> (a
> org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at  
> sun 
> .reflect 
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at  
> sun 
> .reflect 
> .DelegatingMethodAccessorImpl 
> .invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at  
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java: 
> 34)
>        at  
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
>        at  
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:815)
>        at  
> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java: 
> 57)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java: 
> 96)
>        at  
> org.apache.geronimo.kernel.config.EditableConfigurationManager$ 
> $EnhancerByCGLIB$$aa87e78a.loadConfiguration(<generated>)
>        at  
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java: 
> 158)
>        at  
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:78)
>        at  
> org 
> .apache 
> .geronimo 
> .kernel 
> .util 
> .MainConfigurationBootstrapper 
> .main(MainConfigurationBootstrapper.java:45)
>        at  
> org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:65)
>        at  
> org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
>
> Found 1 deadlock.


Mime
View raw message