geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jencks <david_jen...@yahoo.com>
Subject Re: deadlocks found in tomcat-web m2 release when running tck.
Date Sun, 30 Jan 2011 08:14:49 GMT
Hi Ivan,

Here's my current thinking on this.  I could easily be wrong about many aspects of this :-)
and I'm not done...

I think there are two kinds of problems to avoid here.

The easy one is to synchronize access to our configuration tracking objects.  I think we can
do this by synchronizing on configurations for all access to configurations, configurationModel,
and reloadingConfiguration.  This is really similar to your commit, but without the new object
and (I think) synchronizing access to configurationModel.

The harder part is that we don't want two threads trying to change the state of a configuration
at the same time.  So I think we need an Artifact to lock object map and when we are changing
the state of a configuration we secure the appropriate lock object and synchronize on it while
the state change is happening.  Possibly the ConfigurationStatus in the configurationModel
can be the lock object, I'm not sure if the lifecycle is correct for the status object to
be the lock object.

I think we also need to check before each state change (after getting the lock) what state
the configuration is actually in. 

Finally I don't think a single reloadingConfiguration will work any more.  I'm not sure if
the map of lock objects will work for this too or if we need something else.

Many thanks for fixing at least the initial problem so quickly!
david jencks

On Jan 29, 2011, at 5:59 PM, Ivan wrote:

> For this single issue, another lock on the loadingConfiguration is required, too. I attached
a proposed patch on GERONIMO-5782 
> But that is only for this one, if no objection, I would commit it soon. As many of jsf/ejb
cases are blocked by this. Personally, I always get nervous while introducing more locks in
one classes :-) And I guess that we might check some other bundle extender codes. I am still
working on another dead lock on the WAB deployment, it uses a new thread for the deployment,
too.
> Thanks.
> 
> 2011/1/29 David Jencks <david_jencks@yahoo.com>
> We might have to make the SimpleConfigurationManager less synchronized or better synchronized.
 I think the first rule of avoiding deadlocks is to not call unknown code from within a synchronized
block.  We are definitely calling such code here, we're starting the entire configuration
from inside a synchronized block.
> 
> Basically I think we need to review the SimpleConfigurationManager code and make the
locking finer grained and avoid locking when calling out.  For instance we could synchronize
getConfiguration on the artifact-configuration map rather than the configuration manager object.
> 
> I'll try to look at this in the next couple days.  Any other analysis or suggestions
would be more than welcome.
> 
> thanks
> david jencks
> 
> On Jan 28, 2011, at 7:45 PM, Ivan wrote:
> 
>> The related stack should be, it is caused by the new thread used by Tomcat for listener
initialization, MyFaces listener called some functions in the SimpleConfigurationManager
>> I do not have a good idea so far, maybe we should try to disable the function in
Tomcat ? Or any invocation on the configuraiton manager in any listener/servlet init will
cause the dead lock.
>> 
>> --->
>> "Thread-58" daemon prio=10 tid=0x0000000040ac5000 nid=0x3ff7 waiting for monitor
entry [0x00007f6cdd948000]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at org.apache.geronimo.kernel.config.SimpleConfigurationManager.getConfiguration(SimpleConfigurationManager.java:264)
>>        - waiting to lock <0x00007f6cf84b1570> (a org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
>>        at org.apache.geronimo.naming.reference.ConfigurationAwareReference.getConfiguration(ConfigurationAwareReference.java:61)
>>        at org.apache.geronimo.naming.reference.ConfigurationAwareReference.resolveTargetName(ConfigurationAwareReference.java:79)
>>        at org.apache.geronimo.naming.reference.GBeanReference.getContent(GBeanReference.java:46)
>>        at org.apache.xbean.naming.context.ContextUtil.resolve(ContextUtil.java:61)
>>        at org.apache.xbean.naming.context.ContextUtil$ReadOnlyBinding.getObject(ContextUtil.java:199)
>>        at org.apache.xbean.naming.context.ContextFederation.getFederatedBindings(ContextFederation.java:119)
>>        at org.apache.xbean.naming.context.AbstractFederatedContext.getBindings(AbstractFederatedContext.java:99)
>>        at org.apache.xbean.naming.context.AbstractFederatedContext.getBinding(AbstractFederatedContext.java:86)
>>        at org.apache.xbean.naming.context.AbstractContext.lookup(AbstractContext.java:133)
>>        at org.apache.xbean.naming.context.AbstractContext.lookup(AbstractContext.java:605)
>>        at org.apache.geronimo.naming.java.RootContext.lookup(RootContext.java:55)
>>        at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:148)
>>        at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>        at org.apache.myfaces.application.ApplicationImpl.getProjectStage(ApplicationImpl.java:713)
>>        at org.apache.webbeans.jsf.OwbApplication.getProjectStage(OwbApplication.java:443)
>>        at javax.faces.context.FacesContext.isProjectStage(FacesContext.java:411)
>>        at org.apache.myfaces.config.FacesConfigurator.configureLifecycle(FacesConfigurator.java:1011)
>>        at org.apache.myfaces.config.FacesConfigurator.configure(FacesConfigurator.java:465)
>>        at org.apache.myfaces.webapp.AbstractFacesInitializer.buildConfiguration(AbstractFacesInitializer.java:303)
>>        at org.apache.myfaces.webapp.Jsp21FacesInitializer.initContainerIntegration(Jsp21FacesInitializer.java:73)
>>        at org.apache.myfaces.webapp.AbstractFacesInitializer.initFaces(AbstractFacesInitializer.java:126)
>>        at org.apache.myfaces.webapp.StartupServletContextListener.contextInitialized(StartupServletContextListener.java:110)
>>        at org.apache.geronimo.myfaces.webapp.GeronimoStartupServletContextListener.contextInitialized(GeronimoStartupServletContextListener.java:55)
>>        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4521)
>>        at org.apache.catalina.core.StandardContext$1.call(StandardContext.java:5004)
>>        at org.apache.catalina.core.StandardContext$1.call(StandardContext.java:4999)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:619)
>> 
>> "RMI TCP Connection(6)-172.16.21.3" daemon prio=10 tid=0x0000000040c8e800 nid=0x3ff2
waiting on condition [0x00007f6cdf189000]
>>   java.lang.Thread.State: WAITING (parking)
>>        at sun.misc.Unsafe.park(Native Method)
>>        - parking to wait for  <0x00007f6cfbc53170> (a java.util.concurrent.FutureTask$Sync)
>>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>>        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>        at org.apache.tomcat.util.threads.DedicatedThreadExecutor.execute(DedicatedThreadExecutor.java:56)
>>        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4998)
>>        - locked <0x00007f6cfbc53250> (a org.apache.geronimo.tomcat.GeronimoStandardContext)
>>        at org.apache.geronimo.tomcat.GeronimoStandardContext.access$101(GeronimoStandardContext.java:105)
>>        at org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:728)
>>        at org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47)
>>        at org.apache.geronimo.tomcat.valve.ProtectedTargetValve.invoke(ProtectedTargetValve.java:53)
>>        at org.apache.geronimo.tomcat.GeronimoStandardContext.startInternal(GeronimoStandardContext.java:463)
>>        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>>        - locked <0x00007f6cfbc53250> (a org.apache.geronimo.tomcat.GeronimoStandardContext)
>>        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:812)
>>        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:787)
>>        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
>>        at org.apache.geronimo.tomcat.TomcatContainer.addContext(TomcatContainer.java:310)
>>        at org.apache.geronimo.tomcat.TomcatWebAppContext.doStart(TomcatWebAppContext.java:568)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:975)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:271)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:546)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:110)
>>        - locked <0x00007f6cfbae5210> (a org.apache.geronimo.gbean.runtime.GBeanDependency)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:145)
>>        - locked <0x00007f6cfbae5210> (a org.apache.geronimo.gbean.runtime.GBeanDependency)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:119)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:175)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:253)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:301)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:546)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:110)
>>        - locked <0x00007f6cfbadedf0> (a org.apache.geronimo.gbean.runtime.GBeanDependency)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:145)
>>        - locked <0x00007f6cfbadedf0> (a org.apache.geronimo.gbean.runtime.GBeanDependency)
>>        at org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:119)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:175)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44)
>>        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:253)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:301)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:105)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:127)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:560)
>>        at org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:386)
>>        at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:460)
>>        at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:226)
>>        at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:698)
>>        - locked <0x00007f6cf84b1570> (a org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
>>        at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:677)
>>        - locked <0x00007f6cf84b1570> (a org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:131)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:872)
>>        at org.apache.geronimo.kernel.basic.BasicKernel.invoke(BasicKernel.java:245)
>>        at org.apache.geronimo.kernel.KernelGBean.invoke(KernelGBean.java:344)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
>>        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:131)
>>        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:872)
>>        at org.apache.geronimo.kernel.basic.BasicKernel.invoke(BasicKernel.java:245)
>>        at org.apache.geronimo.system.jmx.MBeanGBeanBridge.invoke(MBeanGBeanBridge.java:172)
>>        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
>>        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>>        at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:619)
>> <--
>> 
>> 2011/1/29 Shawn Jiang <genspring@gmail.com>
>> See my comments to the tck results in tck list and the log in GERONIMO-5782.
>> 
>> -- 
>> Shawn
>> 
>> 
>> 
>> -- 
>> Ivan
> 
> 
> 
> 
> -- 
> Ivan


Mime
View raw message