karaf-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Wittmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KARAF-2256) Deadlock when refreshing bundles
Date Tue, 05 Nov 2013 13:41:19 GMT

    [ https://issues.apache.org/jira/browse/KARAF-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813923#comment-13813923
] 

Eric Wittmann commented on KARAF-2256:
--------------------------------------

To pile on a little bit I think I hit another variant of this.  In karaf 2.3.3 (latest 2.x
as of this comment) there is a deadlock when installing the 'war' feature *after* installing
the 'webconsole' feature.  This happens every time for me, here is the output:

{code}
karaf@root> features:install -v webconsole
Installing feature webconsole 2.3.3
Installing feature webconsole-base 2.3.3
Installing feature http 2.3.3
Installing feature jetty 7.6.8.v20121106
Installing bundle mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.activation-api-1.1/2.2.0
Installing bundle mvn:org.apache.geronimo.specs/geronimo-servlet_2.5_spec/1.2
Installing bundle mvn:javax.mail/mail/1.4.5
Installing bundle mvn:org.apache.geronimo.specs/geronimo-jta_1.1_spec/1.1.1
Installing bundle mvn:org.eclipse.jetty/jetty-util/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-io/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-http/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-client/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-continuation/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-jmx/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-server/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-security/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-servlet/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-servlets/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-xml/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-webapp/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-jndi/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-plus/7.6.8.v20121106
Installing bundle mvn:org.eclipse.jetty/jetty-websocket/7.6.8.v20121106
Checking configuration file mvn:org.apache.karaf/apache-karaf/2.3.3/xml/jettyconfig
Installing bundle mvn:org.ops4j.pax.web/pax-web-api/1.1.14
Installing bundle mvn:org.ops4j.pax.web/pax-web-spi/1.1.14
Installing bundle mvn:org.ops4j.pax.web/pax-web-runtime/1.1.14
Installing bundle mvn:org.ops4j.pax.web/pax-web-jetty/1.1.14
Installing bundle mvn:org.apache.felix/org.apache.felix.metatype/1.0.6
Installing bundle mvn:org.apache.karaf.webconsole/org.apache.karaf.webconsole.branding/2.3.3
Installing bundle mvn:org.apache.karaf.webconsole/org.apache.karaf.webconsole.console/2.3.3
Installing bundle mvn:org.apache.karaf.webconsole/org.apache.karaf.webconsole.admin/2.3.3
Installing bundle mvn:org.apache.karaf.webconsole/org.apache.karaf.webconsole.features/2.3.3
Installing bundle mvn:org.apache.karaf.webconsole/org.apache.karaf.webconsole.gogo/2.3.3
Installing bundle mvn:org.apache.felix/org.apache.felix.webconsole.plugins.event/1.0.2
karaf@root> features:install -v war
Installing feature war 2.3.3
Installing feature http-whiteboard 2.3.3
Installing feature http 2.3.3
Installing feature jetty 7.6.8.v20121106
Found installed bundle: org.apache.servicemix.specs.activation-api-1.1 [54]
Found installed bundle: org.apache.geronimo.specs.geronimo-servlet_2.5_spec [55]
Found installed bundle: javax.mail [56]
Found installed bundle: org.apache.geronimo.specs.geronimo-jta_1.1_spec [57]
Found installed bundle: org.eclipse.jetty.util [58]
Found installed bundle: org.eclipse.jetty.io [59]
Found installed bundle: org.eclipse.jetty.http [60]
Found installed bundle: org.eclipse.jetty.client [61]
Found installed bundle: org.eclipse.jetty.continuation [62]
Found installed bundle: org.eclipse.jetty.jmx [63]
Found installed bundle: org.eclipse.jetty.server [64]
Found installed bundle: org.eclipse.jetty.security [65]
Found installed bundle: org.eclipse.jetty.servlet [66]
Found installed bundle: org.eclipse.jetty.servlets [67]
Found installed bundle: org.eclipse.jetty.xml [68]
Found installed bundle: org.eclipse.jetty.webapp [69]
Found installed bundle: org.eclipse.jetty.jndi [70]
Found installed bundle: org.eclipse.jetty.plus [71]
Found installed bundle: org.eclipse.jetty.websocket [72]
Checking configuration file mvn:org.apache.karaf/apache-karaf/2.3.3/xml/jettyconfig
Found installed bundle: org.ops4j.pax.web.pax-web-api [73]
Found installed bundle: org.ops4j.pax.web.pax-web-spi [74]
Found installed bundle: org.ops4j.pax.web.pax-web-runtime [75]
Found installed bundle: org.ops4j.pax.web.pax-web-jetty [76]
Installing bundle mvn:org.ops4j.pax.web/pax-web-extender-whiteboard/1.1.14
Installing bundle mvn:org.apache.karaf.shell/org.apache.karaf.shell.web/2.3.3
Installing bundle mvn:org.apache.karaf.management.mbeans/org.apache.karaf.management.mbeans.web/2.3.3
Installing bundle mvn:org.ops4j.pax.web/pax-web-jsp/1.1.14
Installing bundle mvn:org.ops4j.pax.web/pax-web-extender-war/1.1.14
Installing bundle mvn:org.ops4j.pax.web/pax-web-deployer/1.1.14
Installing bundle mvn:org.ops4j.pax.url/pax-url-war/1.3.6
Refreshing bundles org.ops4j.pax.web.pax-web-runtime (75)
{code}

Hangs on that last line and must be killed and restarted...

> Deadlock when refreshing bundles
> --------------------------------
>
>                 Key: KARAF-2256
>                 URL: https://issues.apache.org/jira/browse/KARAF-2256
>             Project: Karaf
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: 64-bit Linux Oracle JDK 1.7.0_17
>            Reporter: Amichai Rothman
>            Assignee: Achim Nierbeck
>
> When attempting to install the DOSGi feature (by running "features:chooseurl cxf-dosgi
1.4.0" and "features:install cxf-dosgi-discovery-distributed"), the installation hangs along
with some of the bundles which can no longer be started, stopped, checked for imports, etc.
- the Karaf server must be killed and restarted to resume. This is likely not related to this
specific feature, and can happen with other refreshed bundles and installed features as well.
> At a glance it seems like this is caused by the "OPS4J Pax Web - Runtime (1.1.12)" bundle
being stuck in the stopping state due to a deadlock caused by its Activator:
> It receives a removedService notification from a service tracker, which is handled in
a separate thread using a custom executor and eventually tries to resolve some bundle and
ends up waiting for acquireGlobalLock indefinitely.
> This is because at the same time, Felix calls refreshPackages which attempts to stop
the bundle (while holding the lock), whose activator puts a cleanup task in its custom executor
and then attempts to shut down the executor. This never happens, because the previous executor
task initiated from removeService is waiting for the lock, hence the deadlock.
> I'm not entirely sure which of the projects has the underlying bug in it - probably pax
web, possibly Felix if the OSGi specs allow for the behavior that hangs it, but in any case
Karaf is using these versions and exhibiting the deadlock, so at the very least should upgrade
to fixed versions of these libraries, or patch them.
> If anyone who knows these systems better thinks it should be reported in one of the upstream
projects, point me in the right direction and I'll be happy to do it.
> Here is the thread dump, the top two threads show the deadlock, and the other two are
bundles which are stuck as well due to waiting for the same lock (I think).
> "FelixFrameworkWiring" daemon prio=10 tid=0x00007f390002e000 nid=0x35d1 in Object.wait()
[0x00007f3948dd3000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000f1b8ea50> (a java.lang.Object)
>         at java.lang.Object.wait(Object.java:503)
>         at org.ops4j.pax.web.service.internal.Executor.shutdown(Executor.java:91)
>         - locked <0x00000000f1b8ea50> (a java.lang.Object)
>         at org.ops4j.pax.web.service.internal.Activator.stop(Activator.java:140)
>         at org.apache.felix.framework.util.SecureAction.stopActivator(SecureAction.java:667)
>         at org.apache.felix.framework.Felix.stopBundle(Felix.java:2361)
>         at org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4629)
>         at org.apache.felix.framework.Felix.refreshPackages(Felix.java:3951)
>         at org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:172)
>         at java.lang.Thread.run(Thread.java:722)
> "Pax Web Runtime worker" daemon prio=10 tid=0x00007f3904263000 nid=0x370a in Object.wait()
[0x00007f390dfa8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at java.lang.Object.wait(Object.java:503)
>         at org.apache.felix.framework.Felix.acquireGlobalLock(Felix.java:4944)
>         - locked <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at org.apache.felix.framework.StatefulResolver.resolve(StatefulResolver.java:219)
>         at org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1539)
>         at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1439)
>         at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
>         at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>         at org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:1317)
>         at org.apache.felix.framework.ServiceRegistrationImpl$ServiceReferenceImpl.isAssignableTo(ServiceRegistrationImpl.java:521)
>         at org.apache.felix.framework.util.Util.isServiceAssignable(Util.java:280)
>         at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:916)
>         at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
>         at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
>         at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4260)
>         at org.apache.felix.framework.Felix.access$000(Felix.java:74)
>         at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:390)
>         at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:148)
>         at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:127)
>         at org.ops4j.pax.web.service.internal.Activator.updateController(Activator.java:231)
>         at org.ops4j.pax.web.service.internal.Activator$DynamicsServiceTrackerCustomizer$2.run(Activator.java:387)
>         at org.ops4j.pax.web.service.internal.Executor$Future.run(Executor.java:45)
>         at org.ops4j.pax.web.service.internal.Executor$Worker.run(Executor.java:122)
> "fileinstall-/opt/apache-karaf-2.3.1/deploy" daemon prio=10 tid=0x00007f3904018800 nid=0x35a8
in Object.wait() [0x00007f394aba8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at java.lang.Object.wait(Object.java:503)
>         at org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
>         - locked <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at org.apache.felix.framework.Felix.startBundle(Felix.java:1744)
>         at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:944)
>         at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1247)
>         at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1219)
>         at org.apache.felix.fileinstall.internal.DirectoryWatcher.startAllBundles(DirectoryWatcher.java:1208)
>         at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:503)
>         at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:291)
> "NioProcessor-2" prio=10 tid=0x00007f3914014000 nid=0x35fd in Object.wait() [0x00007f394a064000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at java.lang.Object.wait(Object.java:503)
>         at org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
>         - locked <0x00000000e990e018> (a [Ljava.lang.Object;)
>         at org.apache.felix.framework.Felix.startBundle(Felix.java:1744)
>         at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:944)
>         at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:931)
>         at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:479)
>         at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:396)
>         at org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:392)
>         at org.apache.karaf.features.command.InstallFeatureCommand.doExecute(InstallFeatureCommand.java:62)
>         at org.apache.karaf.features.command.FeaturesCommandSupport.doExecute(FeaturesCommandSupport.java:41)
>         at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)
>         at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)
>         at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)
>         at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:474)
>         at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:400)
>         at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
>         at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)
>         at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)
>         at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:89)
>         at org.apache.karaf.shell.ssh.ShellCommandFactory$ShellCommand$1.run(ShellCommandFactory.java:109)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.karaf.shell.ssh.ShellCommandFactory$ShellCommand.start(ShellCommandFactory.java:107)
>         at org.apache.sshd.server.channel.ChannelSession.handleExec(ChannelSession.java:388)
>         at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:235)
>         at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:195)
>         at org.apache.sshd.common.session.AbstractSession.channelRequest(AbstractSession.java:1057)
>         at org.apache.sshd.server.session.ServerSession.running(ServerSession.java:229)
>         at org.apache.sshd.server.session.ServerSession.handleMessage(ServerSession.java:205)
>         at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:566)
>         at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:236)
>         - locked <0x00000000efd56b00> (a java.lang.Object)
>         at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:58)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
>         at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
>         at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:410)
>         at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:710)
>         at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
>         at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
>         at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
>         at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
>         at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:722)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message