From geronimo-dev-return-5100-apmail-incubator-geronimo-dev-archive=incubator.apache.org@incubator.apache.org Sat Nov 22 19:28:17 2003 Return-Path: Delivered-To: apmail-incubator-geronimo-dev-archive@www.apache.org Received: (qmail 74343 invoked from network); 22 Nov 2003 19:28:17 -0000 Received: from daedalus.apache.org (HELO mail.apache.org) (208.185.179.12) by minotaur-2.apache.org with SMTP; 22 Nov 2003 19:28:17 -0000 Received: (qmail 9763 invoked by uid 500); 22 Nov 2003 19:28:00 -0000 Delivered-To: apmail-incubator-geronimo-dev-archive@incubator.apache.org Received: (qmail 9486 invoked by uid 500); 22 Nov 2003 19:27:59 -0000 Mailing-List: contact geronimo-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: list-post: Reply-To: geronimo-dev@incubator.apache.org Delivered-To: mailing list geronimo-dev@incubator.apache.org Received: (qmail 9469 invoked from network); 22 Nov 2003 19:27:59 -0000 Received: from unknown (HELO www.princetongames.org) (66.250.40.202) by daedalus.apache.org with SMTP; 22 Nov 2003 19:27:59 -0000 Received: from localhost (ammulder@localhost) by www.princetongames.org (8.11.6/8.11.6) with ESMTP id hAMJWpJ31312 for ; Sat, 22 Nov 2003 14:32:51 -0500 X-Authentication-Warning: www.princetongames.org: ammulder owned process doing -bs Date: Sat, 22 Nov 2003 14:32:51 -0500 (EST) From: Aaron Mulder X-X-Sender: ammulder@www.princetongames.org To: geronimo-dev@incubator.apache.org Subject: Remote JMX Notifications Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: minotaur-2.apache.org 1.6.2 0/1000/N I'm working through the remote JMX notifications now (thanks Hiram!). The basic operation seems to be fine. There's one strange issue, though. When I remove the listener, after some time has passed (60-90 seconds), there's a stack trace on the server side during firing events. Then there are a flurry of connection events on the client side. Then (after a few more seconds) the server runs out of memory (at both 128M and 256M max heap). Then a couple minutes later, a flurry of connection events on the server side. I don't know for sure that all this is related, but it's kind of weird. Log output below (though I disabled trace on remoting because it was too verbose). All I can think if is that perhaps the NotificationListener proxy that's actually added to the server-side MBeanServer is perhaps not removed properly? Additionally, there's one enhancement that would be nice. Right now it looks like the NotificationListener is passed to the server as a Remote object, but the NotificationFilter either is not, or is not used by the server. In my case, the NotificationFilter is quite simple and does throw out some events, so it would be nice if it were executed on the server side, which would prevent any extraneous events from being passed over the wire. I guess that's probably a loss in the case of a complex NotificationFilter (which might itself be remote). Can we put in an optimization to execute the NotificationFilter on the server side if it's Serializable but not Remote, and leave it to execute on the client side if it's Remote? Thanks, Aaron ----------------------- SERVER --------------------------- [java] 14:19:51,286 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] RuntimeException caught from handleNotification [java] java.lang.reflect.UndeclaredThrowableException [java] at $Proxy0.handleNotification(Unknown Source) [java] at mx4j.server.interceptor.NotificationListenerMBeanServerInterceptor$ListenerWrapper.handleNotification(NotificationListenerMBeanServerInterceptor.java:57) [java] at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:343) [java] at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:317) [java] at org.apache.geronimo.kernel.service.AbstractManagedObject2.sendNotification(AbstractManagedObject2.java:298) [java] at org.apache.geronimo.kernel.service.GeronimoMBeanContext.sendNotification(GeronimoMBeanContext.java:180) [java] at org.apache.geronimo.kernel.deployment.DeploymentController.updateDeploymentStatus(DeploymentController.java:214) [java] at org.apache.geronimo.kernel.deployment.DeploymentController.executeJob(DeploymentController.java:196) [java] at org.apache.geronimo.kernel.deployment.DeploymentController.access$200(DeploymentController.java:89) [java] at org.apache.geronimo.kernel.deployment.DeploymentController$DeploymentWaiter.run(DeploymentController.java:324) [java] at java.lang.Thread.run(Thread.java:534) [java] Caused by: org.apache.geronimo.remoting.transport.TransportException: Request time out. [java] at org.apache.geronimo.remoting.transport.async.ChannelPool.sendRequest(ChannelPool.java:468) [java] at org.apache.geronimo.remoting.transport.async.AsyncClient.sendRequest(AsyncClient.java:96) [java] at org.apache.geronimo.remoting.transport.RemoteTransportInterceptor.invoke(RemoteTransportInterceptor.java:104) [java] at org.apache.geronimo.remoting.MarshalingInterceptor.invoke(MarshalingInterceptor.java:84) [java] at org.apache.geronimo.remoting.InterVMRoutingInterceptor.invoke(InterVMRoutingInterceptor.java:89) [java] at org.apache.geronimo.proxy.SimpleRPCContainer.invoke(SimpleRPCContainer.java:84) [java] at org.apache.geronimo.proxy.ProxyContainer.invoke(ProxyContainer.java:85) [java] ... 11 more [java] java.lang.OutOfMemoryError: unable to create new native thread [java] at java.lang.Thread.start(Native Method) [java] at org.apache.geronimo.remoting.transport.async.bio.BlockingChannel.open(BlockingChannel.java:194) [java] 14:21:31,392 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] at org.apache.geronimo.remoting.transport.async.ChannelPool$PooledAsynchChannel.open(ChannelPool.java:161) [java] at org.apache.geronimo.remoting.transport.async.ChannelPool.associate(ChannelPool.java:236) [java] at org.apache.geronimo.remoting.transport.async.bio.BlockingServer.run(BlockingServer.java:197) [java] at java.lang.Thread.run(Thread.java:534) [java] 14:21:31,970 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:21:50,116 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:05,831 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] java.lang.OutOfMemoryError: unable to create new native thread [java] at java.lang.Thread.start(Native Method) [java] at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:515) [java] at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:870) [java] at org.apache.geronimo.remoting.transport.async.ChannelPool.dispatchRequest(ChannelPool.java:417) [java] at org.apache.geronimo.remoting.transport.async.ChannelPool.dispatch(ChannelPool.java:337) [java] at org.apache.geronimo.remoting.transport.async.ChannelPool.access$300(ChannelPool.java:82) [java] at org.apache.geronimo.remoting.transport.async.ChannelPool$PooledAsynchChannel.receiveEvent(ChannelPool.java:207) [java] at org.apache.geronimo.remoting.transport.async.bio.BlockingChannel.run(BlockingChannel.java:300) [java] at java.lang.Thread.run(Thread.java:534) [java] 14:22:20,783 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:20,938 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:36,819 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:36,920 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,112 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,281 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,452 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,668 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,767 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:37,897 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,211 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,340 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,398 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,552 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,631 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 [java] 14:22:38,656 DEBUG [ChannelPool] channel connecting to: async://falcon:34513 ------------------------ CLIENT ------------------------- 14:19:51,391 DEBUG [JmxProgressObject] Removing notification listener Operation has completed Affected target/modules: org.apache.geronimo.kernel.deployment.GeronimoTargetModule@893d84b1 ------ Deploy Module ------ 1 target currently selected. 1) Select targets (usually servers or clusters) to work with 2) Distribute test.jar to selected targets 3) Deploy test.jar to selected targets 4) Redeploy test.jar to selected targets Action ([1-4] or [B]ack): 14:19:51,449 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:52,252 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:53,161 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:53,696 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:53,906 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:54,549 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:56,524 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:56,868 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:56,876 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:56,882 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:56,889 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:57,114 DEBUG [ChannelPool] channel connecting to: async://falcon:3434/Remoting#0 14:20:59,520 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:00,126 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:00,305 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:01,533 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:01,904 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:02,560 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:03,222 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:03,691 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:04,060 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:04,064 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:05,021 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:05,026 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:06,735 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:07,312 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:07,781 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:08,373 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:09,480 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:10,043 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:10,707 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:11,223 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:11,815 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:12,424 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:12,980 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub 14:21:13,771 DEBUG [ChannelPool] channel connecting to: async://localhost:3434/JMX#geronimo.remoting:target=MBeanServerStub