openejb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quintin Beukes <quin...@last.za.net>
Subject Connection Recovery
Date Mon, 23 Nov 2009 14:20:26 GMT
Hey,

One of the big motivations for shifting to OpenEJB 3.1.2 from GF was
the connection recovery built into it. My inquiry was specifically
about failed remote EJB connections because a server restarts or a
network hiccup. We've always had problems with this, especially with
networks running from surface to underground.

This seems to be working. Great job!! Thanks alot. I'm aiming for
complete recovery in the following situations.
1. System exception (any failure on the part of OpenEJB, even if the
cause was my own)
2. Dropped connections due to network
3. Dropped connections from server restart
4. Dropped connections from redeploy (basically the same as (3) I assume).

Though I have a query though. Every time I redeploy, for ex, I have to
manually go and restart all services because it takes very long for
OpenEJB client to notice something is wrong and recover.

Here is an example: If I undeploy, I get something like the following
on a client.

--------- SNIP -----------

javax.ejb.EJBException: Container has suffered a SystemException
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

java.rmi.RemoteException: The server has encountered a fatal error: No
such deployment java.rmi.RemoteException: No deployment:
VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
	java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBean
    org.apache.openejb.server.ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java:425)
    org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:81)
    org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
    org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)
    org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:619)

java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBean
    org.apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java:191)
    org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:79)
    org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
    org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)
    org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:619)


23 Nov 2009 15:59:20,039 ERROR -- Exception: Service
ConfigurationService failed for instance
ConfigurationService[/10.0.0.25] (java.lang.Exception)
java.lang.Exception: ConfigurationService has an unknown failure while
servicing a request.
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:84)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

javax.ejb.EJBException: Container has suffered a SystemException
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

java.rmi.RemoteException: The server has encountered a fatal error: No
such deployment java.rmi.RemoteException: No deployment:
VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
	java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBean
    org.apache.openejb.server.ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java:425)
    org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:81)
    org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
    org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)
    org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:619)

java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/LampCaptureBean
    org.apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java:191)
    org.apache.openejb.server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java:79)
    org.apache.openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java:149)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
    org.apache.openejb.server.ejbd.KeepAliveServer$Session.service(KeepAliveServer.java:213)
    org.apache.openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:619)

--------- SNIP -----------

Then, for a while, even after the EJB service started up completely
and even though I'm creating new InitialContext instances I get the
following. This isn't too bad, though it does take a few minutes
(between 3 and 5) to recover.

--------- SNIP -----------

23 Nov 2009 16:00:24,802 TRACE -- Received packet [R0010088220000000000000000]
23 Nov 2009 16:00:24,804 ERROR -- Exception: Unknown failure servicing
the request (javax.ejb.EJBException)
javax.ejb.EJBException: Unknown Container Exception:
java.rmi.RemoteException: Received invalid response code from server:
-1
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

java.rmi.RemoteException: Received invalid response code from server: -1
    org.apache.openejb.client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)


23 Nov 2009 16:00:24,804 ERROR -- Exception: Service
ConfigurationService failed for instance
ConfigurationService[/10.0.0.25] (java.lang.Exception)
java.lang.Exception: ConfigurationService has an unknown failure while
servicing a request.
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:84)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

javax.ejb.EJBException: Unknown Container Exception:
java.rmi.RemoteException: Received invalid response code from server:
-1
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

java.rmi.RemoteException: Received invalid response code from server: -1
    org.apache.openejb.client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)
    org.apache.openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)
    org.apache.openejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
    org.apache.openejb.client.proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
    $Proxy2.captureLamp(Unknown Source)
    net.kunye.vds.server.services.configuration.ConfigurationService.service(ConfigurationService.java:73)
    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java:86)

--------- SNIP -----------

After it recovers from the above, all is back to normal.

This is quite serious. How I can get around this, even if I have to
make some code changes?

Thanks again. I'm so relieved this problem is finally over. Previously
I've had to do some serious recovery just because a client was left
running and Glassfish's client library opened so many file descriptors
for each failure that the server crashed with a No Space Left on
Device error.

Quintin Beukes

Mime
View raw message