openejb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Blevins <david.blev...@visi.com>
Subject Re: Connection Recovery
Date Mon, 23 Nov 2009 20:32:45 GMT
This is because there is no concept of blocking connections for  
applications that are being redeployed.  In fact our (geronimo and  
openejb) redeploy is just an undeploy followed by a deploy, so there's  
no special concept propagated throughout the architecture that  
something is only temporarily gone and will be coming back.

That said, if we added all that, you'd still have to wait the exact  
same amount of time for the undeploy/deploy process plus a small shave  
off of total throughput for the synchronization we'd have to add to  
make the wait-for-redeploy logic work.  The bigger your app gets, the  
longer the time your client spends waiting, which is probably going to  
drive you to splinter your app into tiny fragments which only  
complicates things more and adds even more overhead.

This is one of those "change the problem" types of situations.   
Instead enable multicast discovery on your server and just let it  
run.  When you want to "redeploy", just boot a new server with the new  
version of the app.  When you get that server running to your liking  
-- almost everyone has some sort of initialization they like to do in  
their apps -- just shut down the first server and the client will  
immediately roll over to the second server.  No time spent waiting on  
redeploy and you actually get a chance to pound on your new server a  
bit before it goes live, so no more risk that the app doesn't quite  
work after the redeploy.

You can do this all on the same machine with a shell script or ant  
script that is only slightly fancier than your standard start, deploy,  
run type of script.

-David

On Nov 23, 2009, at 6:20 AM, Quintin Beukes wrote:

> Hey,
>
> One of the big motivations for shifting to OpenEJB 3.1.2 from GF was
> the connection recovery built into it. My inquiry was specifically
> about failed remote EJB connections because a server restarts or a
> network hiccup. We've always had problems with this, especially with
> networks running from surface to underground.
>
> This seems to be working. Great job!! Thanks alot. I'm aiming for
> complete recovery in the following situations.
> 1. System exception (any failure on the part of OpenEJB, even if the
> cause was my own)
> 2. Dropped connections due to network
> 3. Dropped connections from server restart
> 4. Dropped connections from redeploy (basically the same as (3) I  
> assume).
>
> Though I have a query though. Every time I redeploy, for ex, I have to
> manually go and restart all services because it takes very long for
> OpenEJB client to notice something is wrong and recover.
>
> Here is an example: If I undeploy, I get something like the following
> on a client.
>
> --------- SNIP -----------
>
> javax.ejb.EJBException: Container has suffered a SystemException
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> java.rmi.RemoteException: The server has encountered a fatal error: No
> such deployment java.rmi.RemoteException: No deployment:
> VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
> 	java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/ 
> LampCaptureBean
>     
> org 
> .apache 
> .openejb 
> .server 
> .ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java: 
> 425)
>     
> org 
> .apache 
> .openejb 
> .server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java: 
> 81)
>     
> org 
> .apache 
> .openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
>    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java: 
> 149)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
>    org.apache.openejb.server.ejbd.KeepAliveServer 
> $Session.service(KeepAliveServer.java:213)
>     
> org 
> .apache 
> .openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
>    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
>    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:886)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:908)
>    java.lang.Thread.run(Thread.java:619)
>
> java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/ 
> LampCaptureBean
>     
> org 
> .apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java: 
> 191)
>     
> org 
> .apache 
> .openejb 
> .server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java: 
> 79)
>     
> org 
> .apache 
> .openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
>    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java: 
> 149)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
>    org.apache.openejb.server.ejbd.KeepAliveServer 
> $Session.service(KeepAliveServer.java:213)
>     
> org 
> .apache 
> .openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
>    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
>    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:886)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:908)
>    java.lang.Thread.run(Thread.java:619)
>
>
> 23 Nov 2009 15:59:20,039 ERROR -- Exception: Service
> ConfigurationService failed for instance
> ConfigurationService[/10.0.0.25] (java.lang.Exception)
> java.lang.Exception: ConfigurationService has an unknown failure while
> servicing a request.
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:84)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> javax.ejb.EJBException: Container has suffered a SystemException
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:178)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> java.rmi.RemoteException: The server has encountered a fatal error: No
> such deployment java.rmi.RemoteException: No deployment:
> VDS-lamps-ejb-3.0.jar/LampCaptureBean; nested exception is:
> 	java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/ 
> LampCaptureBean
>     
> org 
> .apache 
> .openejb 
> .server 
> .ejbd.EjbRequestHandler.replyWithFatalError(EjbRequestHandler.java: 
> 425)
>     
> org 
> .apache 
> .openejb 
> .server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java: 
> 81)
>     
> org 
> .apache 
> .openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
>    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java: 
> 149)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
>    org.apache.openejb.server.ejbd.KeepAliveServer 
> $Session.service(KeepAliveServer.java:213)
>     
> org 
> .apache 
> .openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
>    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
>    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:886)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:908)
>    java.lang.Thread.run(Thread.java:619)
>
> java.rmi.RemoteException: No deployment: VDS-lamps-ejb-3.0.jar/ 
> LampCaptureBean
>     
> org 
> .apache.openejb.server.ejbd.EjbDaemon.getDeployment(EjbDaemon.java: 
> 191)
>     
> org 
> .apache 
> .openejb 
> .server.ejbd.EjbRequestHandler.processRequest(EjbRequestHandler.java: 
> 79)
>     
> org 
> .apache 
> .openejb.server.ejbd.EjbDaemon.processEjbRequest(EjbDaemon.java:196)
>    org.apache.openejb.server.ejbd.EjbDaemon.service(EjbDaemon.java: 
> 149)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:71)
>    org.apache.openejb.server.ejbd.KeepAliveServer 
> $Session.service(KeepAliveServer.java:213)
>     
> org 
> .apache 
> .openejb.server.ejbd.KeepAliveServer.service(KeepAliveServer.java:233)
>    org.apache.openejb.server.ejbd.EjbServer.service(EjbServer.java:66)
>    org.apache.openejb.server.ServicePool$2.run(ServicePool.java:91)
>    org.apache.openejb.server.ServicePool$3.run(ServicePool.java:120)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:886)
>    java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:908)
>    java.lang.Thread.run(Thread.java:619)
>
> --------- SNIP -----------
>
> Then, for a while, even after the EJB service started up completely
> and even though I'm creating new InitialContext instances I get the
> following. This isn't too bad, though it does take a few minutes
> (between 3 and 5) to recover.
>
> --------- SNIP -----------
>
> 23 Nov 2009 16:00:24,802 TRACE -- Received packet  
> [R0010088220000000000000000]
> 23 Nov 2009 16:00:24,804 ERROR -- Exception: Unknown failure servicing
> the request (javax.ejb.EJBException)
> javax.ejb.EJBException: Unknown Container Exception:
> java.rmi.RemoteException: Received invalid response code from server:
> -1
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> java.rmi.RemoteException: Received invalid response code from  
> server: -1
>     
> org 
> .apache 
> .openejb 
> .client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
>
> 23 Nov 2009 16:00:24,804 ERROR -- Exception: Service
> ConfigurationService failed for instance
> ConfigurationService[/10.0.0.25] (java.lang.Exception)
> java.lang.Exception: ConfigurationService has an unknown failure while
> servicing a request.
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:84)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> javax.ejb.EJBException: Unknown Container Exception:
> java.rmi.RemoteException: Received invalid response code from server:
> -1
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:184)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> java.rmi.RemoteException: Received invalid response code from  
> server: -1
>     
> org 
> .apache 
> .openejb 
> .client.EJBObjectHandler.businessMethod(EJBObjectHandler.java:239)
>     
> org 
> .apache 
> .openejb.client.EJBObjectHandler._invoke(EJBObjectHandler.java:157)
>     
> org 
> .apache 
> .openejb 
> .client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:117)
>     
> org 
> .apache 
> .openejb 
> .client 
> .proxy.Jdk13InvocationHandler.invoke(Jdk13InvocationHandler.java:52)
>    $Proxy2.captureLamp(Unknown Source)
>     
> net 
> .kunye 
> .vds 
> .server 
> .services 
> .configuration 
> .ConfigurationService.service(ConfigurationService.java:73)
>    net.kunye.services.tcp.TCPServiceThread.run(TCPServiceThread.java: 
> 86)
>
> --------- SNIP -----------
>
> After it recovers from the above, all is back to normal.
>
> This is quite serious. How I can get around this, even if I have to
> make some code changes?
>
> Thanks again. I'm so relieved this problem is finally over. Previously
> I've had to do some serious recovery just because a client was left
> running and Glassfish's client library opened so many file descriptors
> for each failure that the server crashed with a No Space Left on
> Device error.
>
> Quintin Beukes
>


Mime
View raw message