activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Carlson <jcarl...@e-dialog.com>
Subject Re: Messages stuck after Client host reboot
Date Wed, 14 Apr 2010 13:38:06 GMT
Thanks Gary for the, as usual, helpful information.

It looks like the broker maybe suffering from exactly the same problem 
we encountered when implementing client-side failover. Namely that when 
the master broker went down a subsequent read on the socket by the 
client would hang (well actually take a very long time to fail/timeout). 
In that case our TCP connection was ESTABLISHED and looking at the 
broker I see the same thing after the client host goes away (the 
connection is ESTABLISHED). We fixed this issue in our client by setting 
the socket option SO_RCVTIMEO on the connection to the broker.

I noted what the broker appears to do the same thing with the TCP 
transport option soTimeout. It looks like when this is set it winds up 
as a call to java.net.Socket.setSoTimeout when the socket is getting 
initialized. I have not done any socket programming in Java but my 
assumption is that SO_TIMEOUT maps to both SO_RCVTIMEO and SO_SNDTIMEO 
in the C world.

I was hopeful with this option but when I set in in my transport connector:

<transportConnector name="stomp" uri="stomp://mmq1:61613?soTimeout=60000"/>

the timeout does not occur. I actually ran my test case about 15 hours 
ago and I can still see that the broker still has an ESTABLISHED 
connection to the dead client and has a message dispatched to it.

Am I miss understanding what soTimeout is for? I can see in 
org.apache.activemq.transport.tcp.TcpTransport.initialiseSocket that 
setSoTimeout is getting called unconditionally. So what I'm wondering is 
if it is actually calling it with a 0 value despite the way I set up my 
transport connector. I suppose setting this to 0 would explain why it 
apparently never times out where in our client case it eventually did 
timeout (because we were not setting the option at all before).




On 04/14/2010 05:15 AM, Gary Tully wrote:
> The re-dispatch is triggered by the tcp connection dying, netstat can 
> help with the diagnosis here. Check the connection state of the broker 
> port after the client host is rebooted, if the connection is still 
> active, possibly in a timed_wait state, you may need to configure some 
> additional timeout options on the broker side.
>
> On 13 April 2010 19:43, Josh Carlson <jcarlson@e-dialog.com 
> <mailto:jcarlson@e-dialog.com>> wrote:
>
>     I am using client acknowledgements with a prefetch size of 1 with
>     no message expiration policy. When a consumer subscribes to a
>     queue I can see that the message gets dispatched correctly. If the
>     process gets killed before retrieving and acknowledging the
>     message I see the message getting re-dispatched (correctly). I
>     expected this same behaviour if the host running the process gets
>     rebooted or crashes. However, after reboot I can see that the
>     message is stuck in the dispatched state to the consumer that is
>     long gone. Is there a way that I can get messages re-dispatched
>     when a host hosting consumer processes gets re-booted? How does it
>     detect the case when a process dies (even with SIGKILL)?
>
>     I did notice that if I increase my prefetch size and enqueue
>     another message after the reboot, that activemq will re-dispatch
>     the original message. However with prefetch size equal to one the
>     message never seems to get re-dispatched.
>
>
>
>
> -- 
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message