qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Karlsen <MKarl...@idbs.com>
Subject RE: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client recv() when HornetQ stopped/becomes unavailable]
Date Fri, 08 Apr 2016 09:02:37 GMT
Hello Robbie (and All),

Many thanks for the post.

I am aware that HornetQ has issues w.r.t. ttl and so on. However, I thought I could send it
an empty frame on a regular basis and this would keep the connection alive despite this --
perhaps I was mistaken?

I have now tested the connection-ttl-override in Artemis and it works as expected. Unfortunately
Artemis is not an option at present... and connection-ttl-override in HornetQ does not work.

> It might be possible to tinker via reflection to trick it into thinking it should be.

Thank you for the suggestion -- I will look in to this.

> You also mention sending an empty frame, can you elaborate more on what you are doing
exactly?

Yes, I can. Incidentally, I modified handleFrame() to be public at present (clearly not a
long term solution). Then, on connection init 
I have the following code:

transport = Proton.transport();
transport.bind(connection);
transport.setIdleTimeout(0);

Open open = new Open();
open.setIdleTimeOut(new UnsignedInteger(0));
TransportFrame openFrame = new TransportFrame(0, open, null);
 ((TransportImpl) transport).handleFrame(openFrame);

[plus a Task t = event.getReactor().schedule(10000, this); to schedule a regular event ]

Then in onTimerTask() I have the following:

byte[] emptyFrame = (new AmqpFramer()).createEmptyFrame(0);
 ((TransportImpl) transport).input(emptyFrame, 0, emptyFrame.length);
Task t = e.getReactor().schedule(10000, this);

I have experimented with various timeout values but none have achieve the desired result so
far.

> If you look at the protocol trace (environment variable PN_TRACE_FRM=true) do you actually
see it sending anything?

Yes, at present I get a number of [[596512129:0] <- Empty Frame] messages -- however, when
I do the same thing with Artemis I get Empty Frames going in both directions, so I suspect
HornetQ is not responding correctly.

Thank you & regards,
Matthew


-----Original Message-----
From: Robbie Gemmell [mailto:robbie.gemmell@gmail.com] 
Sent: 07 April 2016 13:14
To: users@qpid.apache.org
Subject: Re: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client
recv() when HornetQ stopped/becomes unavailable]

Hi Matthew,

I think your initial issue here is likely that HornetQ is neither requesting clients send
it heartbeats, or supporting sending heartbeats to them if it so requested. It is however
separately enforcing a timeout in its IO layer. I say that based on the heartbeat handling
only quite recently being fixed in ActiveMQ Artemis, which is based on the previously donated
HornetQ codebase.

In AMQP 1.0 each side announces their own independent timeout requirements to the other side,
with each then doing what they need to satisfy the peers requirements, either through normal
messaging activity or by explicit heartbeat/empty frames in its absense. The proton engine
satisfies/enforces these requirements via the transport tick method, as it seems you might
be aware, which checks that requested heartbeats have been received and required heartbeats
get sent, returning when it next needs to be called to keep on top of that.

The reactor takes care of calling tick itself as far as I am aware and that being the case
you also calling it should have no real effect on overall behaviour, which seems to be what
you observed. In any case, if the broker has indeed not advertised an idleTimeout, the tick()
will not produce heartbeat/empty frames because it has effectively been told it doesn't need
to. It might be possible to tinker via reflection to trick it into thinking it should be.

You also mention sending an empty frame, can you elaborate more on what you are doing exactly?
If you look at the protocol trace (envieonment variable PN_TRACE_FRM=true) do you actually
see it sending anything? The heartbeating stuff is all hidden behind tick(), so are you achieving
that via reflection?

Robbie

On 6 April 2016 at 10:20, Matthew Karlsen <MKarlsen@idbs.com> wrote:
> Hello All,
>
> After being directed towards the Proton-J reactor (away from the messenger), I have been
experimenting with reactor. I modified the supplied reactor examples to send/recv to/from
a queue on HornetQ fairly quickly.
>
> However, I appear to be having less success implementing a heartbeat/keep-alive (HornetQ
2.4.0 will disconnect after a one minute default if it does not receive data from the client).
>
> I have scheduled a regular event using [ e.getReactor().schedule(10000, this); ] with
a view to implementing a heartbeat. I then override onTimerTask() and call tick() or send
an empty frame within onTimerTask() then re-schedule another call of onTimerTask() -- I have
tried several permutations.
>
> However, whether I use tick(), or send an empty frame, it does not seem to maintain the
connection.
>
> I was wondering:
> 1) Are there any better approaches?
> 2) What is the "correct" approach/location for using tick()?
>
> Thank you & regards,
> Matthew
>
>
>
> -----Original Message-----
> From: Robbie Gemmell [mailto:robbie.gemmell@gmail.com]
> Sent: 18 March 2016 17:59
> To: users@qpid.apache.org
> Subject: Re: Rapid generation of IOExceptions on Proton-J client 
> recv() when HornetQ stopped/becomes unavailable
>
> On 18 March 2016 at 12:15, Matthew Karlsen <MKarlsen@idbs.com> wrote:
>> Hello All,
>>
>> We have a queue running in HornetQ 2.4.0 (embedded within Wildfly), with a Proton-J
0.12.0 client periodically connecting to this queue.
>>
>> If HornetQ stops or becomes unavailable when the Proton-J client is running, the
Proton-J client continually generates exceptions similar to that below (very rapidly).
>>
>> The issue is that the IOException generated exception is handled in 
>> MessengerImpl's processActive() by generating an "Error processing connection" message
in the logs, rather than recv() throwing an exception and hence the higher level program logic
is unable to react to the problem.
>>
>> Is this a bug or am I missing something important?
>>
>> Thank you,
>> Matthew
>>
>> java.io.IOException: An existing connection was forcibly closed by the remote host
>>         at sun.nio.ch.SocketDispatcher.read0(Native Method)
>>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
>>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>>         at org.apache.qpid.proton.driver.impl.ConnectorImpl.read(ConnectorImpl.java:129)
>>         at org.apache.qpid.proton.driver.impl.ConnectorImpl.process(ConnectorImpl.java:94)
>>         at org.apache.qpid.proton.messenger.impl.MessengerImpl.processActive(MessengerImpl.java:738)
>>         at org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:895)
>>         at org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
>>         at org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:446)
>>         at org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:451)
>>         ...
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For 
>> additional commands, e-mail: users-help@qpid.apache.org
>>
>
> Hi Matthew,
>
> It certainly doesnt seem desirable, I'd guess its a bug, though someone actually familiar
with Messenger could certainly correct me on that.
>
> Messenger isn't widely used on the proton-j side, and in general Messenger isn't getting
much developer attention these days (arguably never did in proton-j, which is mostly still
used as an pure protocol engine as intended at the outset) since folks are concentrated more
on newer reactive APIs.
>
> Proton-J is a bit behind the curve in terms of the higher level reactive work being/already
done in some of the other lanaguages (something I'll be looking to get to), but there was
a Java port done of the 'reactor' that exists in proton-c and some of its bindings. I'm not
particularly familiar with it either, or how it would handle this situation, but it may be
worth you looking at as alternative to Messenger. You can see some example of it in the examples/java/reactor
dir, e.g. the Send class (the Recv class in this case is actually a server/listener that accepts
incoming connections, such as those made by Send).
>
> Robbie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For 
> additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional commands, e-mail:
users-help@qpid.apache.org

Mime
View raw message