qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Another strange interoperability quirk between Messenger and the Java Broker.
Date Thu, 04 Sep 2014 16:46:18 GMT
Hi again all - and in particular Rob & Gordon.

I posted the stuff below reporting on my observations of different 
behaviour between the C++ and Java Broker when being prodded by a 
Messenger based client that's writing then reading messages off a queue 
with different settings of outgoing and ingoing window (the different 
scenarios are described below along with my observations).

In addition another thing that I noticed was that when I ran a soak test 
using scenario 3 against the Java Broker it initially looked fine and as 
I say with this scenario the incomingWindow was letting the accepts work 
and the Broker was reporting msgTotalEnqueues & msgTotalDequeues match 
each other however what I noticed in my soak test was that after an 
extended period of running (a million or so "Hello World!" type 
messages) the performance started to drop off, when I checked top the 
Java Broker CPU utilisation had risen to 100%.

I'd perhaps have expected this to happen with one of the scenarios where 
msgTotalDequeues wasn't decreasing and msgDepth was growing 'cause then 
I'd guess that messages were being sent to some store, but if the queue 
wasn't growing then I don't see why the performance would drop off and 
the CPU use increase in the way I observed.

I've attached the code I used.

Interested in any thoughts.
Frase



On 29/08/14 13:51, Fraser Adams wrote:
> Thanks Robbie, Rob has just pinged me too.
>
> I've just updated to r1621286 and the following is what I'm currently 
> observing.
>
> 1. With no incoming or Outgoing Window set (my original had 
> messenger.setOutgoingWindow(1024);) I see:
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
> Java Broker - messages seem to be flowing (my consumer is definitely 
> seeing the messages) and there doesn't seem to by a 100 message stall, 
> but I'm seeing from the stats msgTotalDequeues of zero and 
> msgTotalEnqueues & msgDepth growing. It seems very odd to me that I'm 
> seeing msgTotalDequeues of zero, when I'm definitely receiving the 
> messages!! I'm sending a message body of "Message Number " + count; 
> and when I display the received message content I'm seeing the number 
> incrementing, so it's not as if I'm continually receiving the oldest 
> message on the queue, I definitely seem to be getting all the messages 
> I expect, despite it reporting msgTotalDequeues of zero.
>
>
>
> 2. With messenger.setOutgoingWindow(1024); but no explicit settle on 
> the sent message (though still doing messenger.accept(t); on the 
> tracker I get from messenger.get(message);)
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - messages stall at 100, msgTotalDequeues still zero, 
> msgTotalEnqueues & msgDepth both 100
>
>
>
> 3. With messenger.setIncomingWindow(1024); and no Outgoing window.
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
>
>
> 4. With messenger.setIncomingWindow(1024); and 
> messenger.setOutgoingWindow(1024); but no explicit settle on the sent 
> message (though still doing messenger.accept(t); on the tracker I get 
> from messenger.get(message);)
>
> C++ Broker - my client goes *bang*, basically I've got a fixed size 
> array buffer that represents the "virtual heap" and that's being 
> exceeded so it looks like messages are backing up on the internal 
> Messenger queue.
>
> Java Broker - messages stall at 100, msgTotalEnqueues & 
> msgTotalDequeues both 100
>
>
>
> 5. With messenger.setIncomingWindow(1024); and 
> messenger.setOutgoingWindow(1024); doing messenger.settle(); after I 
> do messenger.put(message); message and doing messenger.accept(t); on 
> the tracker I get from messenger.get(message);
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
>
>
>
> So I'm still observing definite differences in behaviour between the 
> C++ and Java Brokers given the same Messenger client code.
>
> TBH I won't pretend that I really understand (or indeed remotely 
> understand!!) what the Messenger incoming/outgoing windows are really 
> doing, but it looks like a non-zero incoming window is required in 
> order to be able to explicitly accept (or I guess reject) a message, 
> that I guess makes sense, I'm not sure what message status gets 
> returned if not explicitly accepted or rejected - the Java Broker 
> appears to need an explicit accept response in order to dequeue the 
> message (though I still don't understand why I'm receiving the most 
> recent message without dequeues happening). The C++ broker appears to 
> dequeue irrespective of whether an explicit accept has been sent. In 
> "old speak" it *looks* like the C++ broker is behaving kind of like 
> link: {reliability: unreliable} by default with Messenger (in other 
> words it doesn't seem to need acknowledgements).
>
> In terms of the outgoing window it seems that if this is zero the Java 
> Broker doesn't require an explicit settle, but if it's non-zero it 
> does require a settle.
>
> I don't understand the interaction between incoming and outgoing 
> windows, so as I say in point 4 above with the C++ broker my client 
> goes bang (out of memory) if I have *both* incoming and outgoing 
> windows non-zero and no explicit settle, but it's fine if I have a 
> non-zero outgoing window and no explicit settle but with a zero 
> incoming window, that seems weird to me, but like I say the whole 
> window/disposition thing is a bit like voodoo magic to me.
>
>
> Hope this stuff gives some useful food for thought.
> Frase
>
>
> On 29/08/14 12:23, Robbie Gemmell wrote:
>> Hi Fraser,
>>
>> I happened to comment to Rob yesterday afternoon about a very similar
>> situation I was seeing. I worked it through this morning with him and
>> identified some probable bugs, one of which was in the Java broker 
>> and Rob
>> has now made a change for:
>>
>> https://issues.apache.org/jira/browse/QPID-6054
>> https://svn.apache.org/r1621281
>>
>> That fixed the issue I noticed, suggest you give it a try.
>>
>> Robbie
>>
>> On 29 August 2014 09:55, Fraser Adams <fraser.adams@blueyonder.co.uk> 
>> wrote:
>>
>>> Hey all,
>>> I've been messing around writing a soak test for the JavaScript port of
>>> Messenger and came across another difference in behaviour between 
>>> the C++
>>> and Java Brokers.
>>>
>>> I've attached the code I've written, it's relatively simple in 
>>> precis its
>>> behaviour is
>>> 1. Subscribe to amqp://guest:guest@localhost:5673/#
>>> 2. When the subscription address is available send a Message to
>>> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
>>> <subscriptionQueue> has been extracted from the subscription Address.
>>> 3. get and accept the Message.
>>> 4. send a new Message then repeat from 3.
>>>
>>> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that 
>>> listens
>>> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
>>> using its WebSocket transport listening on WS port 5673.
>>>
>>> With the C++ broker the code seems to behave as I'd expect, it 
>>> basically
>>> just runs "forever" and I've had it send and receive a couple of 
>>> million
>>> messages and everything is looking OK. On the C++ Broker I can see
>>> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>>>
>>>
>>> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
>>> console goes up to "Content: Message Number 99") but then it stops, 
>>> what's
>>> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 
>>> 0 and
>>> msgDepth = 100. That seems really weird to me, my application has 
>>> clearly
>>> received the Messages, but yet the Broker is not registering any 
>>> Dequeues.
>>>
>>> Does anyone have any thoughts on this? It's possible that I've done
>>> something wrong, but the C++ Broker seems to be behaving exactly as I'd
>>> intuitively expect and after a couple of million messages my 
>>> application
>>> was still going strong and didn't look like it was leaking anything.
>>>
>>> Cheers,
>>> Frase
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>


Mime
View raw message