qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Schloming <...@alum.mit.edu>
Subject Re: Optimising Proton Messenger data transfers & msgr-send/msg-recv oddities
Date Thu, 06 Nov 2014 12:43:49 GMT
On Sat, Nov 1, 2014 at 5:07 AM, Fraser Adams <fraser.adams@blueyonder.co.uk>
wrote:

> Thanks for the responses Rafael.
>
> Re the pn_messenger_send stuff, I guess I should have been clearer; what I
> was really meaning was that I'd seen pn_messenger_put actually transfer
> data without requiring the send. I realise that pn_messenger_put doesn't
> block the question was more about at which point is it likely that
> *transmission* would cause blocking (and thus the message not be sent by
> the put), my hunch was that with small messages the transmission probably
> won't block.
>

I think this has actually changed over the lifespan of the code. At one
point I think I had pn_messenger_put making a non blocking work call so it
would push out anything as soon as it could, but I did some profiling and
this provided to be slower than allowing more batching, so I took it out. I
think now it doesn't actually do anything until you poke it by calling send
or work.


>
> If I'm honest part of my problem is that most of the experiments I'm doing
> with Messenger are in non-blocking mode, so I can see the relationship
> between put and send somewhat more in blocking mode but what goes on in
> non-blocking mode is a bit of a mystery - sorry I should have been clearer.
>

I would say in non-blocking mode just pretend that send doesn't exist and
use work instead.


>
>
> Re the credit stuff, many thanks for the comprehensive explanation, one
> remaining question then - am I correct in thinking that in Messenger it is
> pn_messenger_recv that would be the API call that would be used to set the
> credit? From your explanation of the credit based flow control that seems
> the most likely place, but the documentation for that makes no reference to
> credit, nor that there's a relationship with the sender. If my assumption
> is actually correct it might be useful if you could add your explanation of
> credits below to the API docs for pn_messenger_recv.
>

Yes, so pn_messenger_recv takes credits as an argument, or if you pass it a
-1 then it will allocate credits automatically.

If you file a JIRA, for doc updates for 0.9 and put a pointer to this and
any other threads you think have valuable tidbits, I'll make a point of
getting as much as I can into the docs for 0.9.

--Rafael


>
> Many thanks again for your response,
> Frase
>
>
>
> On 31/10/14 21:21, Rafael Schloming wrote:
>
>> Hi,
>>
>> I'll do my best to answer what I can... comments inline...
>>
>> On Fri, Oct 31, 2014 at 8:10 AM, Fraser Adams <
>> fraser.adams@blueyonder.co.uk
>>
>>> wrote:
>>> Hey all,
>>> OK I'll 'fess up I have to admit that although I've been tinkering with
>>> Messenger for a while now I don't *really* understand some of the terms
>>> that get used such as credit, disposition, settlement. I think that I was
>>> OK with qpid::messaging's setCapacity stuff and how to use that to
>>> optimise
>>> prefetch and also using qpid::messaging link controls in the Address
>>> string
>>> to set/disable reliability but the settings on Messenger remain a mystery
>>> to me.
>>>
>>> What really brought that home in my mind was when I started playing with
>>> the msgr-send and msgr-recv applications in <proton>/tests/tools/apps to
>>> try to figure what sort of throughput I might get with Messenger.
>>>
>>>  First off let me just say that I'm not that familiar with msgr-send and
>> msgr-recv. I think Ken wrote those, and as I understand it they are
>> command
>> line sending and receiving programs that happen to be implemented via
>> messenger, however they aren't intended to be command line interfaces to
>> messenger, so the terminology and parameters, may not line up 100% with
>> the
>> messenger api-doc.
>>
>>
>>  A while back Gordon was involved in a performance conversation and
>>> mentioned testing with the settings below
>>> ./msgr-recv -c 1000000
>>> ./msgr-send -c 1000000 -b 64
>>>
>>> So sending/receiving a million 64 octet messages and seeing what the
>>> performance is - so far so good.
>>>
>>> But I then tinkered around and hacked in some code to display the count
>>> for the sent and received messages and then did:
>>>
>>> ./msgr-recv -c 100
>>> ./msgr-send -c 100 -b 1000000
>>>
>>> in other words sending & receiving 100 1MB messages - I actually used the
>>> large message size as much to slow things down as anything, but what I
>>> observed was that the 100 messages were all being sent before msgr-recv
>>> started to display any received count numbers.
>>>
>>> When I looked at the usage I noticed the -p option "Send batches of #
>>> messages" and sure enough if I do
>>> ./msgr-send -c 100 -p 10 -b 1000000
>>>
>>> I see msgr-recv catch up every 10 messages.
>>>
>>>
>>> What I *think* is going on is that when the count in the internal
>>> Messenger queue (pn_messenger_outgoing(messenger)) exceeds the batch
>>> size
>>> it calls pn_messenger_send(messenger, -1);
>>>
>>>  Yes, from my brief perusal of the code I would concur.
>>
>>
>>  But that makes me unclear in my mind what the differences between
>>> pn_messenger_put and pn_messenger_send actually are, I've certainly seen
>>> pn_messenger_put actually send messages. I realise that there's a comment
>>> "The message may also be sent if transmission would not cause blocking"
>>> but
>>> I'm not clear at exactly which point blocking would occur, I'm guessing
>>> that I'm noticing this because of my large messages? The problem of
>>> course
>>> is that if I use tiny little messages I can't actually see if any
>>> batching
>>> actually occurs or whether in the small message case pn_messenger_put
>>> merrily whizzes out the small messages without really needing
>>> pn_messenger_send to give them a helping kick.
>>>
>>>  I'm a little confused by this question. The pn_messenger_put operation
>> hands a new message over to the messenger and as stated in the API doc is
>> guaranteed not to block. The pn_messenger_send operation on the other hand
>> does not take a new message, and exists solely for the purpose of blocking
>> until previously "put" messages are actually sent. The pn_messenger_send
>> operation is the equivalent of
>> pn_mesenger_work_until_you_send_N_messages(N).
>>
>>
>>  Does that make sense? It'd be useful for someone who knows this stuff to
>>> explain how the Messenger store works and how the various API calls
>>> relate
>>> to credit, disposition & settlement (I'm pretty sure the latter relates
>>> to
>>> the tracker/window/accept/settle stuff but not so sure about the first
>>> two). I'd also quite like to know how this stuff relates to the
>>> capacity/reliability stuff on qpid::messaging.
>>>
>>>  Credit refers to credit based flow control. In a credit based flow
>> control
>> scheme, the receiver maintains a "credit balance". The credit balance is
>> just a number that indicates how many messages the receiver is capable of
>> receiving at any given point in time. The receiver periodically informs
>> the
>> sender of this number, and the sender guarantees never to send unless this
>> number is positive. The sender will also decrement its copy of the credit
>> balance whenever it sends a message. This guarantees that the sender will
>> never send more messages than the receiver has requested.
>>
>> So the term credit can mean a bunch of different things in different
>> contexts, but when used as a "unit", e.g. 10 credits, a credit pretty much
>> translates into permission to send one message. So when the receiver
>> issues
>> 10 credits, it is issuing permission to send up to 10 messages.
>>
>> Now depending on what exact policy you use to issue credits, you can
>> implement a lot of different semantics, e.g. you can send 1 credit to
>> fetch
>> exactly one message and be done with it, or you can renew the sender's
>> credit whenever it falls below a certain threshold, and there are many
>> more
>> options. The concept of capacity is just a specific policy for managing
>> credits. This policy assumes the receiver has a fixed buffer for holding
>> messages, and it issues credit to reflect the number of empty slots in
>> that
>> buffer.
>>
>> Settlement of a delivery refers to an endpoint being done with and
>> forgetting everything about a given delivery. Disposition refers to the
>> state of the delivery at the time of settlement, e.g. was it accepted vs
>> rejected. Sometimes the disposition is null because the state at the time
>> of settlement is unknown, e.g. the sender can choose to "pre-settle" the
>> message as it is sent, i.e. forget about it as soon as it hits the wire.
>> In
>> this case the state when the receiver settles it will never be known. Just
>> like credit is a lower level/more general concept than capacity and can be
>> used to implement a greater variety of semantics, settlement is a lower
>> level/more general concept than reliability and can be used to implement a
>> variety of different QoS levels.
>>
>>
>>  Also I *think* that there is a problem with the python version of
>>> msgr-send.py I'd expect that to run more slowly than the C version, but
>>> when I did:
>>>
>>> ./msgr-recv.py -c 100
>>> ./msgr-send.py -c 100 -b 1000000
>>>
>>> it returned more or less immediately and when I increase the -c value I
>>> appear to be seeing the same throughput irrespective of the value of the
>>> -b
>>> value. I've not really looked too deeply at the code but I wonder if that
>>> rings any bells for anyone?
>>>
>>>  Doesn't ring a bell for me. I'd file a JIRA and maybe poke Ken.
>>
>>
>>  Sorry if these things are obvious to the people who know, but I figured I
>>> probably wasn't the only one who didn't actually know this stuff and as
>>> I've got no shame I thought I'd raise my head above the parapet and
>>> expose
>>> my ignorance to the world :-)
>>>
>>>  I hope this helps a bit, please follow up and let me know if you have
>> further questions.
>>
>> --Rafael
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message