qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toralf Lund <toralf.l...@pgs.com>
Subject Re: [External] Re: qpid::messaging::Sender::send() sometimes takes much longer than expected (in C++)
Date Thu, 10 Dec 2020 09:36:02 GMT
On 09/12/2020 22:55, Gordon Sim wrote:
> On 09/12/2020 18:36, Toralf Lund wrote:
>> On 09/12/2020 19:16, Gordon Sim wrote:
>>> On 09/12/2020 13:35, Toralf Lund wrote:
>>>> My question is simply, what might cause the send to take several 
>>>> seconds? I can't see any reason why the communication itself should 
>>>> be that slow; the software runs on a machine that's connected to 
>>>> the same network switch as the broker. The systems do have a 
>>>> certain load, but I didn't think it would be able to cause anything 
>>>> like this. And shouldn't threading etc. make sure something gets 
>>>> done within a reasonable time in any case? What am I missing?
>>>
>>> One possibility might be lack of capacity (e.g. if the previous 100 
>>> messages have not yet been acknowledged).
>>
>> Maybe. But I still don't get why it would take as long as 6 or 7 
>> seconds.
>>
>> I have a test app that sends messages through a sender with 0 
>> capacity, with a certain delay between transfers, and send() then 
>> generally takes less than 1 ms.
>
> Does that test app publish to the same queue(s)?

I've run some quick tests that way, and didn't see any delays. This is 
something that's kind of hard to do do, though, as I'm seeing the 
problem on a system used "in production" at a remote location. I'm 
struggling to reproduce on a test installation.

> Does the queue depth on the queue(s) that the slow sends occur on ever 
> get large? (Is there some limit set on the queue depth?)

The queues are last-value queues sorted on the "subject" values. They 
don't grow large, but obviously, some house-keeping is involved when 
storing messages.

This leads to another question: When exactly does the broker acknowledge 
message reception? Will it actually update queues first? I mean, it 
might conceivably read the message, buffer it, acknowledge, then do the 
routing.

>
> The only other thing I can think of that would cause the invocation of 
> send() to take a long time would be some sort of locking issue. Are 
> the multiple threads possibly sending concurrently?

There is another thread linked to QPid, but it takes care of 
"request-response" messaging triggered by user actions. I can't see how 
it would be active in all cases where I see delays.

But obviously, it seems quite likely that this is the case of a thread 
waiting for *something*. Is there anything else that may somehow cause a 
block? Note that my application has extra threads that do unrelated 
work. Also, I expect other threads started by the QPid library will be 
active, as there is a receiver with a somewhat large capacity linked to 
the same session. And my request-response thread will probably be 
listening for incoming messages, too.

I should perhaps have mentioned that there is another problem that I 
think may be related: It looks like heartbeat messages are also 
sometimes delayed in a similar manner, unless they are lost completely. 
Every few hours, I get a reconnect attempt during send or receive, even 
though there is probably nothing wrong with the connection. That leads 
to a "session busy" exception.

>
> How long does the slow sending usually last?

What's shown in the traces sent earlier is fairly typical. I think it's 
in the 5-10 s range in nearly all cases.

- Toralf

>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message