qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajith Attapattu <rajit...@gmail.com>
Subject Re: proposal to remove certain features from qpidd
Date Mon, 23 Jul 2012 13:27:51 GMT
Jakub,

I wonder if producer flow control can help here.
If implemented properly this should (at least theoretically) prevent
the broker from going out of memory due to queue growth.
As you correctly point out, flow-2-disk just postpones it at best, in
addition to the fact that it has a serious impact on perf.

While there might be an impact on perf with producer flow control, I'm
sure it's way better than flow-2-disk.

Regards,

Rajith

On Mon, Jul 23, 2012 at 7:40 AM, Jakub Scholz <jakub@scholz.cz> wrote:
> Yes, the use of flow-to-disk queues unfortunately doesn't solve the
> memory issue on 100%. It just decreases the memory consumption, so the
> point when the broker runs out of memory is postponed a bit.
>
> Regards
> Jakub
>
> On Mon, Jul 23, 2012 at 11:09 AM, Gordon Sim <gsim@redhat.com> wrote:
>> On 07/22/2012 09:31 PM, Jakub Scholz wrote:
>>>
>>> We expect the brokers to deliver approximately hundreds of GB of
>>> messages per day. Under normal circumstances, most of the messages
>>> will be consumed by the clients almost immediately, but in some
>>> exceptional situations, they may need to be stored on the broker. And
>>> since the performance isn't the biggest issue for us (while on the
>>> other hand reliability of the broker is), the flow-to-disk queues kind
>>> of help (yes, the headers are still kept in memory).
>>
>>
>> My point was that in such cases, the memory consumed by the queue is not in
>> any way bounded. It will keep growing as the messages pile up. If the
>> content is a significant part of that (i.e. if the messages are relatively
>> large), then flow to disk can at least slow the growth. I assume that slowed
>> growth is useful in your case?
>>
>>
>>> Although you can always say that we can get HW with more memory or
>>> split the load between multiple brokers if necessary, it would still
>>> be better if the flow-to-disk queues are replaced by some better
>>> alternative.
>>
>>
>> I certainly agree a better solution is needed. One where the memory required
>> for such queues can be bounded in a reliable fashion, regardless of their
>> size.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message