qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajith Attapattu <rajit...@gmail.com>
Subject Re: proposal to remove certain features from qpidd
Date Mon, 23 Jul 2012 18:32:29 GMT
Jakub,

I see the difficulties. Looks like it might not be that simple.

Regards,

Rajith

On Mon, Jul 23, 2012 at 11:35 AM, Jakub Scholz <jakub@scholz.cz> wrote:
> Hi Rajith,
>
> Most of the messages are delivered as a broadcasts from one producer
> to multiple receivers. And even when the queue of some receivers is
> full because they are not consuming, we still need to deliver the
> message for the rest of the receivers and at the same time to be aware
> who didn't received the message to eventually resend it later. I'm
> afraid that the producer flow control would not allow this so easily.
> We would probably need to change our producers to send the messages
> individually - N producers for N consumers. Only then we will be able
> to use the producer flow control ...
>
> Regards
> Jakub
>
> On Mon, Jul 23, 2012 at 3:27 PM, Rajith Attapattu <rajith77@gmail.com> wrote:
>> Jakub,
>>
>> I wonder if producer flow control can help here.
>> If implemented properly this should (at least theoretically) prevent
>> the broker from going out of memory due to queue growth.
>> As you correctly point out, flow-2-disk just postpones it at best, in
>> addition to the fact that it has a serious impact on perf.
>>
>> While there might be an impact on perf with producer flow control, I'm
>> sure it's way better than flow-2-disk.
>>
>> Regards,
>>
>> Rajith
>>
>> On Mon, Jul 23, 2012 at 7:40 AM, Jakub Scholz <jakub@scholz.cz> wrote:
>>> Yes, the use of flow-to-disk queues unfortunately doesn't solve the
>>> memory issue on 100%. It just decreases the memory consumption, so the
>>> point when the broker runs out of memory is postponed a bit.
>>>
>>> Regards
>>> Jakub
>>>
>>> On Mon, Jul 23, 2012 at 11:09 AM, Gordon Sim <gsim@redhat.com> wrote:
>>>> On 07/22/2012 09:31 PM, Jakub Scholz wrote:
>>>>>
>>>>> We expect the brokers to deliver approximately hundreds of GB of
>>>>> messages per day. Under normal circumstances, most of the messages
>>>>> will be consumed by the clients almost immediately, but in some
>>>>> exceptional situations, they may need to be stored on the broker. And
>>>>> since the performance isn't the biggest issue for us (while on the
>>>>> other hand reliability of the broker is), the flow-to-disk queues kind
>>>>> of help (yes, the headers are still kept in memory).
>>>>
>>>>
>>>> My point was that in such cases, the memory consumed by the queue is not
in
>>>> any way bounded. It will keep growing as the messages pile up. If the
>>>> content is a significant part of that (i.e. if the messages are relatively
>>>> large), then flow to disk can at least slow the growth. I assume that slowed
>>>> growth is useful in your case?
>>>>
>>>>
>>>>> Although you can always say that we can get HW with more memory or
>>>>> split the load between multiple brokers if necessary, it would still
>>>>> be better if the flow-to-disk queues are replaced by some better
>>>>> alternative.
>>>>
>>>>
>>>> I certainly agree a better solution is needed. One where the memory required
>>>> for such queues can be bounded in a reliable fashion, regardless of their
>>>> size.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message