qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gordon Sim <g...@redhat.com>
Subject Re: Journal with LVQ
Date Tue, 15 Feb 2011 12:01:54 GMT
On 02/08/2011 10:02 AM, Bruno Matos wrote:
>
> On 2011/01/19, at 14:46, Gordon Sim wrote:
>
>> On 01/18/2011 03:31 PM, Bruno Matos wrote:
>>>
>>> On 2011/01/12, at 17:33, Gordon Sim wrote:
>>>
>>>> On 01/11/2011 04:30 PM, Bruno Matos wrote:
>>>>> I'm publishing messages to a durable LVQ, when I run the publisher for
>>>>> the first time the journal stays almost full, then I send the exact
>>>>> same
>>>>> messages, expecting the old ones to be replaced, but after some
>>>>> time the
>>>>> connection is closed with "Enqueue capacity threshold exceeded on
>>>>> queue". With qpid-tool I can see that msgDepth and byteDepth are
>>>>> exactly
>>>>> the same before and after the second run. Is this the expected
>>>>> behavior?
>>>>
>>>> Depends on what you mean by 'expected' :-)
>>>>
>>>> It is certainly the case that the linux journal will reach capacity
>>>> with different queue depths, depending on exact sequences of changes.
>>>> i.e. the capacity is not simply a function of the final state, but
>>>> also of the path to get there. In the second case there would be
>>>> additional dequeue records and the file boundaries might not align the
>>>> same way etc, so the journal space can be used up for fewer enqueued
>>>> messages.
>>>
>>>
>>> Thank you Gordon.
>>> I think I can say that's the expected to me... :) Although I need to
>>> know what journal size is effectively in use, can be a percentage, is
>>> that possible? I have a stored queue for browsing, like a dictionary,
>>> that is updated every night, and I want to reset it only if it reaches
>>> to a given percentage of journal size.
>>
>> Not sure if I understand the question correctly, but there is no
>> simple way of determining the minimum journal size needed to guarantee
>> a specific queue depth. The rule I use is to size the journal much
>> larger than I think I need (e.g. 2x the queue depth). Kim has some
>> text somewhere on the calculations involved I believe.
>>
>> Its also worth then setting a queue policy on the queue with the depth
>> fixed (and significantly less than the journal capacity). That way you
>> get a cleaner error when you hit the limit.
>>
>
> Sorry the delay.
>
> I was asking if there is a way to know the used size of the journal in a
> given time. When I run my publisher for the first time, and in qpid-stat
> the queue size 2m in bytes, how can I know the effectively used journal?

The calculation is quite complex and there tend to be lots of variables 
(exact timings of dequeues v. enqueues, relative messages sizes etc). in 
practical terms I have not found a way to accurately estimate the 
journal size required and so always greatly inflate it. It is an area I 
think we need to make simpler in some way.

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message