qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Journal with LVQ
Date Tue, 15 Feb 2011 14:08:24 GMT
On 02/15/2011 07:01 AM, Gordon Sim wrote:
> On 02/08/2011 10:02 AM, Bruno Matos wrote:
>>
>> On 2011/01/19, at 14:46, Gordon Sim wrote:
>>
>>> On 01/18/2011 03:31 PM, Bruno Matos wrote:
>>>>
>>>> On 2011/01/12, at 17:33, Gordon Sim wrote:
>>>>
>>>>> On 01/11/2011 04:30 PM, Bruno Matos wrote:
>>>>>> I'm publishing messages to a durable LVQ, when I run the publisher
for
>>>>>> the first time the journal stays almost full, then I send the exact
>>>>>> same
>>>>>> messages, expecting the old ones to be replaced, but after some
>>>>>> time the
>>>>>> connection is closed with "Enqueue capacity threshold exceeded on
>>>>>> queue". With qpid-tool I can see that msgDepth and byteDepth are
>>>>>> exactly
>>>>>> the same before and after the second run. Is this the expected
>>>>>> behavior?
>>>>>
>>>>> Depends on what you mean by 'expected' :-)
>>>>>
>>>>> It is certainly the case that the linux journal will reach capacity
>>>>> with different queue depths, depending on exact sequences of changes.
>>>>> i.e. the capacity is not simply a function of the final state, but
>>>>> also of the path to get there. In the second case there would be
>>>>> additional dequeue records and the file boundaries might not align the
>>>>> same way etc, so the journal space can be used up for fewer enqueued
>>>>> messages.
>>>>
>>>>
>>>> Thank you Gordon.
>>>> I think I can say that's the expected to me... :) Although I need to
>>>> know what journal size is effectively in use, can be a percentage, is
>>>> that possible? I have a stored queue for browsing, like a dictionary,
>>>> that is updated every night, and I want to reset it only if it reaches
>>>> to a given percentage of journal size.
>>>
>>> Not sure if I understand the question correctly, but there is no
>>> simple way of determining the minimum journal size needed to guarantee
>>> a specific queue depth. The rule I use is to size the journal much
>>> larger than I think I need (e.g. 2x the queue depth). Kim has some
>>> text somewhere on the calculations involved I believe.
>>>
>>> Its also worth then setting a queue policy on the queue with the depth
>>> fixed (and significantly less than the journal capacity). That way you
>>> get a cleaner error when you hit the limit.
>>>
>>
>> Sorry the delay.
>>
>> I was asking if there is a way to know the used size of the journal in a
>> given time. When I run my publisher for the first time, and in qpid-stat
>> the queue size 2m in bytes, how can I know the effectively used journal?
>
> The calculation is quite complex and there tend to be lots of variables (exact
> timings of dequeues v. enqueues, relative messages sizes etc). in practical
> terms I have not found a way to accurately estimate the journal size required
> and so always greatly inflate it. It is an area I think we need to make simpler
> in some way.
>

Perhaps it would be useful to add some management instrumentation to the 
journals, so it is at least easy to see how much space is actually used when you 
run a test (or indeed in production so you can see if you're getting close to 
the limit)

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message