activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: ActiveMQ-5.00-RC1 - Default AMQMessageStore dont remove old messages
Date Wed, 17 Oct 2007 19:26:39 GMT
Hi Marcio - this is fixed in trunk

On Oct 17, 2007, at 7:01 PM, Márcio Moraes wrote:

> Rob,
>
> May I open a issue about it?
>
> Att,
>
> Márcio Moraes
>
> 2007/10/15, Márcio Moraes <mmoraes.br@gmail.com>:
>>
>> Thanks,
>>
>> You right it does not solve my problem.
>>
>> I am think to workaround  it with a script to cleanup old data files.
>>
>> 2007/10/12, Rob Davies < rajdavies@gmail.com>:
>>>
>>> looks like it might be a bug  - though it doesn't solve your problem
>>> - which is another bug btw - will fix
>>>
>>>
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> http://rajdavies.blogspot.com/
>>>
>>>
>>>
>>> On Oct 12, 2007, at 12:05 AM, Márcio Moraes wrote:
>>>
>>>> I thought found the bug.
>>>>
>>>> --- DataFileAccessorPool.java ---
>>>>
>>>>         public void closeDataFileReader(DataFileAccessor reader) {
>>>>             openCounter--;
>>>>             used = true; // HERE
>>>>             if (pool.size() >= maxOpenReadersPerFile || disposed) {
>>>>                 reader.dispose();
>>>>             } else {
>>>>                 pool.add(reader);
>>>>             }
>>>>         }
>>>>
>>>> Att,
>>>>
>>>> Márcio Moraes
>>>>
>>>> 2007/10/11, Márcio Moraes <mmoraes.br@gmail.com>:
>>>>>
>>>>> I see ...
>>>>>
>>>>> The point is it seems not cleanup  until a restart.
>>>>>
>>>>> I put a log into DataFileAccessorPool.java to see the  
>>>>> disposeUnused
>>>>> method.
>>>>>
>>>>> Then i sent a lot of messages, until my data files look like this:
>>>>>
>>>>> ls -lh data/localhost/journal/
>>>>> total 50M
>>>>> -rw-r--r-- 1 root root  32M 2007-10-11 18:21 data-3
>>>>> -rw-r--r-- 1 root root  32M 2007-10-11 18:31 data-4
>>>>> -rw-r--r-- 1 root root 2.1K 2007-10-11 18:31 data-control
>>>>>
>>>>> and
>>>>>
>>>>> ls -lh data/localhost/kr-store/data/
>>>>> total 40M
>>>>> -rw-r--r-- 1 root root  949 2007-10-11 17:57 data-blob-1
>>>>> -rw-r--r-- 1 root root 3.3K 2007-10-11 17:59 data-kaha-1
>>>>> -rw-r--r-- 1 root root 3.2M 2007-10-11 18:31 data-topic-acks-1
>>>>> -rw-r--r-- 1 root root 6.1M 2007-10-11 18:31 data-topic-data-1
>>>>> -rw-r--r-- 1 root root  17M 2007-10-11 18:31 data-topic-subs-
>>>>> references-1
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:57
>>>>> hash-index-blob_topic___ActiveMQ.Agent-Subscriptions
>>>>> -rw-r--r-- 1 root root  33K 2007-10-11 17:57
>>>>> hash-index-blob_topic___auction.admin-Subscriptions
>>>>> -rw-r--r-- 1 root root  33K 2007-10-11 17:57
>>>>> hash-index-blob_topic___auction.negotiation-Subscriptions
>>>>> -rw-r--r-- 1 root root  17K 2007-10-11 17:57
>>>>> hash-index-blob_topic___auction.notification-Subscriptions
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:59
>>>>> hash-index-blob_topic___auction.response-Subscriptions
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:57
>>>>> hash-index-topic-data_topic___ActiveMQ.Agent
>>>>> -rw-r--r-- 1 root root  17M 2007-10-11 18:31
>>>>> hash-index-topic-data_topic___auction.admin
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:57
>>>>> hash-index-topic-data_topic___auction.negotiation
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:57
>>>>> hash-index-topic-data_topic___auction.notification
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:59
>>>>> hash-index-topic-data_topic___auction.response
>>>>> -rw-r--r-- 1 root root  510 2007-10-11 17:59 index-blob
>>>>> -rw-r--r-- 1 root root 1.1K 2007-10-11 17:59 index-kaha
>>>>> -rw-r--r-- 1 root root 2.4M 2007-10-11 18:31 index-topic-acks
>>>>> -rw-r--r-- 1 root root 2.4M 2007-10-11 18:31 index-topic-data
>>>>> -rw-r--r-- 1 root root 4.4M 2007-10-11 18:31 index-topic-subs-
>>>>> references
>>>>> -rw-r--r-- 1 root root    0 2007-10-11 17:57 lock
>>>>>
>>>>>
>>>>> I thought data-3 should be removed, am i right?
>>>>>
>>>>> But my log says:
>>>>>
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:32:11,381
>>>>> DataFileAccessorPool:112  - file:data-4 number = 4 , length =
>>>>> 20506094
>>>>> refCount = 89444 pool.isUsed():true
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:32:11,381
>>>>> DataFileAccessorPool:112  - file:data-3 number = 3 , length =
>>>>> 33554297
>>>>> refCount = 108283 pool.isUsed():true
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:32:41,385
>>>>> DataFileAccessorPool:112  - file:data-4 number = 4 , length =
>>>>> 21730205
>>>>> refCount = 94782 pool.isUsed():true
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:32:41,385
>>>>> DataFileAccessorPool:112  - file:data-3 number = 3 , length =
>>>>> 33554297
>>>>> refCount = 108283 pool.isUsed():true
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:33:11,382
>>>>> DataFileAccessorPool:112  - file:data-4 number = 4 , length =
>>>>> 22957070
>>>>> refCount = 100132 pool.isUsed():true
>>>>> [DEBUG][ActiveMQ Scheduler]2007-10-11 18:33:11,383
>>>>> DataFileAccessorPool:112  - file:data-3 number = 3 , length =
>>>>> 33554297
>>>>> refCount = 108283 pool.isUsed():true
>>>>>
>>>>> if pool.isUsed() is true nothing is cleanup.
>>>>>
>>>>> It seems that the file into the pool never get unused.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Att,
>>>>>
>>>>> Márcio Moraes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2007/10/11, Rob Davies < rajdavies@gmail.com>:
>>>>>>
>>>>>> that still looks ok - the default data file length is 32mb - I'll
>>>>>> expose the property to make that configurable
>>>>>>
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> http://rajdavies.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 11, 2007, at 8:56 PM, Márcio Moraes wrote:
>>>>>>
>>>>>>> I build a new version from SNAPSHOT (revision 583595) and run
>>>>>>> again my
>>>>>>> tests.
>>>>>>>
>>>>>>> After sent ~15k messages my data dir was 34mb
>>>>>>>
>>>>>>> I wait 10 minutes but data still with 34mb ...
>>>>>>>
>>>>>>> then i restart activemq and data dir was now with 20mb
>>>>>>>
>>>>>>> Is this a ok behavior? How log it takes to clean up? Always do
i
>>>>>>> need
>>>>>>> restart?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Márcio Moraes
>>>>>>>
>>>>>>> 2007/10/11, Márcio Moraes <mmoraes.br@gmail.com>:
>>>>>>>>
>>>>>>>> Ok, I will try and let you know ...
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> 2007/10/11, Rob Davies <rajdavies@gmail.com>:
>>>>>>>>>
>>>>>>>>> ah - in that case - please use a later snapshot - there
was  
>>>>>>>>> a bug
>>>>>>>>> with selectors which is now fixed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Rob
>>>>>>>>>
>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Oct 11, 2007, at 6:59 PM, Márcio Moraes wrote:
>>>>>>>>>
>>>>>>>>>> I think messages have been readed.
>>>>>>>>>>
>>>>>>>>>> Because i have 2 durable consumers with selector.
>>>>>>>>>>
>>>>>>>>>> The first consumer/selector accept the messages the
other one
>>>>>>>>>> does
>>>>>>>>>> not.
>>>>>>>>>>
>>>>>>>>>> I think message should be considered readed even
to the  
>>>>>>>>>> second
>>>>>>>>>> consumer.
>>>>>>>>>>
>>>>>>>>>> Am i right?
>>>>>>>>>>
>>>>>>>>>> Att,
>>>>>>>>>>
>>>>>>>>>> Márcio Moraes
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2007/10/11, Rob Davies < rajdavies@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> Hi Marcio,
>>>>>>>>>>>
>>>>>>>>>>> the messages are stored in a journal file log
- the whole  
>>>>>>>>>>> log
>>>>>>>>>>> will be
>>>>>>>>>>> removed when all the messages have been read.
This is  
>>>>>>>>>>> done for
>>>>>>>>>>> performance
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> cheers,
>>>>>>>>>>>
>>>>>>>>>>> Rob
>>>>>>>>>>>
>>>>>>>>>>> http://rajdavies.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Oct 11, 2007, at 2:59 PM, Márcio Moraes wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm testing ActiveMQ-5.00-RC1 under the following
scenario:
>>>>>>>>>>>>
>>>>>>>>>>>> - Sending N messages to a Durable Topic
>>>>>>>>>>>> - Two consumers identified with its clientId
>>>>>>>>>>>> - Both has a selector and only one can accept
the test
>>>>>>>>>>>> messages
>>>>>>>>>>>>
>>>>>>>>>>>> All works fine, except the messages are never
removed  
>>>>>>>>>>>> from the
>>>>>>>>>>>> data
>>>>>>>>>>>> directory, it grows infinitely.
>>>>>>>>>>>>
>>>>>>>>>>>> Is it a bug?
>>>>>>>>>>>>
>>>>>>>>>>>> My work around may be cron a script to clean
that.
>>>>>>>>>>>>
>>>>>>>>>>>> Att,
>>>>>>>>>>>>
>>>>>>>>>>>> Márcio Moraes
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>


Mime
View raw message