activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diego Rodríguez Martín <drodrig...@altiria.com>
Subject Re: Journal files. Memory management and cursors
Date Wed, 29 Apr 2009 08:41:44 GMT

Hi all,

    I have done some more test and now seems to me that the problem has 
nothing to do with journal files but with journal files in cursor storage.

    My previous thoughts were wrong as messages are committed into the 
database every 5 minutes or as soon as the cursor memory reaches 70% of 
broker max memory, so journal files can be reused (at least if total 
ammount of messages before they are written don't oversize the journal 
files size)

    So, if the exception is not thrown because of the journal files, it 
has to come from another place. I have played with memory configuration 
and this are my conclusions at this moment:

    First, my config. I am using one queue with 1 producer (Servlet 
inside Tomcat with an ActiveMQ PooledConnectionFactory with 5 conns, so 
there can be more than one thread producing) and 10 consumers (Spring 
DefaultMessageListener). Consumers are slower than producer, so the 
queue size increase during peak periods. Reliability is really important 
for us and we don't want to lose a single message. We also don't want to 
activate producer flow control, in case the slowness can exhaust Tomcat 
thread pool and lose messages.

    Then the test begins with JMeter massive post to producer (30 
threads indefinitly). What I expected to happen is that everytime the 
cursor memory size reaches 70% of total memory (5 Mb for test purposes 
to have the result in a few seconds, not production config), messages 
were persisted in database releasing the memory they were consuming, so 
new messages can take their place.

    The real result: when the cursor memory reaches 70% of memory, 
ActiveMQ throws the mentioned exception:
   
27 abr 2009 12:53:07,734 ERROR [ActiveMQ Journal Checkpoint Worker] 
org.apache.activemq.store.journal.JournalPersistenceAdapter  - Failed to 
mark the Journal: 
org.apache.activeio.journal.InvalidRecordLocationException: The location 
is less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location 
is less than the last mark.
    at 
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
    at 
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:416)
    at 
org.apache.activemq.store.journal.JournalPersistenceAdapter$1.iterate(JournalPersistenceAdapter.java:121)
    at 
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
    at 
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
27 abr 2009 12:53:07,734 DEBUG [ActiveMQ Journal Checkpoint Worker] 
org.apache.activemq.store.journal.JournalPersistenceAdapter  - 
Checkpoint done.

    Then the memory stays between 70% and 80%. Despite the exception, no 
messages are lost in their way to the consumers, but as the checkpoint 
fails, no memory is released and the performace of consumers falls from 
45 messages/seconds to 2-3 messages/seconds

    Even more, if there were messages in the queue when I started the 
consumer, the memory reached inmediatly 70% of max memory throwing the 
exception. This gave a clue: prefetch size of consumer. ActiveMQ tries 
to prefetch messages from the queue in database and inmediatly fills the 
memory as its default value (1000) is greater than the memory can afford 
(70% of 5Mb).

    Then I set prefetch size to 100 messages. The memory, even with a 
lot of messages already in the database when starting, is less than 70% 
so everything is ok at startup. When I start Jmeter again, cursor memory 
size again starts to increase and when it reaches 70% the same exception 
is thrown.
   
    Reading the docs about memory cursors, it says that "The default 
message cursor type in ActiveMQ 5.0 is Store based" and "ActiveMQ 5.0.0, 
there is a new memory model that allows messages to be paged in from 
storage when space is available (using Store cursors for persistent 
messages).".
   
    Now it makes sense de exception, as there is a problem storing 
memory cursors. I suppose they are stored in same Journal Store that is 
used for storing messages and transactions with journaledJDBC 
persistence adapter.
   
    The final test is changing message cursor type to file queue cursor
   
    <pendingQueuePolicy>
        <fileQueueCursor />
    </pendingQueuePolicy>

   Now when memory cursor reaches 70% of memory data is written to 
"data\localhost\tmp_storage", and the cursor memory is released so 
performance is not affected.
  
   My questions now are:
  
   Are my suppositions ok?

   Is this a bug? Should I open a JIRA issue?
  
   my options now is to activate producer flow control for the queue so 
cursor memory limits are not reached or use file queue cursor. I prefer 
not to use producer flow control. File queue cursor is using Kaha 
storage, is this ready for production enviroment where reliability is 
its main requirement?
  
    If you need some more info of my config feel free to ask.

   Thank you very much for your help
   
       Diego


Diego Rodríguez Martín escribió:
> Hi again,
>
>    I'm really stuck on this as I need a response to set my production 
> config.
>
>    Is there any extra data I can give you to narrow the problem?
>      Many thanks
>
>       Diego
>
> Diego Rodríguez Martín escribió:
>> Hi all,
>>
>>    I am planning to use ActiveMQ 5.2.0 in a project and I have made 
>> some tests in a windows box to understand how the journal files work.
>>
>>    I am defining a Journaled JDBC with postgres for persistence this 
>> way.
>>
>>        <persistenceAdapter>
>>            <journaledJDBC journalLogFiles="5" 
>> dataDirectory="${activemq.base}/data" dataSource="#postgres-ds"/>
>>        </persistenceAdapter>
>>
>>    Reading the docs, it seems that the journal files are used for 
>> transaction log, and you can define how much files do you want and 
>> the size of the files. ActiveMQ does not create new files, so it 
>> reuses the old files. This is OK, but I was wondering what happens if 
>> the queues (I am going to use only 2 queues) size increase long 
>> enough to use all the files in the journal and need more space. It is 
>> not a weird scenario, as we are decoupling consumer and producer and 
>> consumer can sometines be offline due to maintenance routines. The 
>> result is that journal files get overwritten, and many transactions 
>> has been lost. When I start the consumer, it begins to consume 
>> messages, but the JournalPersistenceAdapater fails everytime it has 
>> to execute the checkpoint code:
>>
>> 2009-04-16 18:48:03,140 [eckpoint Worker] ERROR 
>> JournalPersistenceAdapter      - Failed to mark the Journal: 
>> org.apache.activeio.journal.InvalidRecordLocationException: The 
>> location is less than the last mark.
>> org.apache.activeio.journal.InvalidRecordLocationException: The 
>> location is less than the last mark.
>>    at 
>> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340) 
>>
>>    at 
>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:416)

>>
>>    at 
>> org.apache.activemq.store.journal.JournalPersistenceAdapter$1.iterate(JournalPersistenceAdapter.java:121)

>>
>>    at 
>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)

>>
>>    at 
>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

>>
>>
>>
>> I think the files are corrupted, can you confirm this?
>> Is the exception related to the problem described?
>> It is an expected behaviour? Should the maximum number of messages in 
>> a queue be considered when configuring the journal?
>>
>> Thank you very much for your help.
>>
>

-- 
-------------------------------------------------------------
Diego Rodríguez Martín (drodriguez@altiria.com)
ALTIRIA TIC - Servicios SMS - Desarrollo Web
http://www.altiria.com
-------------------------------------------------------------


Mime
View raw message