activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Journal files. Memory management and cursors
Date Wed, 29 Apr 2009 09:38:57 GMT
the fileQueueCursor should work well for you but the error with the journal
looks like an issue that requires a jira entry. It should be better able to
deal with wrap around.
If you have time to produce a simple test case that demonstrates this usage
pattern and attach it to a new jira, it would be really helpful towards
producing a fast resolution.


2009/4/29 Diego Rodríguez Martín <drodriguez@altiria.com>

>
> Hi all,
>
>   I have done some more test and now seems to me that the problem has
> nothing to do with journal files but with journal files in cursor storage.
>
>   My previous thoughts were wrong as messages are committed into the
> database every 5 minutes or as soon as the cursor memory reaches 70% of
> broker max memory, so journal files can be reused (at least if total ammount
> of messages before they are written don't oversize the journal files size)
>
>   So, if the exception is not thrown because of the journal files, it has
> to come from another place. I have played with memory configuration and this
> are my conclusions at this moment:
>
>   First, my config. I am using one queue with 1 producer (Servlet inside
> Tomcat with an ActiveMQ PooledConnectionFactory with 5 conns, so there can
> be more than one thread producing) and 10 consumers (Spring
> DefaultMessageListener). Consumers are slower than producer, so the queue
> size increase during peak periods. Reliability is really important for us
> and we don't want to lose a single message. We also don't want to activate
> producer flow control, in case the slowness can exhaust Tomcat thread pool
> and lose messages.
>
>   Then the test begins with JMeter massive post to producer (30 threads
> indefinitly). What I expected to happen is that everytime the cursor memory
> size reaches 70% of total memory (5 Mb for test purposes to have the result
> in a few seconds, not production config), messages were persisted in
> database releasing the memory they were consuming, so new messages can take
> their place.
>
>   The real result: when the cursor memory reaches 70% of memory, ActiveMQ
> throws the mentioned exception:
>  27 abr 2009 12:53:07,734 ERROR [ActiveMQ Journal Checkpoint Worker]
> org.apache.activemq.store.journal.JournalPersistenceAdapter  - Failed to
> mark the Journal:
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
> org.apache.activeio.journal.InvalidRecordLocationException: The location is
> less than the last mark.
>   at
> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>   at
> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:416)
>   at
> org.apache.activemq.store.journal.JournalPersistenceAdapter$1.iterate(JournalPersistenceAdapter.java:121)
>   at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>   at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> 27 abr 2009 12:53:07,734 DEBUG [ActiveMQ Journal Checkpoint Worker]
> org.apache.activemq.store.journal.JournalPersistenceAdapter  - Checkpoint
> done.
>
>   Then the memory stays between 70% and 80%. Despite the exception, no
> messages are lost in their way to the consumers, but as the checkpoint
> fails, no memory is released and the performace of consumers falls from 45
> messages/seconds to 2-3 messages/seconds
>
>   Even more, if there were messages in the queue when I started the
> consumer, the memory reached inmediatly 70% of max memory throwing the
> exception. This gave a clue: prefetch size of consumer. ActiveMQ tries to
> prefetch messages from the queue in database and inmediatly fills the memory
> as its default value (1000) is greater than the memory can afford (70% of
> 5Mb).
>
>   Then I set prefetch size to 100 messages. The memory, even with a lot of
> messages already in the database when starting, is less than 70% so
> everything is ok at startup. When I start Jmeter again, cursor memory size
> again starts to increase and when it reaches 70% the same exception is
> thrown.
>     Reading the docs about memory cursors, it says that "The default
> message cursor type in ActiveMQ 5.0 is Store based" and "ActiveMQ 5.0.0,
> there is a new memory model that allows messages to be paged in from storage
> when space is available (using Store cursors for persistent messages).".
>     Now it makes sense de exception, as there is a problem storing memory
> cursors. I suppose they are stored in same Journal Store that is used for
> storing messages and transactions with journaledJDBC persistence adapter.
>     The final test is changing message cursor type to file queue cursor
>     <pendingQueuePolicy>
>       <fileQueueCursor />
>   </pendingQueuePolicy>
>
>  Now when memory cursor reaches 70% of memory data is written to
> "data\localhost\tmp_storage", and the cursor memory is released so
> performance is not affected.
>    My questions now are:
>    Are my suppositions ok?
>
>  Is this a bug? Should I open a JIRA issue?
>    my options now is to activate producer flow control for the queue so
> cursor memory limits are not reached or use file queue cursor. I prefer not
> to use producer flow control. File queue cursor is using Kaha storage, is
> this ready for production enviroment where reliability is its main
> requirement?
>     If you need some more info of my config feel free to ask.
>
>  Thank you very much for your help
>        Diego
>
>
> Diego Rodríguez Martín escribió:
>
>> Hi again,
>>
>>   I'm really stuck on this as I need a response to set my production
>> config.
>>
>>   Is there any extra data I can give you to narrow the problem?
>>     Many thanks
>>
>>      Diego
>>
>> Diego Rodríguez Martín escribió:
>>
>>> Hi all,
>>>
>>>   I am planning to use ActiveMQ 5.2.0 in a project and I have made some
>>> tests in a windows box to understand how the journal files work.
>>>
>>>   I am defining a Journaled JDBC with postgres for persistence this way.
>>>
>>>       <persistenceAdapter>
>>>           <journaledJDBC journalLogFiles="5"
>>> dataDirectory="${activemq.base}/data" dataSource="#postgres-ds"/>
>>>       </persistenceAdapter>
>>>
>>>   Reading the docs, it seems that the journal files are used for
>>> transaction log, and you can define how much files do you want and the size
>>> of the files. ActiveMQ does not create new files, so it reuses the old
>>> files. This is OK, but I was wondering what happens if the queues (I am
>>> going to use only 2 queues) size increase long enough to use all the files
>>> in the journal and need more space. It is not a weird scenario, as we are
>>> decoupling consumer and producer and consumer can sometines be offline due
>>> to maintenance routines. The result is that journal files get overwritten,
>>> and many transactions has been lost. When I start the consumer, it begins to
>>> consume messages, but the JournalPersistenceAdapater fails everytime it has
>>> to execute the checkpoint code:
>>>
>>> 2009-04-16 18:48:03,140 [eckpoint Worker] ERROR JournalPersistenceAdapter
>>>      - Failed to mark the Journal:
>>> org.apache.activeio.journal.InvalidRecordLocationException: The location is
>>> less than the last mark.
>>> org.apache.activeio.journal.InvalidRecordLocationException: The location
>>> is less than the last mark.
>>>   at
>>> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
>>>
>>>   at
>>> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:416)
>>>
>>>   at
>>> org.apache.activemq.store.journal.JournalPersistenceAdapter$1.iterate(JournalPersistenceAdapter.java:121)
>>>
>>>   at
>>> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>>>
>>>   at
>>> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
>>>
>>>
>>>
>>> I think the files are corrupted, can you confirm this?
>>> Is the exception related to the problem described?
>>> It is an expected behaviour? Should the maximum number of messages in a
>>> queue be considered when configuring the journal?
>>>
>>> Thank you very much for your help.
>>>
>>>
>>
> --
> -------------------------------------------------------------
> Diego Rodríguez Martín (drodriguez@altiria.com)
> ALTIRIA TIC - Servicios SMS - Desarrollo Web
> http://www.altiria.com
> -------------------------------------------------------------
>
>


-- 
http://blog.garytully.com

Open Source SOA
http://FUSESource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message