activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Date Mon, 27 Aug 2012 14:59:13 GMT
not at the moment, a temp queue always keeps messages in memory. I
know this has come up before and i think it is a sensible enhancement
to allow the pending message cursor to be configured for a temp
destination such that a temp destination need not be limited by
memory.
Can you raise a jira issue to track this enhancement so that it can
get scheduled.

On the temp store cleanup, that depends a bit in the journal size used
in the temp store, until a journal is fully unreferenced it can't be
removed so that may be why it is not immediate in your case. But a
test case should be able to demonstrate if there is a bug there.

On 27 August 2012 13:36, Martin C. <martinc@gmx.at> wrote:
> Hi,
>
> after some experiments, it seems that maybe one part of the problem is that
> temp-queue messages are not spooled to disk. Is there a possibility to tell
> ActiveMQ 5.5 to spool them to the temp store as well?
>
> Another issue I noticed, after purging a queue with non-persistent
> messages, the space in the temp-store seems to not be freed.
>
> Best regards,
> Martin
>
> On Mon, Aug 27, 2012 at 1:05 PM, Martin C. <martinc@gmx.at> wrote:
>
>> Hi,
>>
>> in one of the instances I didn't find anything in the broker log, just in
>> the client log, in the second log I found the same log entry in both client
>> and server:
>>
>> javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached.
>> Stopping producer (ID:hostname:26:33:1) to prevent flooding
>> temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0
>>
>> What would you suggest to use as a persistent <pendingQueuePolicy/> in my
>> case, if I'd just want messages to go to disk if the memory-usage is
>> approached?
>>
>> Thanks in advance!
>>
>> Best regards,
>> Martin
>>
>> On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <gary.tully@gmail.com> wrote:
>>
>>> what is out put to the broker log when it stops processing? It should
>>> report the limit that is reached.
>>>
>>> The pendingQueueCursor will stop caching messages when the systemusage
>>> limit reaches 70%, but it won't spool to disk. The persistent messages
>>> will just remain in the store till they are needed.
>>>
>>> On 27 August 2012 10:59, Martin C. <martinc@gmx.at> wrote:
>>> > Hi Gary,
>>> >
>>> > sorry for getting back to this rather old issue, but I just got an
>>> issue in
>>> > the field (twice) with the configuration below. The broker stopped
>>> > accepting messages after it reached the 64MB memory limit, instead of
>>> > spooling them out to disk. It basically is the previously discussed
>>> > configuration with the <pendingQueuePolicy /> section removed.
>>> >
>>> > After increasing the MemoryLimit of the broker via JMX, the messages got
>>> > processed again. The temp-store usage was at 0, the persistent store
>>> usage
>>> > at 11%. So, from my understanding, the broker should not have stopped
>>> > accepting messages. Any hints? Or should I fall back to using
>>> > <fileQueueCursor /> again?
>>> >
>>> > I am still on 5.5.0 and cannot upgrade that easily.
>>> >
>>> > Thanks in advance!
>>> >
>>> > Best regards,
>>> > Martin
>>> >
>>> > The relevant parts of the configuration again.
>>> >
>>> > <broker xmlns="http://activemq.apache.org/schema/core"
>>> > brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> >     populateJMSXUserID="true" useJmx="true">
>>> >
>>> >     <destinationPolicy>
>>> >       <policyMap>
>>> >         <policyEntries>
>>> >           <policyEntry topic=">" producerFlowControl="false"
>>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>>> >               <vmCursor />
>>> >           </policyEntry>
>>> >           <policyEntry queue=">" producerFlowControl="false"
>>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>>> >             <deadLetterStrategy>
>>> >               <individualDeadLetterStrategy queuePrefix="DLQ."
>>> > useQueueForQueueMessages="true"
>>> >                 processExpired="true" processNonPersistent="false" />
>>> >             </deadLetterStrategy>
>>> >           </policyEntry>
>>> >         </policyEntries>
>>> >       </policyMap>
>>> >     </destinationPolicy>
>>> >
>>> >     <persistenceAdapter>
>>> >     <kahaDB directory="${activemq.base}/data/kahadb"
>>> >          checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>>> >          concurrentStoreAndDispatchTopics="false"
>>> >          concurrentStoreAndDispatchQueues="false"
>>> >          />
>>> >     </persistenceAdapter>
>>> >
>>> >     <systemUsage>
>>> >       <systemUsage sendFailIfNoSpace="true">
>>> >         <memoryUsage>
>>> >           <memoryUsage limit="64 mb" />
>>> >         </memoryUsage>
>>> >         <storeUsage>
>>> >           <storeUsage limit="10 gb" />
>>> >         </storeUsage>
>>> >         <tempUsage>
>>> >           <tempUsage limit="1 gb" />
>>> >         </tempUsage>
>>> >       </systemUsage>
>>> >     </systemUsage>
>>> >   </broker>
>>> >
>>> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <gary.tully@gmail.com>
>>> wrote:
>>> >
>>> >> yes.
>>> >>
>>> >> On 16 November 2011 12:01, Martin C. <martinc@gmx.at> wrote:
>>> >> > Hi Gary,
>>> >> >
>>> >> > thanks for the very fast response. I will change the configuration
>>> >> > accordingly. So basically you say I should remove the
>>> >> > <pendingQueuePolicy> as a whole?
>>> >> >
>>> >> > Best regards,
>>> >> > Martin
>>> >> >
>>> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <gary.tully@gmail.com>
>>> >> wrote:
>>> >> >> You should not be using the file pending message cursor in
this
>>> case.
>>> >> >> The default store cursor is best, it will use a file cursor
for non
>>> >> >> persistent messages and leave excess (exceed the cache) persistent
>>> >> >> messages in the store.
>>> >> >> When it is specified as the default cursor in configuration,
it is
>>> >> >> used for both persistent and non persistent messages.
>>> >> >>
>>> >> >> The file pending message cursor replays all messages on startup
such
>>> >> >> that it can do an append to the cursor on the next add but
it is
>>> only
>>> >> >> intended for really slow stores, where reading from the local
file
>>> >> >> system is faster than reading from the store. This is not the
case
>>> >> >> with kahaDB.
>>> >> >>
>>> >> >> On 16 November 2011 10:26, Martin C. <martinc@gmx.at>
wrote:
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> I a have a question regarding fileQueueCursor and systemUsage
on
>>> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits
the
>>> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>>> >> >>>
>>> >> >>> I have a KahaDB with about 300000 messages, using 330MB
in the
>>> kahadb
>>> >> >>> directory, all messages are persistent. If I start ActiveMQ,
I see
>>> the
>>> >> >>> tmp_storage directory grow to 1GB, where it finally blocks
and
>>> >> >>> recovery of the messages starts hanging:
>>> >> >>>
>>> >> >>> [...]
>>> >> >>> DEBUG | default:memory: usage change from: 2% of available
memory,
>>> to:
>>> >> >>> 1% of available memory
>>> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory:
usage
>>> >> >>> change from: 1% of available memory, to: 0% of available
memory
>>> >> >>> DEBUG | default:memory: usage change from: 1% of available
memory,
>>> to:
>>> >> >>> 0% of available memory
>>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
50000
>>> >> >>> messages. 15% complete
>>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
100000
>>> >> >>> messages. 31% complete
>>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
150000
>>> >> >>> messages. 47% complete
>>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
200000
>>> >> >>> messages. 62% complete
>>> >> >>>
>>> >> >>> The memory usage goes up to around 75%, where it starts
dropping
>>> down
>>> >> >>> to the 0% from the logfile above. This is where it keeps
hanging,
>>> with
>>> >> >>> the tmp_storage directory at 1GB diskspace.
>>> >> >>>
>>> >> >>> If I increase the tempUsage to 10GB, the startup works
fine
>>> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you
explain to
>>> me,
>>> >> >>> why I need to increase tempUsage in order to restart my
broker
>>> with a
>>> >> >>> lot of persistent messages left? As far as I understood
the
>>> tempUsage
>>> >> >>> is for storing non-persistent messages, persistent messages
should
>>> be
>>> >> >>> affected by storeUsage?
>>> >> >>>
>>> >> >>> Thanks in advance!
>>> >> >>>
>>> >> >>> Best regards,
>>> >> >>> Martin
>>> >> >>>
>>> >> >>> PS: the relevant parts from my configuration
>>> >> >>>
>>> >> >>>  <broker xmlns="http://activemq.apache.org/schema/core"
>>> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> >> >>>    populateJMSXUserID="true" useJmx="true">
>>> >> >>>
>>> >> >>>    <destinationPolicy>
>>> >> >>>      <policyMap>
>>> >> >>>        <policyEntries>
>>> >> >>>          <policyEntry topic=">" producerFlowControl="false"
>>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>>> >> >>>              <vmCursor />
>>> >> >>>          </policyEntry>
>>> >> >>>          <policyEntry queue=">" producerFlowControl="false"
>>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>>> >> >>>            <deadLetterStrategy>
>>> >> >>>              <individualDeadLetterStrategy queuePrefix="DLQ."
>>> >> >>> useQueueForQueueMessages="true"
>>> >> >>>                processExpired="true" processNonPersistent="false"
>>> />
>>> >> >>>            </deadLetterStrategy>
>>> >> >>>            <pendingQueuePolicy>
>>> >> >>>              <fileQueueCursor />
>>> >> >>>            </pendingQueuePolicy>
>>> >> >>>          </policyEntry>
>>> >> >>>        </policyEntries>
>>> >> >>>      </policyMap>
>>> >> >>>    </destinationPolicy>
>>> >> >>>
>>> >> >>>    <persistenceAdapter>
>>> >> >>>    <kahaDB directory="${activemq.base}/data/kahadb"
>>> >> >>>         checkForCorruptJournalFiles="true"
>>> checksumJournalFiles="true"
>>> >> >>>         concurrentStoreAndDispatchTopics="false"
>>> >> >>>         concurrentStoreAndDispatchQueues="false"
>>> >> >>>         />
>>> >> >>>    </persistenceAdapter>
>>> >> >>>
>>> >> >>>    <systemUsage>
>>> >> >>>      <systemUsage sendFailIfNoSpace="true">
>>> >> >>>        <memoryUsage>
>>> >> >>>          <memoryUsage limit="64 mb" />
>>> >> >>>        </memoryUsage>
>>> >> >>>        <storeUsage>
>>> >> >>>          <storeUsage limit="10 gb" />
>>> >> >>>        </storeUsage>
>>> >> >>>        <tempUsage>
>>> >> >>>          <tempUsage limit="1 gb" />
>>> >> >>>        </tempUsage>
>>> >> >>>      </systemUsage>
>>> >> >>>    </systemUsage>
>>> >> >>>  </broker>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> http://fusesource.com
>>> >> >> http://blog.garytully.com
>>> >> >>
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> http://fusesource.com
>>> >> http://blog.garytully.com
>>> >>
>>>
>>>
>>>
>>> --
>>> http://fusesource.com
>>> http://blog.garytully.com
>>>
>>
>>



-- 
http://fusesource.com
http://blog.garytully.com

Mime
View raw message