activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin C." <mart...@gmx.at>
Subject Re: ActiveMQ needs tempUsage for recovery of persistent messages on startup?
Date Mon, 27 Aug 2012 12:36:31 GMT
Hi,

after some experiments, it seems that maybe one part of the problem is that
temp-queue messages are not spooled to disk. Is there a possibility to tell
ActiveMQ 5.5 to spool them to the temp store as well?

Another issue I noticed, after purging a queue with non-persistent
messages, the space in the temp-store seems to not be freed.

Best regards,
Martin

On Mon, Aug 27, 2012 at 1:05 PM, Martin C. <martinc@gmx.at> wrote:

> Hi,
>
> in one of the instances I didn't find anything in the broker log, just in
> the client log, in the second log I found the same log entry in both client
> and server:
>
> javax.jms.ResourceAllocationException: Usage Manager Memory Limit reached.
> Stopping producer (ID:hostname:26:33:1) to prevent flooding
> temp-queue://e1413fd6-2e46-76d5-dc62-df3260914cac:0
>
> What would you suggest to use as a persistent <pendingQueuePolicy/> in my
> case, if I'd just want messages to go to disk if the memory-usage is
> approached?
>
> Thanks in advance!
>
> Best regards,
> Martin
>
> On Mon, Aug 27, 2012 at 12:25 PM, Gary Tully <gary.tully@gmail.com> wrote:
>
>> what is out put to the broker log when it stops processing? It should
>> report the limit that is reached.
>>
>> The pendingQueueCursor will stop caching messages when the systemusage
>> limit reaches 70%, but it won't spool to disk. The persistent messages
>> will just remain in the store till they are needed.
>>
>> On 27 August 2012 10:59, Martin C. <martinc@gmx.at> wrote:
>> > Hi Gary,
>> >
>> > sorry for getting back to this rather old issue, but I just got an
>> issue in
>> > the field (twice) with the configuration below. The broker stopped
>> > accepting messages after it reached the 64MB memory limit, instead of
>> > spooling them out to disk. It basically is the previously discussed
>> > configuration with the <pendingQueuePolicy /> section removed.
>> >
>> > After increasing the MemoryLimit of the broker via JMX, the messages got
>> > processed again. The temp-store usage was at 0, the persistent store
>> usage
>> > at 11%. So, from my understanding, the broker should not have stopped
>> > accepting messages. Any hints? Or should I fall back to using
>> > <fileQueueCursor /> again?
>> >
>> > I am still on 5.5.0 and cannot upgrade that easily.
>> >
>> > Thanks in advance!
>> >
>> > Best regards,
>> > Martin
>> >
>> > The relevant parts of the configuration again.
>> >
>> > <broker xmlns="http://activemq.apache.org/schema/core"
>> > brokerName="localhost" dataDirectory="${activemq.base}/data"
>> >     populateJMSXUserID="true" useJmx="true">
>> >
>> >     <destinationPolicy>
>> >       <policyMap>
>> >         <policyEntries>
>> >           <policyEntry topic=">" producerFlowControl="false"
>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> >               <vmCursor />
>> >           </policyEntry>
>> >           <policyEntry queue=">" producerFlowControl="false"
>> > gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> >             <deadLetterStrategy>
>> >               <individualDeadLetterStrategy queuePrefix="DLQ."
>> > useQueueForQueueMessages="true"
>> >                 processExpired="true" processNonPersistent="false" />
>> >             </deadLetterStrategy>
>> >           </policyEntry>
>> >         </policyEntries>
>> >       </policyMap>
>> >     </destinationPolicy>
>> >
>> >     <persistenceAdapter>
>> >     <kahaDB directory="${activemq.base}/data/kahadb"
>> >          checkForCorruptJournalFiles="true" checksumJournalFiles="true"
>> >          concurrentStoreAndDispatchTopics="false"
>> >          concurrentStoreAndDispatchQueues="false"
>> >          />
>> >     </persistenceAdapter>
>> >
>> >     <systemUsage>
>> >       <systemUsage sendFailIfNoSpace="true">
>> >         <memoryUsage>
>> >           <memoryUsage limit="64 mb" />
>> >         </memoryUsage>
>> >         <storeUsage>
>> >           <storeUsage limit="10 gb" />
>> >         </storeUsage>
>> >         <tempUsage>
>> >           <tempUsage limit="1 gb" />
>> >         </tempUsage>
>> >       </systemUsage>
>> >     </systemUsage>
>> >   </broker>
>> >
>> > On Wed, Nov 16, 2011 at 1:50 PM, Gary Tully <gary.tully@gmail.com>
>> wrote:
>> >
>> >> yes.
>> >>
>> >> On 16 November 2011 12:01, Martin C. <martinc@gmx.at> wrote:
>> >> > Hi Gary,
>> >> >
>> >> > thanks for the very fast response. I will change the configuration
>> >> > accordingly. So basically you say I should remove the
>> >> > <pendingQueuePolicy> as a whole?
>> >> >
>> >> > Best regards,
>> >> > Martin
>> >> >
>> >> > On Wed, Nov 16, 2011 at 11:34 AM, Gary Tully <gary.tully@gmail.com>
>> >> wrote:
>> >> >> You should not be using the file pending message cursor in this
>> case.
>> >> >> The default store cursor is best, it will use a file cursor for
non
>> >> >> persistent messages and leave excess (exceed the cache) persistent
>> >> >> messages in the store.
>> >> >> When it is specified as the default cursor in configuration, it
is
>> >> >> used for both persistent and non persistent messages.
>> >> >>
>> >> >> The file pending message cursor replays all messages on startup
such
>> >> >> that it can do an append to the cursor on the next add but it is
>> only
>> >> >> intended for really slow stores, where reading from the local file
>> >> >> system is faster than reading from the store. This is not the case
>> >> >> with kahaDB.
>> >> >>
>> >> >> On 16 November 2011 10:26, Martin C. <martinc@gmx.at> wrote:
>> >> >>> Hi,
>> >> >>>
>> >> >>> I a have a question regarding fileQueueCursor and systemUsage
on
>> >> >>> ActiveMQ 5.5.0: Below you see my configuration, which limits
the
>> >> >>> memory to 64MB, store usage to 10GB and tempUsage to 1GB.
>> >> >>>
>> >> >>> I have a KahaDB with about 300000 messages, using 330MB in
the
>> kahadb
>> >> >>> directory, all messages are persistent. If I start ActiveMQ,
I see
>> the
>> >> >>> tmp_storage directory grow to 1GB, where it finally blocks
and
>> >> >>> recovery of the messages starts hanging:
>> >> >>>
>> >> >>> [...]
>> >> >>> DEBUG | default:memory: usage change from: 2% of available
memory,
>> to:
>> >> >>> 1% of available memory
>> >> >>> DEBUG | default:memory:queue://local.queue.myqueue:memory:
usage
>> >> >>> change from: 1% of available memory, to: 0% of available memory
>> >> >>> DEBUG | default:memory: usage change from: 1% of available
memory,
>> to:
>> >> >>> 0% of available memory
>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
50000
>> >> >>> messages. 15% complete
>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
100000
>> >> >>> messages. 31% complete
>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
150000
>> >> >>> messages. 47% complete
>> >> >>>  INFO | cursor for queue://local.queue.myqueue has recovered
200000
>> >> >>> messages. 62% complete
>> >> >>>
>> >> >>> The memory usage goes up to around 75%, where it starts dropping
>> down
>> >> >>> to the 0% from the logfile above. This is where it keeps hanging,
>> with
>> >> >>> the tmp_storage directory at 1GB diskspace.
>> >> >>>
>> >> >>> If I increase the tempUsage to 10GB, the startup works fine
>> >> >>> (tmp_storage becomes about 1.4GB in this case). Can you explain
to
>> me,
>> >> >>> why I need to increase tempUsage in order to restart my broker
>> with a
>> >> >>> lot of persistent messages left? As far as I understood the
>> tempUsage
>> >> >>> is for storing non-persistent messages, persistent messages
should
>> be
>> >> >>> affected by storeUsage?
>> >> >>>
>> >> >>> Thanks in advance!
>> >> >>>
>> >> >>> Best regards,
>> >> >>> Martin
>> >> >>>
>> >> >>> PS: the relevant parts from my configuration
>> >> >>>
>> >> >>>  <broker xmlns="http://activemq.apache.org/schema/core"
>> >> >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> >> >>>    populateJMSXUserID="true" useJmx="true">
>> >> >>>
>> >> >>>    <destinationPolicy>
>> >> >>>      <policyMap>
>> >> >>>        <policyEntries>
>> >> >>>          <policyEntry topic=">" producerFlowControl="false"
>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="60000">
>> >> >>>              <vmCursor />
>> >> >>>          </policyEntry>
>> >> >>>          <policyEntry queue=">" producerFlowControl="false"
>> >> >>> gcInactiveDestinations="false" inactiveTimoutBeforeGC="86400000">
>> >> >>>            <deadLetterStrategy>
>> >> >>>              <individualDeadLetterStrategy queuePrefix="DLQ."
>> >> >>> useQueueForQueueMessages="true"
>> >> >>>                processExpired="true" processNonPersistent="false"
>> />
>> >> >>>            </deadLetterStrategy>
>> >> >>>            <pendingQueuePolicy>
>> >> >>>              <fileQueueCursor />
>> >> >>>            </pendingQueuePolicy>
>> >> >>>          </policyEntry>
>> >> >>>        </policyEntries>
>> >> >>>      </policyMap>
>> >> >>>    </destinationPolicy>
>> >> >>>
>> >> >>>    <persistenceAdapter>
>> >> >>>    <kahaDB directory="${activemq.base}/data/kahadb"
>> >> >>>         checkForCorruptJournalFiles="true"
>> checksumJournalFiles="true"
>> >> >>>         concurrentStoreAndDispatchTopics="false"
>> >> >>>         concurrentStoreAndDispatchQueues="false"
>> >> >>>         />
>> >> >>>    </persistenceAdapter>
>> >> >>>
>> >> >>>    <systemUsage>
>> >> >>>      <systemUsage sendFailIfNoSpace="true">
>> >> >>>        <memoryUsage>
>> >> >>>          <memoryUsage limit="64 mb" />
>> >> >>>        </memoryUsage>
>> >> >>>        <storeUsage>
>> >> >>>          <storeUsage limit="10 gb" />
>> >> >>>        </storeUsage>
>> >> >>>        <tempUsage>
>> >> >>>          <tempUsage limit="1 gb" />
>> >> >>>        </tempUsage>
>> >> >>>      </systemUsage>
>> >> >>>    </systemUsage>
>> >> >>>  </broker>
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> http://fusesource.com
>> >> >> http://blog.garytully.com
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> http://fusesource.com
>> >> http://blog.garytully.com
>> >>
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message