activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Smith <joesmithc...@yahoo.com>
Subject Re: how can a producer detect that temp disk is full
Date Fri, 20 Jan 2012 20:15:19 GMT
Hi Gary,

Thanks for the link to test source file.  Finally got a chance to add a test method and ran
some tests.


Scenario 1:
1. Producer sends msgs
2. Pause 30 sec for Producer to send msgs

3. Consumer receives msgs but at a slower rate than Producer
4. Compare # msgs sent, received, and # of ResourceAllocationException received.

Result:
1. 1600 msgs were sent by Producer

2. 1499 msgs were received by Consumer

3.   101 ResourceAlloctionExcpetions were received by Producer on the onException() callback.

However, the disk remained full - space was not released.  When sending a 2nd batch of 1600
msgs, Producer received 1600 ResourceAllocationExcetions.


I wasn't sure if there are other settings that need to be satisfied to release the space or
the connection need to be closed...

Here is a snippet of the exception from producer2 sending:

2012-01-19 17:27:31,114 [127.0.0.1:50460] - WARN  Service                       
- Async error occurred: javax.jms.ResourceAllocationException: Usage Manager Temp Store is
Full (99% of 33554432). Stopping producer (ID:US137440.local-50458-1327011882287-1:2:2:1)
to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html
for more info
javax.jms.ResourceAllocationException: Usage Manager Temp Store is Full (99% of 33554432).
Stopping producer (ID:US137440.local-50458-1327011882287-1:2:2:1) to prevent flooding queue://testqueue.
See http://activemq.apache.org/producer-flow-control.html for more info
        at org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:579)
        at org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:573)
        at org.apache.activemq.broker.region.Queue.checkUsage(Queue.java:757)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:674)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:653)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)


Scenario 2:
- Same as 1, but I didn't start the consumer until Producer had sent all 1600 messages. 
Consumer read all messages, but the temp store remained at 100% usage - and not more msgs
could be sent.

I read somewhere the min allocation for temp store should be 32mb or greater.  I increased
the temp store limit to 38mb and ran w/increased number of msgs to max out the space.  Resulting
behavior is the same.

----------------------------

Also, when I used the same settings and ran against the downloaded binary broker, the behavior
seems to be differ.  Producer received ResourceAllocationException as expected, and it maxed
out temp store.  I then started a Consumer to read from the queue, the responses were:
1. Consumer received some messages but then stopped.

2. Consumer received no message at all.

Guess:

Once the temp store is maxed out, somehow the space was not freed up.  Subsequent connection/sends
were blocked while waiting for spaces.  Let me know if I mis-configured some options.


I am attaching the src code 
(*.java), full log (activemq-test.log) and the summary log w/o the 
error/warn trace in the zip attachment  src-log.zip.  I'm not sure if 
you will get it.  Let me know if I should  post it somewhere.


Thanks for any help.
Jason



________________________________
 From: Gary Tully <gary.tully@gmail.com>
To: users@activemq.apache.org; Joe Smith <joesmithcomm@yahoo.com> 
Sent: Thursday, January 19, 2012 8:39 AM
Subject: Re: how can a producer detect that temp disk is full
 
Simplest config is to use the default cursor, which is the store
cursor that has an embedded file cursor for non persistent messages.

To have the file cursor spool to disk, it needs to reach 70% of its
destination limit, so the system usage memory limits need to exceed
this value, which the appear to do in your config below: 20mb > 1mb

The spooling will continue till the temp usage limit is reached and
the resource exception is throws as a result of sendFailIfNoSpace.

I though there was a simple junit test case for this but it seems not,
I just added one to an existing test case, see:
http://svn.apache.org/viewvc?view=revision&revision=1233367

The new test is:
org.apache.activemq.bugs.TempStorageBlockedBrokerTest#testFillTempAndConsume

play around with that or compare to what you are doing and if you find
any problem use that test case as the basis for a bug report.

Note, the vm cursors do not spool to disk, so they will not use any
temp store (disk space)
Mime
View raw message