activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: out of memory using producer flow control and fileQueueCursor
Date Fri, 24 Feb 2012 12:36:24 GMT
>From a fix perspective, have a look at
org.apache.activemq.broker.region.IndirectMessageReference

it seems to be what is needed but I don't think it is currently
utilised to the full or fleshed out.

On 24 February 2012 11:52, Gary Tully <gary.tully@gmail.com> wrote:
> That seems to be a reasonable expectation, that transaction size is
> not limited to available memory.
> There was some work on trunk to have an in-memory send transaction
> overflow to a temp file but it looks
> like the pending massages cursor is a limitation for the receive side.
>
> If you can produce with a simple junit test case and open an issue, it
> would be a good start point for further investigation.
>
> Is the the behavior the same when u use the default store cursor? My
> guess is that it is but just incase.
>
> On 24 February 2012 00:47, mserrano <martin@attivio.com> wrote:
>> version 5.5.1
>> broker cfg:  memory: 256M, Store:10g, Swap/tmp:10g, persistence=true,
>> producerFlowControl=true
>>
>> I'm using the  http://activemq.apache.org/message-cursors.html
>> fileQueueCursor  destination policy for a queue.  It appears to properly
>> page in messages in the cursor without blocking when it gets to 70% memory
>> use.  The queue is being read by a transacted session consumer which is
>> keeping open the transaction for a many messages (>10k).  The messages
>> themselves are large (800k or so).  What I see is that the Queue
>> pagedInMessages data structure will grow without bound and contains a
>> reference to every message dispatched to the consumer.   Eventually the
>> broker will run out of memory.
>>
>> Producer flow control does not get triggered (but if it did my transaction
>> would not be able to get all the messages it needs and would never
>> complete).
>>
>> Is it just not possible to use ActiveMQ in this way?  What I mean by this is
>> that I'd like the total size of the number of messages * the size of the
>> messages of an open transaction to be bounded by the disk store limits not
>> by memory limits.  Since the messages are persisted, it seems like this
>> should be possible.
>>
>> Any suggestions for what I should do differently?  Would the
>> connection.setTransactedIndividualAck setting in 5.6 help here?
>>
>> I'm happy to work on fixes/tests for this but need some guidance.
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4415752.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com



-- 
http://fusesource.com
http://blog.garytully.com

Mime
View raw message