activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bonnyr <bon...@optusnet.com.au>
Subject Re: ConcurrentModificationException while closing consumer
Date Wed, 11 Feb 2009 08:00:27 GMT

Rob,

Like in a sliding window protocol, an acknowledged message acks all before
it (by the same consumer).
I guess I'm assuming either one consumer per session on the consuming client
side or some way to 
track this on a per session basis in case there are multiple consumers.

Cheers,

Bonny

rajdavies wrote:
> 
> just for clarification - do you mean all messages delivered to the the  
> MessageConsumer  would be acked  on WINDOW_ACK instead of Session wide ?
> 
> On 10 Feb 2009, at 23:42, bonnyr wrote:
> 
>>
>> Rob,
>>
>> Can I hijack this issue and bring back an issue I have filed in Jira  
>> and is
>> assigned to you (AMQ-1916) and
>> that is the issue of a new ack mode (I called it WINDOW_ACK) that is  
>> a mix
>> between CLIENT_ACK and
>> INDIV_ACK in that the acknowledgement is related to the message but  
>> unlike
>> in CLIENT_ACK, does not
>> ack all messages in a session, but rather all messages delivered  
>> before the
>> acked message (requires
>> ordering of messages and tracking them by the broker but I think the
>> building blocks are there)
>>
>> any idea if/when this will be implemented?
>>
>> Cheers,
>>
>> Bonny
>>
>> rajdavies wrote:
>>>
>>> Bonny - agreed - looks like that bit of code has never been
>>> synchronized - will add a fix ...
>>>
>>> cheers,
>>>
>>> Rob
>>>
>>> Rob Davies
>>> http://fusesource.com
>>> http://rajdavies.blogspot.com/
>>>
>>>
>>> On 9 Feb 2009, at 23:14, bonnyr wrote:
>>>
>>>>
>>>> Gary,
>>>>
>>>> Why would the collection be used in CLIENT_ACK mode then?
>>>>
>>>> In our application we've got to work in this mode since there's a
>>>> processing
>>>> chain
>>>> that may take a while between delivery of the message and its ack,  
>>>> so
>>>> chaging this
>>>> mode is not possible for our use case.
>>>>
>>>> We're aware of the fact that CLIENT_ACK actually acks all the
>>>> messages in a
>>>> session and
>>>> it's not really related to the message itself, but that's a story  
>>>> for
>>>> another day and another
>>>> version of AMQ...
>>>>
>>>> However, wouldn't static analysis of the code be sufficient in this
>>>> case?
>>>> The code segment
>>>> above is the only thing that is not protected by synchronization,
>>>> whilst
>>>> every other use of
>>>> this collection is.
>>>>
>>>> Cheers,
>>>>
>>>> Bonny
>>>>
>>>>
>>>> Gary Tully wrote:
>>>>>
>>>>>> I'm not quite able to cause this to happen in a simple test case.
>>>>>> Perhaps
>>>>>> this is to do with some
>>>>>> thread timing/scheduling issues in the simple test case vs our
>>>>>> application.
>>>>>>
>>>>> Yea, that is fairly typical.
>>>>>
>>>>>> However, are you in a
>>>>>> position to explain what the deliveredMessage collection is used
>>>>>> for? Is
>>>>>> it
>>>>>> not used in all connection/
>>>>>> ack modes?
>>>>>>
>>>>> Sure, the deliveredMessage list is used to generate acks in the  
>>>>> Auto
>>>>> ack mode. In addition,
>>>>> with the consumer optimizeAcknowledge option, there is more  
>>>>> likely to
>>>>> be some outstanding
>>>>> delivered messages that require acks. On close, any outstanding  
>>>>> acks
>>>>> will require delivery so
>>>>> you may run into contention with concurrent delivery and close
>>>>> execution as you suggest.
>>>>>
>>>>> May I suggest using AUTO_ACK and additionally using
>>>>> optimizeAcknowledge if needed.
>>>>>
>>>>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>>>>> to construct a test
>>>>>> case that simulate our environment.
>>>>>>
>>>>> Hopefully :-)
>>>>>
>>>>> Gary.
>>>>>
>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Bonny
>>>>>>
>>>>>>
>>>>>> Gary Tully wrote:
>>>>>>>
>>>>>>> Hi Bonny,
>>>>>>> that looks like a bug indeed, should be easy to replicate in
a
>>>>>>> Junit
>>>>>>> tests case I think. Could you raise a jira issue for this and
if
>>>>>>> you
>>>>>>> have some tests code that demonstrates, please include it.
>>>>>>>
>>>>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Gary.
>>>>>>>
>>>>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>>>>
>>>>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as
of  
>>>>>>>> today)
>>>>>>>>
>>>>>>>> My setup:
>>>>>>>> * Broker is configured with a single queue, full with messages,
>>>>>>>> on a
>>>>>>>> host
>>>>>>>> accessible via the network.
>>>>>>>> * Application configured with a single consumer, connected
to a
>>>>>>>> single
>>>>>>>> sesssion, running in its own thread.
>>>>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>>>>> ActiveMQSessionTask
>>>>>>>> threads.
>>>>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>>>>
>>>>>>>>
>>>>>>>> Since the queue is full of messages, delivery of these messages
>>>>>>>> happen
>>>>>>>> as
>>>>>>>> fast as the network
>>>>>>>> can deliver and the AMQ thread is invoking the onMessage
with
>>>>>>>> each new
>>>>>>>> message. These messages
>>>>>>>> are then processed by the consumer thread. The consumer thread
>>>>>>>> then
>>>>>>>> decides
>>>>>>>> to close the connection
>>>>>>>> and the following ensues:
>>>>>>>> [code]
>>>>>>>> java.util.ConcurrentModificationException
>>>>>>>>      at
>>>>>>>> java.util.LinkedList
>>>>>>>> $ListItr.checkForComodification(LinkedList.java:617)
>>>>>>>>      at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:

>>>>>>>> 663)
>>>>>>>>      at
>>>>>>>> org
>>>>>>>> .apache
>>>>>>>> .activemq
>>>>>>>> .ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>>>>      at
>>>>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:

>>>>>>>> 351)
>>>>>>>>      at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>>>>
>>>>>>>>      ... snip ...
>>>>>>>>
>>>>>>>> [/code]
>>>>>>>>
>>>>>>>> This happens because AMQ is busy delivering messages and
there
>>>>>>>> is a
>>>>>>>> collection [b]deliveredMessages[/b] that is
>>>>>>>> not protected by synchronisation in exactly one place
>>>>>>>> (everywhere else
>>>>>>>> it
>>>>>>>> is...) which is executing the
>>>>>>>> following code (in the dispose method):
>>>>>>>> [code]
>>>>>>>> ... snip ...
>>>>>>>>          if (session.isClientAcknowledge()) {
>>>>>>>>              if (!this.info.isBrowser()) {
>>>>>>>>                  // rollback duplicates that aren't acknowledged
>>>>>>>>                  for (MessageDispatch old : deliveredMessages)
{
>>>>>>>>                      session.connection.rollbackDuplicate(this,
>>>>>>>> old.getMessage());
>>>>>>>>                  }
>>>>>>>>              }
>>>>>>>>          }
>>>>>>>> ... snip ...
>>>>>>>> [/code]
>>>>>>>>
>>>>>>>> Since the code iterates over the collection, and the iterator
>>>>>>>> checks
>>>>>>>> for
>>>>>>>> modifications to the underlying collection, and since the
AMQ
>>>>>>>> message
>>>>>>>> delivery thread has managed to sneak in a couple more messages
>>>>>>>> while the consumer thread attempted to close the connection,
the
>>>>>>>> problem
>>>>>>>> occurs (it could be that
>>>>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>>>>
>>>>>>>>
>>>>>>>> Is this an ommission or is there a reason for not synchronising
>>>>>>>> access
>>>>>>>> to
>>>>>>>> this collection? If it's not a bug
>>>>>>>> then how should the consumer be disconnected?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Bonny
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>>>>> Sent from the ActiveMQ - User mailing list archive at  
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> http://blog.garytully.com
>>>>>>>
>>>>>>> Open Source SOA
>>>>>>> http://FUSESource.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> http://blog.garytully.com
>>>>>
>>>>> Open Source SOA
>>>>> http://FUSESource.com
>>>>>
>>>>>
>>>>
>>>> -- 
>>>> View this message in context:
>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945368.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21949975.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message