activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: ConcurrentModificationException while closing consumer
Date Tue, 10 Feb 2009 07:01:12 GMT
Bonny - agreed - looks like that bit of code has never been  
synchronized - will add a fix ...

cheers,

Rob

Rob Davies
http://fusesource.com
http://rajdavies.blogspot.com/


On 9 Feb 2009, at 23:14, bonnyr wrote:

>
> Gary,
>
> Why would the collection be used in CLIENT_ACK mode then?
>
> In our application we've got to work in this mode since there's a  
> processing
> chain
> that may take a while between delivery of the message and its ack, so
> chaging this
> mode is not possible for our use case.
>
> We're aware of the fact that CLIENT_ACK actually acks all the  
> messages in a
> session and
> it's not really related to the message itself, but that's a story for
> another day and another
> version of AMQ...
>
> However, wouldn't static analysis of the code be sufficient in this  
> case?
> The code segment
> above is the only thing that is not protected by synchronization,  
> whilst
> every other use of
> this collection is.
>
> Cheers,
>
> Bonny
>
>
> Gary Tully wrote:
>>
>>> I'm not quite able to cause this to happen in a simple test case.  
>>> Perhaps
>>> this is to do with some
>>> thread timing/scheduling issues in the simple test case vs our
>>> application.
>>>
>> Yea, that is fairly typical.
>>
>>> However, are you in a
>>> position to explain what the deliveredMessage collection is used  
>>> for? Is
>>> it
>>> not used in all connection/
>>> ack modes?
>>>
>> Sure, the deliveredMessage list is used to generate acks in the Auto
>> ack mode. In addition,
>> with the consumer optimizeAcknowledge option, there is more likely to
>> be some outstanding
>> delivered messages that require acks. On close, any outstanding acks
>> will require delivery so
>> you may run into contention with concurrent delivery and close
>> execution as you suggest.
>>
>> May I suggest using AUTO_ACK and additionally using
>> optimizeAcknowledge if needed.
>>
>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>> to construct a test
>>> case that simulate our environment.
>>>
>> Hopefully :-)
>>
>> Gary.
>>
>>
>>> Cheers,
>>>
>>> Bonny
>>>
>>>
>>> Gary Tully wrote:
>>>>
>>>> Hi Bonny,
>>>> that looks like a bug indeed, should be easy to replicate in a  
>>>> Junit
>>>> tests case I think. Could you raise a jira issue for this and if  
>>>> you
>>>> have some tests code that demonstrates, please include it.
>>>>
>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>
>>>> Thanks,
>>>> Gary.
>>>>
>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>
>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today)
>>>>>
>>>>> My setup:
>>>>> * Broker is configured with a single queue, full with messages,  
>>>>> on a
>>>>> host
>>>>> accessible via the network.
>>>>> * Application configured with a single consumer, connected to a  
>>>>> single
>>>>> sesssion, running in its own thread.
>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>> ActiveMQSessionTask
>>>>> threads.
>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>
>>>>>
>>>>> Since the queue is full of messages, delivery of these messages  
>>>>> happen
>>>>> as
>>>>> fast as the network
>>>>> can deliver and the AMQ thread is invoking the onMessage with  
>>>>> each new
>>>>> message. These messages
>>>>> are then processed by the consumer thread. The consumer thread  
>>>>> then
>>>>> decides
>>>>> to close the connection
>>>>> and the following ensues:
>>>>> [code]
>>>>> java.util.ConcurrentModificationException
>>>>>       at
>>>>> java.util.LinkedList 
>>>>> $ListItr.checkForComodification(LinkedList.java:617)
>>>>>       at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>       at
>>>>> org 
>>>>> .apache 
>>>>> .activemq 
>>>>> .ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663)
>>>>>       at
>>>>> org 
>>>>> .apache 
>>>>> .activemq 
>>>>> .ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>       at
>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351)
>>>>>       at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>
>>>>>       ... snip ...
>>>>>
>>>>> [/code]
>>>>>
>>>>> This happens because AMQ is busy delivering messages and there  
>>>>> is a
>>>>> collection [b]deliveredMessages[/b] that is
>>>>> not protected by synchronisation in exactly one place  
>>>>> (everywhere else
>>>>> it
>>>>> is...) which is executing the
>>>>> following code (in the dispose method):
>>>>> [code]
>>>>> ... snip ...
>>>>>           if (session.isClientAcknowledge()) {
>>>>>               if (!this.info.isBrowser()) {
>>>>>                   // rollback duplicates that aren't acknowledged
>>>>>                   for (MessageDispatch old : deliveredMessages) {
>>>>>                       session.connection.rollbackDuplicate(this,
>>>>> old.getMessage());
>>>>>                   }
>>>>>               }
>>>>>           }
>>>>> ... snip ...
>>>>> [/code]
>>>>>
>>>>> Since the code iterates over the collection, and the iterator  
>>>>> checks
>>>>> for
>>>>> modifications to the underlying collection, and since the AMQ  
>>>>> message
>>>>> delivery thread has managed to sneak in a couple more messages
>>>>> while the consumer thread attempted to close the connection, the
>>>>> problem
>>>>> occurs (it could be that
>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>
>>>>>
>>>>> Is this an ommission or is there a reason for not synchronising  
>>>>> access
>>>>> to
>>>>> this collection? If it's not a bug
>>>>> then how should the consumer be disconnected?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bonny
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://blog.garytully.com
>>>>
>>>> Open Source SOA
>>>> http://FUSESource.com
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> -- 
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>










Mime
View raw message