activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bonnyr <bon...@optusnet.com.au>
Subject Re: ConcurrentModificationException while closing consumer
Date Tue, 10 Feb 2009 23:42:36 GMT

Rob,

Can I hijack this issue and bring back an issue I have filed in Jira and is
assigned to you (AMQ-1916) and
that is the issue of a new ack mode (I called it WINDOW_ACK) that is a mix
between CLIENT_ACK and 
INDIV_ACK in that the acknowledgement is related to the message but unlike
in CLIENT_ACK, does not 
ack all messages in a session, but rather all messages delivered before the
acked message (requires 
ordering of messages and tracking them by the broker but I think the
building blocks are there)

any idea if/when this will be implemented?

Cheers,

Bonny

rajdavies wrote:
> 
> Bonny - agreed - looks like that bit of code has never been  
> synchronized - will add a fix ...
> 
> cheers,
> 
> Rob
> 
> Rob Davies
> http://fusesource.com
> http://rajdavies.blogspot.com/
> 
> 
> On 9 Feb 2009, at 23:14, bonnyr wrote:
> 
>>
>> Gary,
>>
>> Why would the collection be used in CLIENT_ACK mode then?
>>
>> In our application we've got to work in this mode since there's a  
>> processing
>> chain
>> that may take a while between delivery of the message and its ack, so
>> chaging this
>> mode is not possible for our use case.
>>
>> We're aware of the fact that CLIENT_ACK actually acks all the  
>> messages in a
>> session and
>> it's not really related to the message itself, but that's a story for
>> another day and another
>> version of AMQ...
>>
>> However, wouldn't static analysis of the code be sufficient in this  
>> case?
>> The code segment
>> above is the only thing that is not protected by synchronization,  
>> whilst
>> every other use of
>> this collection is.
>>
>> Cheers,
>>
>> Bonny
>>
>>
>> Gary Tully wrote:
>>>
>>>> I'm not quite able to cause this to happen in a simple test case.  
>>>> Perhaps
>>>> this is to do with some
>>>> thread timing/scheduling issues in the simple test case vs our
>>>> application.
>>>>
>>> Yea, that is fairly typical.
>>>
>>>> However, are you in a
>>>> position to explain what the deliveredMessage collection is used  
>>>> for? Is
>>>> it
>>>> not used in all connection/
>>>> ack modes?
>>>>
>>> Sure, the deliveredMessage list is used to generate acks in the Auto
>>> ack mode. In addition,
>>> with the consumer optimizeAcknowledge option, there is more likely to
>>> be some outstanding
>>> delivered messages that require acks. On close, any outstanding acks
>>> will require delivery so
>>> you may run into contention with concurrent delivery and close
>>> execution as you suggest.
>>>
>>> May I suggest using AUTO_ACK and additionally using
>>> optimizeAcknowledge if needed.
>>>
>>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>>> to construct a test
>>>> case that simulate our environment.
>>>>
>>> Hopefully :-)
>>>
>>> Gary.
>>>
>>>
>>>> Cheers,
>>>>
>>>> Bonny
>>>>
>>>>
>>>> Gary Tully wrote:
>>>>>
>>>>> Hi Bonny,
>>>>> that looks like a bug indeed, should be easy to replicate in a  
>>>>> Junit
>>>>> tests case I think. Could you raise a jira issue for this and if  
>>>>> you
>>>>> have some tests code that demonstrates, please include it.
>>>>>
>>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>>
>>>>> Thanks,
>>>>> Gary.
>>>>>
>>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>>
>>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today)
>>>>>>
>>>>>> My setup:
>>>>>> * Broker is configured with a single queue, full with messages, 

>>>>>> on a
>>>>>> host
>>>>>> accessible via the network.
>>>>>> * Application configured with a single consumer, connected to a 

>>>>>> single
>>>>>> sesssion, running in its own thread.
>>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>>> ActiveMQSessionTask
>>>>>> threads.
>>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>>
>>>>>>
>>>>>> Since the queue is full of messages, delivery of these messages 

>>>>>> happen
>>>>>> as
>>>>>> fast as the network
>>>>>> can deliver and the AMQ thread is invoking the onMessage with  
>>>>>> each new
>>>>>> message. These messages
>>>>>> are then processed by the consumer thread. The consumer thread  
>>>>>> then
>>>>>> decides
>>>>>> to close the connection
>>>>>> and the following ensues:
>>>>>> [code]
>>>>>> java.util.ConcurrentModificationException
>>>>>>       at
>>>>>> java.util.LinkedList 
>>>>>> $ListItr.checkForComodification(LinkedList.java:617)
>>>>>>       at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>>       at
>>>>>> org 
>>>>>> .apache 
>>>>>> .activemq 
>>>>>> .ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663)
>>>>>>       at
>>>>>> org 
>>>>>> .apache 
>>>>>> .activemq 
>>>>>> .ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>>       at
>>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351)
>>>>>>       at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>>
>>>>>>       ... snip ...
>>>>>>
>>>>>> [/code]
>>>>>>
>>>>>> This happens because AMQ is busy delivering messages and there  
>>>>>> is a
>>>>>> collection [b]deliveredMessages[/b] that is
>>>>>> not protected by synchronisation in exactly one place  
>>>>>> (everywhere else
>>>>>> it
>>>>>> is...) which is executing the
>>>>>> following code (in the dispose method):
>>>>>> [code]
>>>>>> ... snip ...
>>>>>>           if (session.isClientAcknowledge()) {
>>>>>>               if (!this.info.isBrowser()) {
>>>>>>                   // rollback duplicates that aren't acknowledged
>>>>>>                   for (MessageDispatch old : deliveredMessages) {
>>>>>>                       session.connection.rollbackDuplicate(this,
>>>>>> old.getMessage());
>>>>>>                   }
>>>>>>               }
>>>>>>           }
>>>>>> ... snip ...
>>>>>> [/code]
>>>>>>
>>>>>> Since the code iterates over the collection, and the iterator  
>>>>>> checks
>>>>>> for
>>>>>> modifications to the underlying collection, and since the AMQ  
>>>>>> message
>>>>>> delivery thread has managed to sneak in a couple more messages
>>>>>> while the consumer thread attempted to close the connection, the
>>>>>> problem
>>>>>> occurs (it could be that
>>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>>
>>>>>>
>>>>>> Is this an ommission or is there a reason for not synchronising 

>>>>>> access
>>>>>> to
>>>>>> this collection? If it's not a bug
>>>>>> then how should the consumer be disconnected?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Bonny
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://blog.garytully.com
>>>>>
>>>>> Open Source SOA
>>>>> http://FUSESource.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> -- 
>>> http://blog.garytully.com
>>>
>>> Open Source SOA
>>> http://FUSESource.com
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945368.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message