activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: ConcurrentModificationException while closing consumer
Date Wed, 11 Feb 2009 22:54:41 GMT
the fix can be seen in the linked subversion commits tab of the issue:
https://issues.apache.org/activemq/browse/AMQ-2100?page=com.atlassian.jira.plugin.ext.subversion:subversion-commits-tabpanel
click on the modified files links to see the diffs.

2009/2/10 bonnyr <bonnyr@optusnet.com.au>:
>
> Gary,
>
> Agreed about the test case - my familiarity with the AMQ test suite is not
> that good (excuses, excuses :) ) so what I tried to do, unsuccessfully, was
> to play with the ConsumerTool without much luck
> on my dev machine which is a WinXP running on a Centrino Duo, and which is
> different from our target
> boxes - SunFire 420s. Scheduling is possibly different?
>
> In any case, I'm glad you've been able to reproduce this. Is the fix going
> to be linked to the JIRA item?
> If so, I would be able to apply it (in case it really is localised) to our
> version of AMQ and continue
> with out tests.
>
> Cheers,
>
> Bonny
>
>
> Gary Tully wrote:
>>
>> Bonnie,
>> in CLIENT_ACK, the list is used on close to deal with duplicate
>> suppression, these messages will
>> get redelivered so they need to be removed from audit that tracks
>> their receipt. That logic is at fault.
>>
>> Cheers,
>> Gary.
>>
>> 2009/2/9 bonnyr <bonnyr@optusnet.com.au>:
>>>
>>> Gary,
>>>
>>> Why would the collection be used in CLIENT_ACK mode then?
>>>
>>> In our application we've got to work in this mode since there's a
>>> processing
>>> chain
>>> that may take a while between delivery of the message and its ack, so
>>> chaging this
>>> mode is not possible for our use case.
>>>
>>> We're aware of the fact that CLIENT_ACK actually acks all the messages in
>>> a
>>> session and
>>> it's not really related to the message itself, but that's a story for
>>> another day and another
>>> version of AMQ...
>>>
>>> However, wouldn't static analysis of the code be sufficient in this case?
>>> The code segment
>>> above is the only thing that is not protected by synchronization, whilst
>>> every other use of
>>> this collection is.
>>>
>>> Cheers,
>>>
>>> Bonny
>>>
>>>
>>> Gary Tully wrote:
>>>>
>>>>> I'm not quite able to cause this to happen in a simple test case.
>>>>> Perhaps
>>>>> this is to do with some
>>>>> thread timing/scheduling issues in the simple test case vs our
>>>>> application.
>>>>>
>>>> Yea, that is fairly typical.
>>>>
>>>>> However, are you in a
>>>>> position to explain what the deliveredMessage collection is used for?
>>>>> Is
>>>>> it
>>>>> not used in all connection/
>>>>> ack modes?
>>>>>
>>>> Sure, the deliveredMessage list is used to generate acks in the Auto
>>>> ack mode. In addition,
>>>> with the consumer optimizeAcknowledge option, there is more likely to
>>>> be some outstanding
>>>> delivered messages that require acks. On close, any outstanding acks
>>>> will require delivery so
>>>> you may run into contention with concurrent delivery and close
>>>> execution as you suggest.
>>>>
>>>> May I suggest using AUTO_ACK and additionally using
>>>> optimizeAcknowledge if needed.
>>>>
>>>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>>>> to construct a test
>>>>> case that simulate our environment.
>>>>>
>>>> Hopefully :-)
>>>>
>>>> Gary.
>>>>
>>>>
>>>>> Cheers,
>>>>>
>>>>> Bonny
>>>>>
>>>>>
>>>>> Gary Tully wrote:
>>>>>>
>>>>>> Hi Bonny,
>>>>>> that looks like a bug indeed, should be easy to replicate in a Junit
>>>>>> tests case I think. Could you raise a jira issue for this and if
you
>>>>>> have some tests code that demonstrates, please include it.
>>>>>>
>>>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>>>
>>>>>> Thanks,
>>>>>> Gary.
>>>>>>
>>>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>>>
>>>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today)
>>>>>>>
>>>>>>> My setup:
>>>>>>> * Broker is configured with a single queue, full with messages,
on a
>>>>>>> host
>>>>>>> accessible via the network.
>>>>>>> * Application configured with a single consumer, connected to
a
>>>>>>> single
>>>>>>> sesssion, running in its own thread.
>>>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>>>> ActiveMQSessionTask
>>>>>>> threads.
>>>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>>>
>>>>>>>
>>>>>>> Since the queue is full of messages, delivery of these messages
>>>>>>> happen
>>>>>>> as
>>>>>>> fast as the network
>>>>>>> can deliver and the AMQ thread is invoking the onMessage with
each
>>>>>>> new
>>>>>>> message. These messages
>>>>>>> are then processed by the consumer thread. The consumer thread
then
>>>>>>> decides
>>>>>>> to close the connection
>>>>>>> and the following ensues:
>>>>>>> [code]
>>>>>>> java.util.ConcurrentModificationException
>>>>>>>        at
>>>>>>> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:617)
>>>>>>>        at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>>>        at
>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663)
>>>>>>>        at
>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>>>        at
>>>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351)
>>>>>>>        at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>>>
>>>>>>>        ... snip ...
>>>>>>>
>>>>>>> [/code]
>>>>>>>
>>>>>>> This happens because AMQ is busy delivering messages and there
is a
>>>>>>> collection [b]deliveredMessages[/b] that is
>>>>>>> not protected by synchronisation in exactly one place (everywhere
>>>>>>> else
>>>>>>> it
>>>>>>> is...) which is executing the
>>>>>>> following code (in the dispose method):
>>>>>>> [code]
>>>>>>>  ... snip ...
>>>>>>>            if (session.isClientAcknowledge()) {
>>>>>>>                if (!this.info.isBrowser()) {
>>>>>>>                    // rollback duplicates that aren't acknowledged
>>>>>>>                    for (MessageDispatch old : deliveredMessages)
{
>>>>>>>                        session.connection.rollbackDuplicate(this,
>>>>>>> old.getMessage());
>>>>>>>                    }
>>>>>>>                }
>>>>>>>            }
>>>>>>>  ... snip ...
>>>>>>> [/code]
>>>>>>>
>>>>>>> Since the code iterates over the collection, and the iterator
checks
>>>>>>> for
>>>>>>> modifications to the underlying collection, and since the AMQ
message
>>>>>>> delivery thread has managed to sneak in a couple more messages
>>>>>>> while the consumer thread attempted to close the connection,
the
>>>>>>> problem
>>>>>>> occurs (it could be that
>>>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>>>
>>>>>>>
>>>>>>> Is this an ommission or is there a reason for not synchronising
>>>>>>> access
>>>>>>> to
>>>>>>> this collection? If it's not a bug
>>>>>>> then how should the consumer be disconnected?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Bonny
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://blog.garytully.com
>>>>>>
>>>>>> Open Source SOA
>>>>>> http://FUSESource.com
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://blog.garytully.com
>>>>
>>>> Open Source SOA
>>>> http://FUSESource.com
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945059.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>



-- 
http://blog.garytully.com

Open Source SOA
http://FUSESource.com

Mime
View raw message