activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bonnyr <bon...@optusnet.com.au>
Subject Re: ConcurrentModificationException while closing consumer
Date Tue, 10 Feb 2009 23:19:29 GMT

Gary,

Agreed about the test case - my familiarity with the AMQ test suite is not
that good (excuses, excuses :) ) so what I tried to do, unsuccessfully, was
to play with the ConsumerTool without much luck
on my dev machine which is a WinXP running on a Centrino Duo, and which is
different from our target
boxes - SunFire 420s. Scheduling is possibly different?

In any case, I'm glad you've been able to reproduce this. Is the fix going
to be linked to the JIRA item?
If so, I would be able to apply it (in case it really is localised) to our
version of AMQ and continue 
with out tests.

Cheers,

Bonny


Gary Tully wrote:
> 
> Bonnie,
> in CLIENT_ACK, the list is used on close to deal with duplicate
> suppression, these messages will
> get redelivered so they need to be removed from audit that tracks
> their receipt. That logic is at fault.
> 
> Cheers,
> Gary.
> 
> 2009/2/9 bonnyr <bonnyr@optusnet.com.au>:
>>
>> Gary,
>>
>> Why would the collection be used in CLIENT_ACK mode then?
>>
>> In our application we've got to work in this mode since there's a
>> processing
>> chain
>> that may take a while between delivery of the message and its ack, so
>> chaging this
>> mode is not possible for our use case.
>>
>> We're aware of the fact that CLIENT_ACK actually acks all the messages in
>> a
>> session and
>> it's not really related to the message itself, but that's a story for
>> another day and another
>> version of AMQ...
>>
>> However, wouldn't static analysis of the code be sufficient in this case?
>> The code segment
>> above is the only thing that is not protected by synchronization, whilst
>> every other use of
>> this collection is.
>>
>> Cheers,
>>
>> Bonny
>>
>>
>> Gary Tully wrote:
>>>
>>>> I'm not quite able to cause this to happen in a simple test case.
>>>> Perhaps
>>>> this is to do with some
>>>> thread timing/scheduling issues in the simple test case vs our
>>>> application.
>>>>
>>> Yea, that is fairly typical.
>>>
>>>> However, are you in a
>>>> position to explain what the deliveredMessage collection is used for?
>>>> Is
>>>> it
>>>> not used in all connection/
>>>> ack modes?
>>>>
>>> Sure, the deliveredMessage list is used to generate acks in the Auto
>>> ack mode. In addition,
>>> with the consumer optimizeAcknowledge option, there is more likely to
>>> be some outstanding
>>> delivered messages that require acks. On close, any outstanding acks
>>> will require delivery so
>>> you may run into contention with concurrent delivery and close
>>> execution as you suggest.
>>>
>>> May I suggest using AUTO_ACK and additionally using
>>> optimizeAcknowledge if needed.
>>>
>>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>>> to construct a test
>>>> case that simulate our environment.
>>>>
>>> Hopefully :-)
>>>
>>> Gary.
>>>
>>>
>>>> Cheers,
>>>>
>>>> Bonny
>>>>
>>>>
>>>> Gary Tully wrote:
>>>>>
>>>>> Hi Bonny,
>>>>> that looks like a bug indeed, should be easy to replicate in a Junit
>>>>> tests case I think. Could you raise a jira issue for this and if you
>>>>> have some tests code that demonstrates, please include it.
>>>>>
>>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>>
>>>>> Thanks,
>>>>> Gary.
>>>>>
>>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>>
>>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today)
>>>>>>
>>>>>> My setup:
>>>>>> * Broker is configured with a single queue, full with messages, on
a
>>>>>> host
>>>>>> accessible via the network.
>>>>>> * Application configured with a single consumer, connected to a
>>>>>> single
>>>>>> sesssion, running in its own thread.
>>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>>> ActiveMQSessionTask
>>>>>> threads.
>>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>>
>>>>>>
>>>>>> Since the queue is full of messages, delivery of these messages
>>>>>> happen
>>>>>> as
>>>>>> fast as the network
>>>>>> can deliver and the AMQ thread is invoking the onMessage with each
>>>>>> new
>>>>>> message. These messages
>>>>>> are then processed by the consumer thread. The consumer thread then
>>>>>> decides
>>>>>> to close the connection
>>>>>> and the following ensues:
>>>>>> [code]
>>>>>> java.util.ConcurrentModificationException
>>>>>>        at
>>>>>> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:617)
>>>>>>        at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>>        at
>>>>>> org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663)
>>>>>>        at
>>>>>> org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>>        at
>>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351)
>>>>>>        at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>>
>>>>>>        ... snip ...
>>>>>>
>>>>>> [/code]
>>>>>>
>>>>>> This happens because AMQ is busy delivering messages and there is
a
>>>>>> collection [b]deliveredMessages[/b] that is
>>>>>> not protected by synchronisation in exactly one place (everywhere
>>>>>> else
>>>>>> it
>>>>>> is...) which is executing the
>>>>>> following code (in the dispose method):
>>>>>> [code]
>>>>>>  ... snip ...
>>>>>>            if (session.isClientAcknowledge()) {
>>>>>>                if (!this.info.isBrowser()) {
>>>>>>                    // rollback duplicates that aren't acknowledged
>>>>>>                    for (MessageDispatch old : deliveredMessages)
{
>>>>>>                        session.connection.rollbackDuplicate(this,
>>>>>> old.getMessage());
>>>>>>                    }
>>>>>>                }
>>>>>>            }
>>>>>>  ... snip ...
>>>>>> [/code]
>>>>>>
>>>>>> Since the code iterates over the collection, and the iterator checks
>>>>>> for
>>>>>> modifications to the underlying collection, and since the AMQ message
>>>>>> delivery thread has managed to sneak in a couple more messages
>>>>>> while the consumer thread attempted to close the connection, the
>>>>>> problem
>>>>>> occurs (it could be that
>>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>>
>>>>>>
>>>>>> Is this an ommission or is there a reason for not synchronising
>>>>>> access
>>>>>> to
>>>>>> this collection? If it's not a bug
>>>>>> then how should the consumer be disconnected?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Bonny
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://blog.garytully.com
>>>>>
>>>>> Open Source SOA
>>>>> http://FUSESource.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source SOA
>>> http://FUSESource.com
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source SOA
> http://FUSESource.com
> 
> 

-- 
View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945059.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message