Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com
 designates 209.85.218.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=fIV9TRA9vLOcjoHC8umsCVG6VFzE61djm4BdOy+qtPMvLe6jCDtoK0/AE7NvcA9kk+
         hD/3e9/WZRGt86O9p5Qk5/fDZIk7aMkdI6z9BPX/l6qhRo4nk/fYQA6RPl0R66OsYijf
         +z20Cfob79Hl5rPh+gbW5fODE6/kFCth/97bQ=
MIME-Version: 1.0
In-Reply-To: <21924323.post@talk.nabble.com>
References: <21867250.post@talk.nabble.com>
	 <3a73c17c0902060338t3d033ae0xab6cae278a966a0@mail.gmail.com>
	 <21907669.post@talk.nabble.com>
	 <3a73c17c0902090242r5ee3de73h9238575d6566b02@mail.gmail.com>
	 <21924323.post@talk.nabble.com>
Date: Tue, 10 Feb 2009 12:16:41 +0000
Message-ID: <3a73c17c0902100416h6221b2c5yaad933d94d6f811@mail.gmail.com>
Subject: Re: ConcurrentModificationException while closing consumer
From: Gary Tully <gary.tully@gmail.com>
To: users@activemq.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Bonnie,
in CLIENT_ACK, the list is used on close to deal with duplicate
suppression, these messages will
get redelivered so they need to be removed from audit that tracks
their receipt. That logic is at fault.

Cheers,
Gary.

2009/2/9 bonnyr <bonnyr@optusnet.com.au>:
>
> Gary,
>
> Why would the collection be used in CLIENT_ACK mode then?
>
> In our application we've got to work in this mode since there's a processing
> chain
> that may take a while between delivery of the message and its ack, so
> chaging this
> mode is not possible for our use case.
>
> We're aware of the fact that CLIENT_ACK actually acks all the messages in a
> session and
> it's not really related to the message itself, but that's a story for
> another day and another
> version of AMQ...
>
> However, wouldn't static analysis of the code be sufficient in this case?
> The code segment
> above is the only thing that is not protected by synchronization, whilst
> every other use of
> this collection is.
>
> Cheers,
>
> Bonny
>
>
> Gary Tully wrote:
>>
>>> I'm not quite able to cause this to happen in a simple test case. Perhaps
>>> this is to do with some
>>> thread timing/scheduling issues in the simple test case vs our
>>> application.
>>>
>> Yea, that is fairly typical.
>>
>>> However, are you in a
>>> position to explain what the deliveredMessage collection is used for? Is
>>> it
>>> not used in all connection/
>>> ack modes?
>>>
>> Sure, the deliveredMessage list is used to generate acks in the Auto
>> ack mode. In addition,
>> with the consumer optimizeAcknowledge option, there is more likely to
>> be some outstanding
>> delivered messages that require acks. On close, any outstanding acks
>> will require delivery so
>> you may run into contention with concurrent delivery and close
>> execution as you suggest.
>>
>> May I suggest using AUTO_ACK and additionally using
>> optimizeAcknowledge if needed.
>>
>>> Perhaps if I knew what the behaviour should be like, I'd be able
>>> to construct a test
>>> case that simulate our environment.
>>>
>> Hopefully :-)
>>
>> Gary.
>>
>>
>>> Cheers,
>>>
>>> Bonny
>>>
>>>
>>> Gary Tully wrote:
>>>>
>>>> Hi Bonny,
>>>> that looks like a bug indeed, should be easy to replicate in a Junit
>>>> tests case I think. Could you raise a jira issue for this and if you
>>>> have some tests code that demonstrates, please include it.
>>>>
>>>> for more info see: http://activemq.apache.org/contributing.html
>>>>
>>>> Thanks,
>>>> Gary.
>>>>
>>>> 2009/2/6 bonnyr <bonnyr@optusnet.com.au>:
>>>>>
>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today)
>>>>>
>>>>> My setup:
>>>>> * Broker is configured with a single queue, full with messages, on a
>>>>> host
>>>>> accessible via the network.
>>>>> * Application configured with a single consumer, connected to a single
>>>>> sesssion, running in its own thread.
>>>>> * ActiveMQ delivers lots of messages using one of the
>>>>> ActiveMQSessionTask
>>>>> threads.
>>>>> * Session configured as CLIENT_ACKNOLEDGE
>>>>>
>>>>>
>>>>> Since the queue is full of messages, delivery of these messages happen
>>>>> as
>>>>> fast as the network
>>>>> can deliver and the AMQ thread is invoking the onMessage with each new
>>>>> message. These messages
>>>>> are then processed by the consumer thread. The consumer thread then
>>>>> decides
>>>>> to close the connection
>>>>> and the following ensues:
>>>>> [code]
>>>>> java.util.ConcurrentModificationException
>>>>>        at
>>>>> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:617)
>>>>>        at java.util.LinkedList$ListItr.next(LinkedList.java:552)
>>>>>        at
>>>>> org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663)
>>>>>        at
>>>>> org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583)
>>>>>        at
>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351)
>>>>>        at com.xxx.app.AMsgQueueConsumer.suspend
>>>>>
>>>>>        ... snip ...
>>>>>
>>>>> [/code]
>>>>>
>>>>> This happens because AMQ is busy delivering messages and there is a
>>>>> collection [b]deliveredMessages[/b] that is
>>>>> not protected by synchronisation in exactly one place (everywhere else
>>>>> it
>>>>> is...) which is executing the
>>>>> following code (in the dispose method):
>>>>> [code]
>>>>>  ... snip ...
>>>>>            if (session.isClientAcknowledge()) {
>>>>>                if (!this.info.isBrowser()) {
>>>>>                    // rollback duplicates that aren't acknowledged
>>>>>                    for (MessageDispatch old : deliveredMessages) {
>>>>>                        session.connection.rollbackDuplicate(this,
>>>>> old.getMessage());
>>>>>                    }
>>>>>                }
>>>>>            }
>>>>>  ... snip ...
>>>>> [/code]
>>>>>
>>>>> Since the code iterates over the collection, and the iterator checks
>>>>> for
>>>>> modifications to the underlying collection, and since the AMQ message
>>>>> delivery thread has managed to sneak in a couple more messages
>>>>> while the consumer thread attempted to close the connection, the
>>>>> problem
>>>>> occurs (it could be that
>>>>> the for syntax hides the explicit iterator calls, but...)
>>>>>
>>>>>
>>>>> Is this an ommission or is there a reason for not synchronising access
>>>>> to
>>>>> this collection? If it's not a bug
>>>>> then how should the consumer be disconnected?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bonny
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://blog.garytully.com
>>>>
>>>> Open Source SOA
>>>> http://FUSESource.com
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source SOA
http://FUSESource.com