activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Sitsky (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-1251) Broker stops delivering messages to some consumers
Date Thu, 06 Sep 2007 21:29:22 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_40104
] 

David Sitsky commented on AMQ-1251:
-----------------------------------

Hiram - thanks for your comment.  Your unit test confirms that what we are dealing with here
is a race condition in the activemq code, somewhere.  Although the original unit tests do
have excessive synchronization, it is not invalid code from what I can tell, and it should
complete to the end.

I started to have a little look at the activemq code under a debugger, and noticed the case
when both workers are stuck, is when they have non-empty pending queues where all message
references have dropped is true.  When the dispatcher sends the next message from the master,
it is just added to both pending queues for the worker, and according to the logic I saw,
it wasn't immediately dispatched since the pending queue was not empty.

I tried pressing the gc() operation on various objects from JMX, but it didn't seem to clear
out any of the messages.

At this stage - nothing seems to happen, the pending queues stay non-empty, and the new message
is never delivered.

> Broker stops delivering messages to some consumers
> --------------------------------------------------
>
>                 Key: AMQ-1251
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1251
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 4.1.0
>         Environment: WinXP
>            Reporter: Vadim Pesochinskiy
>            Assignee: Rob Davies
>             Fix For: 5.0.0
>
>         Attachments: QueueWorkerPrefetchTest.java, TestActiveMQ.java, TestActiveMQSyncReceive.java
>
>
> I have around 40 consumers taking messages from a single queue. After awhile 1 or 2 consumers
stop receiveing any messages. Going to JMX and stopping corresponding connection causes re-connect
and messages are delivered again.
> I reproduced it twice in QA enviroment and now it happened in production. I tried to
instrument the code and set the log in debug, but that changed timing and I failed to reproduce
it after the changes.
> I suspect that runtime association b/w Queue and Consumer objects is lost on the Broker
side. 
> One of the suspects is the empty catch block in the RoundRobinDispatchPolicy (line 64)
class. It is possible that the CopyOnWrite array list is messed up and it fails when removed
consumer is added back. 
> BTW CopyOnWrite list is good when you mostly read, but not so good when you write for
every message delivery and empty catch blocks are bad in any case.
> if (firstMatchingConsumer != null) {
>       // Rotate the consumer list.
>       try {
>                 consumers.remove(firstMatchingConsumer);
>                 consumers.add(firstMatchingConsumer);
>       } catch (Throwable bestEffort) {
>       }
> }

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message