Return-Path: Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: (qmail 39051 invoked from network); 3 Nov 2008 17:45:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 3 Nov 2008 17:45:26 -0000 Received: (qmail 13836 invoked by uid 500); 3 Nov 2008 17:45:32 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 13810 invoked by uid 500); 3 Nov 2008 17:45:32 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 13799 invoked by uid 99); 3 Nov 2008 17:45:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 09:45:32 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 17:44:24 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id A6FB8234C267 for ; Mon, 3 Nov 2008 09:45:05 -0800 (PST) Message-ID: <340112097.1225734305682.JavaMail.jira@brutus> Date: Mon, 3 Nov 2008 09:45:05 -0800 (PST) From: "Gary Tully (JIRA)" To: dev@activemq.apache.org Subject: [jira] Commented: (AMQ-1918) AbstractStoreCursor.size gets out of synch with Store size and blocks consumers In-Reply-To: <261173565.1219933852621.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/activemq/browse/AMQ-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47030#action_47030 ] Gary Tully commented on AMQ-1918: --------------------------------- The following fix may be relevant, the root cause was the cursor and store being out of sync like you describe in the description. https://issues.apache.org/activemq/browse/AMQ-1984 > AbstractStoreCursor.size gets out of synch with Store size and blocks consumers > ------------------------------------------------------------------------------- > > Key: AMQ-1918 > URL: https://issues.apache.org/activemq/browse/AMQ-1918 > Project: ActiveMQ > Issue Type: Bug > Components: Message Store > Affects Versions: 5.1.0 > Reporter: Richard Yarger > Assignee: Rob Davies > Priority: Critical > Fix For: 5.3.0 > > Attachments: activemq.xml, testAMQMessageStore.zip, testdata.zip > > > In version 5.1.0, we are seeing our queue consumers stop consuming for no reason. > We have a staged queue environment and we occasionally see one queue display negative pending message counts that hang around -x, rise to -x+n gradually and then fall back to -x abruptly. The messages are building up and being processed in bunches but its not easy to see because the counts are negative. We see this behavior in the messages coming out of the system. Outbound messages come out in bunches and are synchronized with the queue pending count dropping to -x. > This issue does not happen ALL of the time. It happens about once a week and the only way to fix it is to bounce the broker. It doesn't happen to the same queue everytime, so it is not our consuming code. > Although we don't have a reproducible scenario, we have been able to debug the issue in our test environment. > We traced the problem to the cached store size in the AbstractStoreCursor. > This value becomes 0 or negative and prevents the AbstractStoreCursor from retrieving more messages from the store. (see AbstractStoreCursor.fillBatch() ) > We have seen size value go lower than -1000. > We have also forced it to fix itself by sending in n+1 messages. Once the size goes above zero, the cached value is refreshed and things work ok again. > Unfortunately, during low volume times, it could be hours before n+1 messages are received, so our message latency can rise during low volume times.... :( > I have attached our broker config. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.