activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Yarger (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-1918) AbstractStoreCursor.size gets out of synch with Store size and blocks consumers
Date Thu, 02 Apr 2009 14:53:34 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=50997#action_50997
] 

Richard Yarger commented on AMQ-1918:
-------------------------------------

I should have mentioned that this test case has been written against tag 5.1.0.
I have been debugging the code and I have some concerns about whether the unit test is producing
the same issue that we are seeing in our production environment.

While the unit test decrements the pending message count extra times and it goes negative,
 it does not cause a cursor size to go negative.

The unit test is producing a negative queue because duplicate messages are being dispatched
to the consumer from the RecoveryDispatch.
PagedInMessages are cached in a RecoveryDispatch when a new subscriber is added.
I was able to resolve the issue by adding a lock check to the Queue.iterate() method.
Which I thought was great until I saw that the RecoveryDispatch had been removed in later
versions.

I am going to run the unit test in trunk.


> AbstractStoreCursor.size gets out of synch with Store size and blocks consumers
> -------------------------------------------------------------------------------
>
>                 Key: AMQ-1918
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1918
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.1.0
>            Reporter: Richard Yarger
>            Assignee: Rob Davies
>            Priority: Critical
>             Fix For: 5.3.0
>
>         Attachments: activemq.xml, NegativeQueueCursorSupport.java, testAMQMessageStore.zip,
testdata.zip
>
>
> In version 5.1.0, we are seeing our queue consumers stop consuming for no reason.
> We have a staged queue environment and we occasionally see one queue display negative
pending message counts that hang around -x, rise to -x+n gradually and then fall back to -x
abruptly. The messages are building up and being processed in bunches but its not easy to
see because the counts are negative. We see this behavior in the messages coming out of the
system. Outbound messages come out in bunches and are synchronized with the queue pending
count dropping to -x.
> This issue does not happen ALL of the time. It happens about once a week and the only
way to fix it is to bounce the broker. It doesn't happen to the same queue everytime, so it
is not our consuming code.
> Although we don't have a reproducible scenario, we have been able to debug the issue
in our test environment.
> We traced the problem to the cached store size in the AbstractStoreCursor.
> This value becomes 0 or negative and prevents the AbstractStoreCursor from retrieving
more messages from the store. (see AbstractStoreCursor.fillBatch() )
> We have seen size value go lower than -1000.
> We have also forced it to fix itself by sending in n+1 messages. Once the size goes above
zero, the cached value is refreshed and things work ok again.
> Unfortunately, during low volume times, it could be hours before n+1 messages are received,
so our message latency can rise during low volume times.... :(
> I have attached our broker config.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message