activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pankaj Takawale (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AMQ-5459) leveldb failed to page in queue messages
Date Tue, 25 Nov 2014 21:14:13 GMT
Pankaj Takawale created AMQ-5459:
------------------------------------

             Summary: leveldb failed to page in queue messages
                 Key: AMQ-5459
                 URL: https://issues.apache.org/jira/browse/AMQ-5459
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.10.0
         Environment: ActiveMQ Replica LevelDB cluster (3 nodes)
Zookeeper 3 nodes cluster
OS: CentOS 6.5

Size of messages: 50K to 500K
Number of messages sent: 9,500
Consumed: around 6000

I recycled master & slave at the same time while client was sending/consuming messages.
ActiveMQ cluster recovered fine, and client resumed producing/consuming messages.

After around two hours, Master faced following exceptions. It demoted itself to slave. New
master got elected, and it faced same issue. It kept happening.
I tried to restart ActiveMQ service on all three nodes. But activemq is facing same issue.
Is there any way to recover out of this state without loosing messages in queue?

Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor  - cbtesta             
                 - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true
- Failed to fill batch
java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException:
ShutdownBrokerInitiated
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
        at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
        at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue                - cbtesta             
                 - Problem retrieving message for browse
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException:
org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
        at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
        at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
        at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]

            Reporter: Pankaj Takawale
            Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message