Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 40FC2200BA1 for ; Mon, 17 Oct 2016 21:17:01 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3F5DA160AE2; Mon, 17 Oct 2016 19:17:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1230D160AF0 for ; Mon, 17 Oct 2016 21:16:59 +0200 (CEST) Received: (qmail 46102 invoked by uid 500); 17 Oct 2016 19:16:59 -0000 Mailing-List: contact issues-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list issues@activemq.apache.org Received: (qmail 45976 invoked by uid 99); 17 Oct 2016 19:16:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Oct 2016 19:16:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 86A552C4C7A for ; Mon, 17 Oct 2016 19:16:58 +0000 (UTC) Date: Mon, 17 Oct 2016 19:16:58 +0000 (UTC) From: "Vijay Govindarajulu (JIRA)" To: issues@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (AMQ-6174) LevelDB gets corrupted when Primary ActiveMQ server is shutdown while messages are queued to it MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 17 Oct 2016 19:17:01 -0000 [ https://issues.apache.org/jira/browse/AMQ-6174?page=3Dcom.atlassian.j= ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D155831= 54#comment-15583154 ]=20 Vijay Govindarajulu edited comment on AMQ-6174 at 10/17/16 7:16 PM: -------------------------------------------------------------------- I am faced with the same problem reported by [~sunilasfvish], my cluster co= nfig is a little different Affects Version/s: 5.11.1 ActiveMQ 5.11.1 with replicated LevelDB (3 node cluster). Zookeeper 3.4.6 (3 node cluster). File system: ext4 {code} 2016-09-28 10:54:40,849 | INFO | Stopping BrokerService[localhost] due to = exception, java.io.EOFException: File '/opt/activemq/activemq-batch/data/le= veldb/000000000c8002d7.log' offset: 3012576207 | org.apache.activemq.u til.DefaultIOExceptionHandler | LevelDB IOException handler. java.io.EOFException: File '/opt/activemq/activemq-batch/data/leveldb/00000= 0000c8002d7.log' offset: 3012576207 at org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.s= cala:389)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(Reco= rdLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(Reco= rdLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala= :644)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:654)[= activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.getMessage(LevelDBClie= nt.scala:1335)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1= .apply(LevelDBClient.scala:1274)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1= .apply(LevelDBClient.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1359)[activemq-lev= eldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1358)[activemq-lev= eldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDB= Client.scala:323)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(Lev= elDBClient.scala:325)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply$mcV$sp(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.= jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.1= 1.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.1= 1.1] at org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClie= nt.scala:1038)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_us= ing_index$1.apply(LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.j= ar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClie= nt.scala:549)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index= (LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.collectionCursor(Level= DBClient.scala:1357)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBCli= ent.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.s= cala:735)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.rec= overNextMessages(LevelDBStore.scala:860)[activemq-leveldb-store-5.11.1.jar:= 5.11.1] at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doF= illBatch(QueueStorePrefetch.java:109)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fi= llBatch(AbstractStoreCursor.java:381)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.re= set(AbstractStoreCursor.java:142)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset= (StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queu= e.java:1897)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.jav= a:2119)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116= )[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.expireMessages(Queue.jav= a:909)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.access$100(Queue.java:10= 0)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[ac= tivemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimer= Task.java:33)[activemq-client-5.11.1.jar:5.11.1] at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0_92] at java.util.TimerThread.run(Timer.java:505)[:1.8.0_92]" {code} activemq.xml snip {code:xml} {code} I was able to resolve the issue by removing the data folder on all the clus= ter nodes and restarting the Brokers.=20 *This error causes the activemq cluster to failover continuously* was (Author: vgovindarajulu): I am faced with the same problem reported by [~sunilasfvish], my cluster co= nfig is a little different Affects Version/s: 5.11.1 ActiveMQ 5.11.1 with replicated LevelDB (3 node cluster). Zookeeper 3.4.6 (3 node cluster). File system: ext4 {code} 2016-09-28 10:54:40,849 | INFO | Stopping BrokerService[localhost] due to = exception, java.io.EOFException: File '/opt/activemq/activemq-batch/data/le= veldb/000000000c8002d7.log' offset: 3012576207 | org.apache.activemq.u til.DefaultIOExceptionHandler | LevelDB IOException handler. java.io.EOFException: File '/opt/activemq/activemq-batch/data/leveldb/00000= 0000c8002d7.log' offset: 3012576207 at org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.s= cala:389)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(Reco= rdLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(Reco= rdLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala= :644)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:654)[= activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.getMessage(LevelDBClie= nt.scala:1335)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1= .apply(LevelDBClient.scala:1274)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1= .apply(LevelDBClient.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1359)[activemq-lev= eldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1358)[activemq-lev= eldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDB= Client.scala:323)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(Lev= elDBClient.scala:325)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply$mcV$sp(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.= jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.1= 1.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCur= sor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.1= 1.1] at org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClie= nt.scala:1038)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_us= ing_index$1.apply(LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.j= ar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClie= nt.scala:549)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index= (LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.collectionCursor(Level= DBClient.scala:1357)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBCli= ent.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.s= cala:735)[activemq-leveldb-store-5.11.1.jar:5.11.1] at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.rec= overNextMessages(LevelDBStore.scala:860)[activemq-leveldb-store-5.11.1.jar:= 5.11.1] at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doF= illBatch(QueueStorePrefetch.java:109)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fi= llBatch(AbstractStoreCursor.java:381)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.re= set(AbstractStoreCursor.java:142)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset= (StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queu= e.java:1897)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.jav= a:2119)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116= )[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.expireMessages(Queue.jav= a:909)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue.access$100(Queue.java:10= 0)[activemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[ac= tivemq-broker-5.11.1.jar:5.11.1] at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimer= Task.java:33)[activemq-client-5.11.1.jar:5.11.1] at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0_92] at java.util.TimerThread.run(Timer.java:505)[:1.8.0_92]" {code} activemq.xml snip {code:xml} {code} I was able to resolve the issue by removing the data folder on all the clus= ter nodes and restarting the Brokers.=20 This error causes the activemq cluster to failover repeatedly. > LevelDB gets corrupted when Primary ActiveMQ server is shutdown while mes= sages are queued to it > -------------------------------------------------------------------------= ---------------------- > > Key: AMQ-6174 > URL: https://issues.apache.org/jira/browse/AMQ-6174 > Project: ActiveMQ > Issue Type: Bug > Components: activemq-leveldb-store > Affects Versions: 5.13.0 > Environment: Virtual type detected as xen-para. > Last rubix: Mon Feb 1 11:02:37 2016 release: 74867 version: 2.0.7 > Installed kernel: 2.6.18-308.0.0.0.1.el5xen x86_64 > Reporter: Sunil Vishwanath > Priority: Critical > > Currently I am testing the following setup: > ActiveMQ 5.13.0 with LevelDB (3 node cluster). > Zookeeper 3.4.6 (3 node cluster). > File system: NFSv3 > Started up all 3 Zookeeper nodes. (aamqzk1, aamqzk2 and aamqzk3) > Started up all 5 ActiveMQ nodes. (aamql1, aamql2, aamql3, aamql4 and aamq= l5) > I started aamql1 first and all others in order. aamql1 is the master and = I am able to see all the queue statistics for aamql1 via ActiveMQ Web Conso= le. > I am also watching all 5 AMQ's "application.log" file using "tail -f appl= ication.log=E2=80=9D command. > The message producer starts sending messages (about 120,000 of them). Wh= ile the messages are being queued and also being consumed, I stopped the ma= ster instance (aamql1). Now aamql2 becomes the master. About 10 seconds la= ter after all the slave aamq reports to the new master, aamql2 throws the f= ollowing exception and the instance dies. This keeps repeating as it failo= ver to the next instance. > 2016-02-17T15:43:48.358885-08:00 aamql2.bus.jetqa1.syseng.tmcs severity= =3DINFO datetime=3D2016-02-17 15:43:48,354 thread=3DLevelDB IOException han= dler. category=3Dorg.apache.activemq.util.DefaultIOExceptionHandler Stoppin= g BrokerService[localhost] due to exception, java.io.EOFException: File '/a= amql/local/activemq/data/leveldb/0000000038409848.log' offset: 477491777 > 2016-02-17T15:43:48.370881-08:00 aamql2.bus.jetqa1.syseng.tmcs java.io.EO= FException: File '/aamql/local/activemq/data/leveldb/0000000038409848.log' = offset: 477491777 > 2016-02-17T15:43:48.371003-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:389) > 2016-02-17T15:43:48.371082-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:65= 4) > 2016-02-17T15:43:48.371148-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:65= 4) > 2016-02-17T15:43:48.371219-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:644) > 2016-02-17T15:43:48.371380-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.RecordLog.read(RecordLog.scala:654) > 2016-02-17T15:43:48.371454-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.getMessage(LevelDBClient.scala:1335) > 2016-02-17T15:43:48.371526-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDB= Client.scala:1274) > 2016-02-17T15:43:48.371604-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDB= Client.scala:1271) > 2016-02-17T15:43:48.371675-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun= $apply$mcV$sp$12.apply(LevelDBClient.scala:1359) > 2016-02-17T15:43:48.371746-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun= $apply$mcV$sp$12.apply(LevelDBClient.scala:1358) > 2016-02-17T15:43:48.371818-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDBClient.scala:3= 23) > 2016-02-17T15:43:48.371888-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(LevelDBClient.sca= la:325) > 2016-02-17T15:43:48.371960-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply$mc= V$sp(LevelDBClient.scala:1358) > 2016-02-17T15:43:48.372034-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(Le= velDBClient.scala:1358) > 2016-02-17T15:43:48.372104-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(Le= velDBClient.scala:1358) > 2016-02-17T15:43:48.372175-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:1038) > 2016-02-17T15:43:48.372259-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.ap= ply(LevelDBClient.scala:1044) > 2016-02-17T15:43:48.372334-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549) > 2016-02-17T15:43:48.372406-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient= .scala:1044) > 2016-02-17T15:43:48.372488-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala= :1357) > 2016-02-17T15:43:48.372573-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1271= ) > 2016-02-17T15:43:48.372635-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:735) > 2016-02-17T15:43:48.372705-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessag= es(LevelDBStore.scala:860) > 2016-02-17T15:43:48.372774-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(Queue= StorePrefetch.java:120) > 2016-02-17T15:43:48.372844-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(Abstra= ctStoreCursor.java:406) > 2016-02-17T15:43:48.372913-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractSt= oreCursor.java:149) > 2016-02-17T15:43:48.372990-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCur= sor.java:169) > 2016-02-17T15:43:48.373060-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1938) > 2016-02-17T15:43:48.373130-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2149) > 2016-02-17T15:43:48.373201-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue.doBrowse(Queue.java:1137) > 2016-02-17T15:43:48.373275-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue.expireMessages(Queue.java:917) > 2016-02-17T15:43:48.373344-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue.access$100(Queue.java:103) > 2016-02-17T15:43:48.373413-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.broker.region.Queue$2.run(Queue.java:146) > 2016-02-17T15:43:48.373484-08:00 aamql2.bus.jetqa1.syseng.tmcs at org= .apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) > 2016-02-17T15:43:48.373553-08:00 aamql2.bus.jetqa1.syseng.tmcs at jav= a.util.TimerThread.mainLoop(Timer.java:555) > 2016-02-17T15:43:48.373622-08:00 aamql2.bus.jetqa1.syseng.tmcs at jav= a.util.TimerThread.run(Timer.java:505) -- This message was sent by Atlassian JIRA (v6.3.4#6332)