activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Leach (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6415) LevelDB IOException
Date Tue, 06 Sep 2016 10:19:20 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15467053#comment-15467053
] 

John Leach commented on AMQ-6415:
---------------------------------

Thanks Gary. I've tested it with the native leveldb library but have the same problem. I'm
now running a test with maxExpireMesssagePeriod=0. I already had gcInactiveDestinations enabled,
if it matters. Here's my destinationPolicy section now:

{code:xml}
      <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb"
gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000" expireMessagesPeriod="0">
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"
gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000" expireMessagesPeriod="0">
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>
{code}

We switched to leveldb a couple of years ago after having some pretty bad performance problems
with kahadb, but I see there has been some work on kahadb in the mean time which might help
so I'll give that a go later (though I'm happy to continue debugging this leveldb problem)


> LevelDB IOException
> -------------------
>
>                 Key: AMQ-6415
>                 URL: https://issues.apache.org/jira/browse/AMQ-6415
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.14.0
>         Environment: Ubuntu 14.04, 32bit, OpenJDK 7u111
>            Reporter: John Leach
>              Labels: leveldb
>
> two brokers bridged together, producers and consumers spread across the two.
> After about 8 hours or so processing ~30 messages/second, amq logs a java.io.IOException,
shuts down, and is restarted where it seems to continue on its way:
> {quote}
> 2016-09-04 19:19:01,315 | INFO  | Stopping BrokerService[a.staging_queue.example.com]
due to exception, java.io.IOException | org.apache.activemq.util.DefaultIOExceptionHandler
| LevelDB IOException handler.
> java.io.IOException
>         at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.22.jar:1.22]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_111]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_111]
>         at java.lang.Thread.run(Thread.java:745)[:1.7.0_111]
> 2016-09-04 19:19:01,321 | INFO  | Apache ActiveMQ 5.14.0 (a.staging_queue.example.com,,
ID:srv-2uv19-35903-1472985997586-1:1) is shutting down | org.apache.activemq.broker.BrokerService
| IOExceptionHandler: stopping BrokerService[a.staging_queue.example.com]
> ...
> 2016-09-04 19:19:01,414 | ERROR | Failed to remove expired Message from the store  |
org.apache.activemq.broker.region.Queue | ActiveMQ Broker[a.staging_queue.example.com] Scheduler
> java.io.IOException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
>         at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)[activemq-client-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBStore$.toIOException(LevelDBStore.scala:65)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBStore$.waitOn(LevelDBStore.scala:74)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.removeAsyncMessage(LevelDBStore.scala:811)[activemq-leveldb-store-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:922)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1737)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1729)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1799)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1790)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.doBrowseList(Queue.java:1153)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1131)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:908)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue.access$100(Queue.java:103)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.broker.region.Queue$2.run(Queue.java:146)[activemq-broker-5.14.0.jar:5.14.0]
>         at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.14.0.jar:5.14.0]
>         at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_111]
>         at java.util.TimerThread.run(Timer.java:505)[:1.7.0_111]
> 2016-09-04 19:19:01,465 | INFO  | Stopped LevelDB[/path/to/data/leveldb] | org.apache.activemq.leveldb.LevelDBStore
| LevelDB IOException handler.
> {quote}
> We'd been running 5.9.1 on these exact same servers for many months using leveldb without
this problem. Upgraded to 5.14.0 yesterday and hit it twice already. I tried wiping out the
leveldb database before starting as a test and it still reoccurred.
> Interesting thing is, it happened on both servers within a couple minutes of each other,
even though they had quite different uptimes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message