activemq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Miles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APLO-360) leveldb "short write" causes recovery mode to use up all memory & crash apollo
Date Thu, 26 Jun 2014 21:59:24 GMT

    [ https://issues.apache.org/jira/browse/APLO-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045253#comment-14045253
] 

Daniel Miles commented on APLO-360:
-----------------------------------

Tried to change from openJDK to Sun's HotSpot JDK (Version: Java HotSpot(TM) 64-Bit Server
VM 1.8.0_05 (Oracle Corporation)). When LevelDB had its short write, "recovery mode" used
up all the memory again.
{code}
2014-06-26 21:12:50,136 | INFO  | Compacted the leveldb index at: /media/ephemeral0/data/dirty.index
in 588.49 ms | 
2014-06-26 21:12:57,235 | WARN  | DB operation failed. (entering recovery mode): java.io.IOException:
Short write | 146d90544d3
2014-06-26 21:17:38,088 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d90544d4
2014-06-26 21:20:49,299 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
        at org.eclipse.jetty.io.ByteArrayBuffer.<init>(ByteArrayBuffer.java:33)
        at org.eclipse.jetty.io.nio.IndirectNIOBuffer.<init>(IndirectNIOBuffer.java:26)
        at org.eclipse.jetty.io.nio.SslConnection$SslBuffers.<init>(SslConnection.java:79)
        at org.eclipse.jetty.io.nio.SslConnection.allocateBuffers(SslConnection.java:141)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:178)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2014-06-26 21:24:21,082 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d90544d6
2014-06-26 21:25:21,435 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:26:27,991 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:31:41,923 | WARN  |  | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:33:33,991 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:33:33,992 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:34:39,663 | WARN  |  | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:36:19,106 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:36:59,988 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:40:24,706 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:41:40,509 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:46:53,696 | WARN  | handle failed | 
java.lang.OutOfMemoryError: Java heap space
2014-06-26 21:46:38,842 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d90544d7
2014-06-26 21:45:52,760 | WARN  |  | 
java.lang.OutOfMemoryError: Java heap space
{code}

> leveldb "short write" causes recovery mode to use up all memory & crash apollo
> ------------------------------------------------------------------------------
>
>                 Key: APLO-360
>                 URL: https://issues.apache.org/jira/browse/APLO-360
>             Project: ActiveMQ Apollo
>          Issue Type: Bug
>          Components: apollo-broker
>    Affects Versions: 1.7
>         Environment:  * OS: Amazon Linux AMI release 2014.03 (centos-like)
>  * CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GH
>  * MemTotal: 3843028 kB
>  * Startup command: apollo -ea -server -XX:+AggressiveOpts -XX:+OptimizeStringConcat
-XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Xmx2662m -Xms128m -Dcom.sun.management.jmxremote
-Dapollo.home=/opt/apache-apollo-1.7 -Dapollo.base=/var/lib/subscriptions -classpath /opt/apache-apollo-1.7/lib/apollo-boot.jar
org.apache.activemq.apollo.boot.Apollo /var/lib/subscriptions/lib\;/opt/apache-apollo-1.7/lib
org.apache.activemq.apollo.cli.Apollo run
>  * Java: OpenJDK 64-Bit Server VM 1.7.0_55
>            Reporter: Daniel Miles
>         Attachments: loadtest.tar.bz2, log.tar.bz2
>
>
> Apollo has been failing under load, and I can consistently reproduce it with this formula.
I've attached two things to this bug report:
>  * All apollo logs
>  * The load-test application (both source code and compiled binary)
> After a few hours of load testing (one producer, five consumers, roughly 7000 messages/second,
message size 1.4k), leveldb has a "Short Write" exception then Apollo goes into recovery mode
and runs out of memory. All logs are attached, relevant section of stacktrace.log looks like
this:
> {code}
> 2014-06-25 23:54:55,593 | WARN  | stackref=146d56da433
> java.io.IOException: Short write
>         at org.apache.activemq.apollo.broker.store.leveldb.RecordLog$LogAppender.flush(RecordLog.scala:206)
>         at org.apache.activemq.apollo.broker.store.leveldb.RecordLog.appender(RecordLog.scala:464)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.using_index(LevelDBClient.scala:664)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$retry_using_index$1.apply(LevelDBClient.scala:670)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:758)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry_using_index(LevelDBClient.scala:670)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.store(LevelDBClient.scala:869)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$store$1.apply$mcV$sp(LevelDBStore.scala:82)
>         at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN  | stackref=146d56da434
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN  | stackref=146d56da435
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:16:40,941 | WARN  | stackref=146d56da438
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:30,467 | WARN  | stackref=146d56da439
> java.lang.OutOfMemoryError: Java heap space
> {code}
> And the same section of apollo.log looks like this:
> {code}
> 2014-06-25 23:54:55,584 | WARN  | DB operation failed. (entering recovery mode): java.io.IOException:
Short write | 146d56da433
> 2014-06-26 00:01:35,273 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 	at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
> 	at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:76)
> 	at sun.security.ssl.EngineOutputRecord.<init>(EngineOutputRecord.java:65)
> 	at sun.security.ssl.HandshakeOutStream.<init>(HandshakeOutStream.java:63)
> 	at sun.security.ssl.Handshaker.activate(Handshaker.java:491)
> 	at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:711)
> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:809)
> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:761)
> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> 	at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519)
> 	at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:354)
> 	at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> 	at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> 	at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> 	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:04:00,378 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 	at sun.security.ssl.InputRecord.<init>(InputRecord.java:84)
> 	at sun.security.ssl.EngineInputRecord.<init>(EngineInputRecord.java:63)
> 	at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:407)
> 	at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:343)
> 	at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:208)
> 	at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:362)
> 	at org.eclipse.jetty.util.ssl.SslContextFactory.newSslEngine(SslContextFactory.java:1495)
> 	at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.createSSLEngine(SslSelectChannelConnector.java:588)
> 	at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.newConnection(SslSelectChannelConnector.java:551)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newConnection(SelectChannelConnector.java:320)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector.newEndPoint(SelectChannelConnector.java:267)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newEndPoint(SelectChannelConnector.java:326)
> 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.createEndPoint(SelectorManager.java:836)
> 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:491)
> 	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da434
> 2014-06-26 00:10:39,220 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 	at java.math.BigInteger.multiplyToLen(BigInteger.java:1226)
> 	at java.math.BigInteger.oddModPow(BigInteger.java:1764)
> 	at java.math.BigInteger.modPow(BigInteger.java:1599)
> 	at sun.security.rsa.RSACore.crtCrypt(RSACore.java:154)
> 	at sun.security.rsa.RSACore.rsa(RSACore.java:101)
> 	at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177)
> 	at java.security.Signature$Delegate.engineSign(Signature.java:1160)
> 	at java.security.Signature.sign(Signature.java:553)
> 	at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012)
> 	at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:772)
> 	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
> 	at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:370)
> 	at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> 	at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> 	at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> 	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:12:09,137 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:12:54,045 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da435
> 2014-06-26 00:16:33,678 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da438
> 2014-06-26 00:20:41,583 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:21:53,264 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,730 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,731 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da439
> 2014-06-26 00:22:07,731 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:28,222 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:49,662 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:22,976 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:37,279 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:05,740 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:34,485 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:17,858 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:39,331 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message