activemq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Miles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APLO-360) regular, reproducable OutOfMemory exceptions
Date Thu, 26 Jun 2014 16:39:24 GMT

    [ https://issues.apache.org/jira/browse/APLO-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044830#comment-14044830
] 

Daniel Miles commented on APLO-360:
-----------------------------------

Oh, one more thing, here's my apollo config (my load test was using a queue called "fourth":
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements. See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version
  2.0 (the "License"); you may not use this file except in compliance
  with the License. You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0 Unless required by
  applicable law or agreed to in writing, software distributed under
  the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
  OR CONDITIONS OF ANY KIND, either express or implied. See the
  License for the specific language governing permissions and
  limitations under the License.
-->

<!--
  For more information on how configure this file please
  reference:

  http://activemq.apache.org/apollo/versions/1.7/website/documentation/user-manual.html
  -->
<broker xmlns="http://activemq.apache.org/schema/activemq/apollo">

  <notes>
    The default configuration with tls/ssl enabled.
  </notes>

  <log_category console="console" security="security" connection="connection" audit="audit"/>

  
  <authentication domain="apollo"/>
  <!-- Give admins full access -->
  <access_rule allow="admins" action="*"/>
  <access_rule allow="*" action="connect" kind="connector"/>
  

  <virtual_host id="subscriptions">
    <!--
      You should add all the host names that this virtual host is known as
      to properly support the STOMP 1.1 virtual host feature.
      -->
    <host_name>subscriptions</host_name>
    <host_name>localhost</host_name>
    <host_name>127.0.0.1</host_name>
    <!-- shotgunning the following out of ansible: -->
    <host_name>broker0.monsoontools.com</host_name>
    <host_name>broker0.internal.monsoontools.com</host_name>
    <host_name>broker-store.monsoontools.com</host_name>
    <host_name>broker-sub.monsoontools.com</host_name>

    <queue id="distribute" tail_buffer="250M" full_policy="drop head" />
    <queue id="store" tail_buffer="700M" full_policy="drop head" />
    <queue id="cust*" tail_buffer="50M" full_policy="drop head" />
    <queue id="dead*" tail_buffer="50M" full_policy="drop head" />
    <queue id="brokertorture" tail_buffer="3G" full_policy="drop head" />
    <queue id="another" tail_buffer="50M" full_policy="drop head" />
    <queue id="fourth" tail_buffer="700M" quota="3G" full_policy="drop head" />
    <!-- Uncomment to disable security for the virtual host -->
    <!-- <authentication enabled="false"/> -->

    <!-- Uncomment to disable security for the virtual host -->
    <!-- <authentication enabled="false"/> -->
    <access_rule allow="users" action="connect create destroy send receive consume"/>
    

    <!-- You can delete this element if you want to disable persistence for this virtual
host -->
    <leveldb_store directory="/media/ephemeral0/data"/>
    
  </virtual_host>

  <web_admin bind="http://127.0.0.1:61680"/>
  <web_admin bind="https://0.0.0.0:61681"/>

  <connector id="tcp" bind="tcp://0.0.0.0:61613" connection_limit="2000"/>
  <connector id="ssl" bind="tls://0.0.0.0:61614" connection_limit="2000"/>
  <connector id="ws"  bind="ws://0.0.0.0:61623"  connection_limit="2000"/>
  <connector id="wss" bind="wss://0.0.0.0:61624" connection_limit="2000"/>

  <key_storage file="${apollo.base}/etc/keystore" password="password" key_password="password"/>

</broker>
{code}

> regular, reproducable OutOfMemory exceptions
> --------------------------------------------
>
>                 Key: APLO-360
>                 URL: https://issues.apache.org/jira/browse/APLO-360
>             Project: ActiveMQ Apollo
>          Issue Type: Bug
>          Components: apollo-broker
>    Affects Versions: 1.7
>         Environment:  * OS: Amazon Linux AMI release 2014.03 (centos-like)
>  * CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GH
>  * MemTotal: 3843028 kB
>  * Startup command: apollo -ea -server -XX:+AggressiveOpts -XX:+OptimizeStringConcat
-XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Xmx2662m -Xms128m -Dcom.sun.management.jmxremote
-Dapollo.home=/opt/apache-apollo-1.7 -Dapollo.base=/var/lib/subscriptions -classpath /opt/apache-apollo-1.7/lib/apollo-boot.jar
org.apache.activemq.apollo.boot.Apollo /var/lib/subscriptions/lib\;/opt/apache-apollo-1.7/lib
org.apache.activemq.apollo.cli.Apollo run
>  * Java: OpenJDK 64-Bit Server VM 1.7.0_55
>            Reporter: Daniel Miles
>         Attachments: loadtest.tar.bz2, log.tar.bz2
>
>
> Apollo has been failing under load, and I can consistently reproduce it with this formula.
I've attached two things to this bug report:
>  * All apollo logs
>  * The load-test application (both source code and compiled binary)
> After a few hours of load testing (one producer, five consumers, roughly 7000 messages/second,
message size 1.4k), leveldb has a "Short Write" exception then Apollo goes into recovery mode
and runs out of memory. All logs are attached, relevant section of stacktrace.log looks like
this:
> {code}
> 2014-06-25 23:54:55,593 | WARN  | stackref=146d56da433
> java.io.IOException: Short write
>         at org.apache.activemq.apollo.broker.store.leveldb.RecordLog$LogAppender.flush(RecordLog.scala:206)
>         at org.apache.activemq.apollo.broker.store.leveldb.RecordLog.appender(RecordLog.scala:464)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.using_index(LevelDBClient.scala:664)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$retry_using_index$1.apply(LevelDBClient.scala:670)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:758)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry_using_index(LevelDBClient.scala:670)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.store(LevelDBClient.scala:869)
>         at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$store$1.apply$mcV$sp(LevelDBStore.scala:82)
>         at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN  | stackref=146d56da434
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN  | stackref=146d56da435
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:16:40,941 | WARN  | stackref=146d56da438
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:30,467 | WARN  | stackref=146d56da439
> java.lang.OutOfMemoryError: Java heap space
> {code}
> And the same section of apollo.log looks like this:
> {code}
> 2014-06-25 23:54:55,584 | WARN  | DB operation failed. (entering recovery mode): java.io.IOException:
Short write | 146d56da433
> 2014-06-26 00:01:35,273 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 	at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
> 	at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:76)
> 	at sun.security.ssl.EngineOutputRecord.<init>(EngineOutputRecord.java:65)
> 	at sun.security.ssl.HandshakeOutStream.<init>(HandshakeOutStream.java:63)
> 	at sun.security.ssl.Handshaker.activate(Handshaker.java:491)
> 	at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:711)
> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:809)
> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:761)
> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> 	at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519)
> 	at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:354)
> 	at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> 	at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> 	at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> 	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:04:00,378 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 	at sun.security.ssl.InputRecord.<init>(InputRecord.java:84)
> 	at sun.security.ssl.EngineInputRecord.<init>(EngineInputRecord.java:63)
> 	at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:407)
> 	at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:343)
> 	at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:208)
> 	at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:362)
> 	at org.eclipse.jetty.util.ssl.SslContextFactory.newSslEngine(SslContextFactory.java:1495)
> 	at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.createSSLEngine(SslSelectChannelConnector.java:588)
> 	at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.newConnection(SslSelectChannelConnector.java:551)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newConnection(SelectChannelConnector.java:320)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector.newEndPoint(SelectChannelConnector.java:267)
> 	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newEndPoint(SelectChannelConnector.java:326)
> 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.createEndPoint(SelectorManager.java:836)
> 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:491)
> 	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:10:39,219 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da434
> 2014-06-26 00:10:39,220 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 	at java.math.BigInteger.multiplyToLen(BigInteger.java:1226)
> 	at java.math.BigInteger.oddModPow(BigInteger.java:1764)
> 	at java.math.BigInteger.modPow(BigInteger.java:1599)
> 	at sun.security.rsa.RSACore.crtCrypt(RSACore.java:154)
> 	at sun.security.rsa.RSACore.rsa(RSACore.java:101)
> 	at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177)
> 	at java.security.Signature$Delegate.engineSign(Signature.java:1160)
> 	at java.security.Signature.sign(Signature.java:553)
> 	at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012)
> 	at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:772)
> 	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
> 	at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:370)
> 	at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
> 	at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661)
> 	at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030)
> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275)
> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
> 	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622)
> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 2014-06-26 00:12:09,137 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:12:54,045 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:14:44,191 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da435
> 2014-06-26 00:16:33,678 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da438
> 2014-06-26 00:20:41,583 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:21:53,264 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,730 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:22:07,731 | WARN  | java.lang.OutOfMemoryError: Java heap space | 146d56da439
> 2014-06-26 00:22:07,731 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:28,222 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:23:49,662 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:22,976 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:25:37,279 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:05,740 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:26:34,485 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:17,858 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2014-06-26 00:27:39,331 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message