Mailing-List: contact commits-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Date: Tue, 2 Oct 2012 16:57:07 +1100 (NCT)
From: "Lionel Cons (JIRA)" <jira@apache.org>
To: commits@activemq.apache.org
Message-ID: <71183206.153209.1349157427841.JavaMail.jiratomcat@arcas>
In-Reply-To: <2076889276.31615.1346739307605.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (APLO-257) Better memory usage (avoid Java OOM
 exceptions)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/APLO-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467502#comment-13467502 ] 

Lionel Cons commented on APLO-257:
----------------------------------

We ran the same benchmark scenario on the same hardware with the latest snapshot (apollo-99-trunk-20120929.031419-121) with two changes:
 - more memory: -server -Xmx16384m -Xms4096m -XX:-UseBiasedLocking
 - using the default slow_consumer_policy="block"

We still see memory exceptions, however, these are "GC overhead limit exceeded". Here are some stack traces:

2012-10-01 18:49:51,934 | WARN  | stackref=13a1c2bf7a8
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.fusesource.hawtdispatch.internal.util.TimerHeap.executeReadyTimers(TimerHeap.java:79)
        at org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:144)
2012-10-01 18:50:14,710 | WARN  | stackref=13a1c2bf7a9
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:671)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:721)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.fusesource.hawtdispatch.package$RichExecutor.execute(hawtdispatch.scala:171)
        at org.fusesource.hawtdispatch.package$RichExecutorTrait$class.apply(hawtdispatch.scala:68)
        at org.fusesource.hawtdispatch.package$RichExecutor.apply(hawtdispatch.scala:169)
        at org.apache.activemq.apollo.broker.security.JaasAuthenticator.authenticate(JaasAuthenticator.scala:66)
        at org.apache.activemq.apollo.stomp.StompProtocolHandler$$anonfun$on_stomp_connect$3$$anonfun$apply$mcV$sp$1.apply$mcV$sp(StompProtocolHandler.scala:1107)
        at org.fusesource.hawtdispatch.package$$anon$3.run(hawtdispatch.scala:357)
        at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:96)
        at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77)
2012-10-01 18:50:48,827 | WARN  | stackref=13a1c2bf7aa
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:3065)
        at org.fusesource.hawtdispatch.transport.AbstractProtocolCodec.read(AbstractProtocolCodec.java:307)
        at org.fusesource.hawtdispatch.transport.TcpTransport.drainInbound(TcpTransport.java:657)
        at org.fusesource.hawtdispatch.transport.TcpTransport$6.run(TcpTransport.java:538)
        at org.fusesource.hawtdispatch.internal.NioDispatchSource$3.run(NioDispatchSource.java:226)
        at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:96)
        at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77)
2012-10-01 18:51:08,070 | WARN  | stackref=13a1c2bf7ab
java.lang.OutOfMemoryError: GC overhead limit exceeded
2012-10-01 18:51:38,771 | WARN  | stackref=13a1c2bf7ac
java.lang.OutOfMemoryError: GC overhead limit exceeded

It would be good if Apollo could run happily in its 16GB without causing memory related exceptions...
                
> Better memory usage (avoid Java OOM exceptions)
> -----------------------------------------------
>
>                 Key: APLO-257
>                 URL: https://issues.apache.org/jira/browse/APLO-257
>             Project: ActiveMQ Apollo
>          Issue Type: Improvement
>         Environment: apollo-99-trunk-20120902.143447-110
>            Reporter: Lionel Cons
>
> When running a stomp-benchmark based stress test against Apollo, it generated several OOM exceptions like:
> 2012-09-03 23:51:45,275 | WARN  | handle failed | 
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:02,428 | WARN  |  | 
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:52:01,372 | WARN  | Error for /broker | 
> java.lang.OutOfMemoryError: Java heap space
> 2012-09-03 23:51:55,554 | WARN  | Problem scavenging sessions | 
> java.lang.OutOfMemoryError: Java heap space
> The JVM already had 8GB of heap. FWIW, here are the options used: "-server -Xmx8192m -Xms4096m -XX:-UseBiasedLocking".
> Would it be possible for Apollo to detect when its getting close to the JVM limits and adjust its memory usage accordingly (reducing buffers, swapping messages to disk...)?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira