activemq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiram Chirino (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (APLO-160) Apollo becoming unresponsive under stress test
Date Thu, 16 Feb 2012 19:16:59 GMT

    [ https://issues.apache.org/jira/browse/APLO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209625#comment-13209625
] 

Hiram Chirino edited comment on APLO-160 at 2/16/12 7:15 PM:
-------------------------------------------------------------

Hi Lionel, thanks for sending me that heap.  I just loaded it up and it looks like it had
48,317 connection objects in memory which were retaining 6,044 M of memory.  Which comes out
to about 128k per connection. Which seems right since we use a 64k send and receive buffer
per connection[1]. 

That 6,044 M is quickly going to make it into the tenured GC Generation since are associated
with connection buffer which remain in memory as long as the connection is in memory.  Since
your -Xmx is set to 8192 M and assuming [2] your JVM is using 1:2 young to tenured GC ratio,
then that means. 2,730 M is dedicated to the young generation and 5,462 is dedicated to the
tenured generation.  So to me it's not surprising that your having GC issues.

Things that you can try to help reduce your problems, use smaller values for the receive_buffer_size
and send_buffer_size transport settings.  Using a 4k buffer should let you support 16x more
connections.  I'm not sure what the bottom limit is on this configuration value.
Increase your -Xmx setting.  Increase the new generation ratio.  For example -XX:NewRatio=3
(If you increase this too much transient data will go into the old gen and you will end up
doing more old gen collections which are more expensive).

[1]: http://activemq.apache.org/apollo/documentation/user-manual.html#TCP_Transports
[2]: http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20The%20Young%20Generation%7coutline

                
      was (Author: chirino):
    Hi Lionel, thanks for sending me that heap.  I just loaded it up and it looks like it
had 48,317 connection objects in memory which were retaining 6,044 M of memory.  Which comes
out to about 128k per connection. Which seems right since we use a 64k send and receive buffer
per connection[1]. 

That 6,044 M is quickly going to make it into the tenured GC Generation since are associated
with connection buffer which remain in memory as long as the connection is in memory.  Since
your -Xmx is set to 8192 M and assuming [2] your JVM is using 1:2 young to tenured GC ratio,
then that means. 2,730 M is dedicated to the young generation and 5,462 is dedicated to the
tenured generation.  So to me it's not surprising that your having GC issues.

Things that you can try to help reduce your problems, use smaller values for the receive_buffer_size
and send_buffer_size transport settings.  Using a 4k buffer should let you support 16x more
connections.  I'm not sure what the bottom limit is on this configuration value.
Increase your -Xmx setting.  Increase the new generation ratio.  For example -XX:NewRatio=3
(If you increase this too much transient data will go into the old gen and you will end up
doing more old gen collections which are more expensive).

[1]: http://activemq.apache.org/apollo/documentation/user-manual.html#TCP_Transports
[2]: http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20The%20Young%20Generation|outline

                  
> Apollo becoming unresponsive under stress test
> ----------------------------------------------
>
>                 Key: APLO-160
>                 URL: https://issues.apache.org/jira/browse/APLO-160
>             Project: ActiveMQ Apollo
>          Issue Type: Bug
>         Environment: apollo-1.1-20120209.032648-24
>            Reporter: Lionel Cons
>         Attachments: apollo.dump
>
>
> While running a stress test against apollo-1.1-20120209.032648-24 (many concurrent TCP
connections), the broker became unresponsive.
> It logged several times: java.lang.OutOfMemoryError: GC overhead limit exceeded
> It also logged other warnings, probably related:
> 2012-02-14 14:14:49,273 | WARN  | handle failed | org.eclipse.jetty.io.nio | Apollo Task
> 2012-02-14 14:18:39,073 | WARN  | Problem scavenging sessions | org.eclipse.jetty.server.session
| HashSessionScavenger-0
> It could not be stopped either, I had to kill -9 it.
> What can be done to avoid these problems?
> FWIW, java has been started with -server -Xmx8192m -Xms4096m

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message