activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxence Dunnewind (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6482) Possible thread leak ?
Date Wed, 02 Nov 2016 22:04:58 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630685#comment-15630685
] 

Maxence Dunnewind commented on AMQ-6482:
----------------------------------------

New websocket connections are definitively created quiet often. They are created every time
a user log in to the website (or throug mobile app) or switch to a new page while being logged
in. 

Something (maybe) interesting is that we had a bug in our code which was creating too many
topics (~ 100k in a few seconds) by sending messages to lot of these new topics. 95% of them
were never read. We fixed it and reduced to ~ 8000 topics (no GC of old destination right
now), but the thread leak is still present and seems to grow at same speed than before (from
~ 300 right after start to ~ 1000 in 3 hours), so looks like this is not related to topics/messages.
So the only remaining possibility is client side.  Moreover, looking at the graph, it seems
that the number of thread increases by "step". For ex between 7.50 and 8.10 pm (utc+2) it
only increase of 100threads,  then in 15 minutes I have a 500 thread bump, then almost stable
for 10 minutes, then +400 threads in 5 minutes etc ... It seems that the number of threads
increases faster when I have a lot of "outgoing messages", and slowly when I don't have much,
maybe a hint.

Though this is a production service, we can accept small downtime (like for a restart, if
config update is needed) and I'm available as Sp4rKy on freenode if you want some live debug
(baiscally between 9 and 19 utc+2).

> Possible thread leak ?
> ----------------------
>
>                 Key: AMQ-6482
>                 URL: https://issues.apache.org/jira/browse/AMQ-6482
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.14.1
>            Reporter: Maxence Dunnewind
>              Labels: threading
>         Attachments: thread_dump
>
>
> We have an activemq broker running for ~ 24 hours, handling ~ 100k topics and up to 8k
connections (for ~ 24k consumers).
> Looking at my graph, I see that I get a big increase of threads count happended (in 4
steps, ~ 500 more threads each time) as shown here : https://snapshot.raintank.io/dashboard/snapshot/ExUb4pgNYnmBo92JgbZrvJ6fiBm3PNrI
> However, even if the traffic dropped in the night, the thread count never decreased and
is still ~ 2500.
> I checked a thread dump and it looks like (full dump attached) :
> ```
> # grep  Thread.State thread_dump   |awk '{ print $2 }' |sort|uniq -c
>      26 RUNNABLE
>      70 TIMED_WAITING
>    2446 WAITING
> ```
> Mhh, so many WAITING threads, let's check why :
> ```
> # grep   'parking to' thread_dump    |cut -d '>' -f 2 |sort |uniq -c
>       1  (a java.util.concurrent.CountDownLatch$Sync)
>    1308  (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>    1159  (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>      41  (a java.util.concurrent.SynchronousQueue$TransferStack)
> ```
> I'm not a big java expert but seems a bit unexpected behavior to me.
> For various reason (big number of topics etc ) we use custom GC config, here it is :
> ```
> ACTIVEMQ_OPTS_MEMORY="-Xms128M -Xmx10G -Xloggc:/home/log/activemq/gc.log -XX:+PrintGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=75
-XX:NewRatio=4"
> ```
> Maxence



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message