cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked
Date Sat, 25 Jun 2016 00:46:16 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348955#comment-15348955
] 

Paulo Motta edited comment on CASSANDRA-11713 at 6/25/16 12:45 AM:
-------------------------------------------------------------------

Attaching new patch encapsulating thread dumping capability in {{ThreadDumper}} class, which
registers itself as a {{org.apache.cassandra.utils.ThreadDumper}} mbean and logs a warn in
case it's not able to register the MBean. 

Also added a new parameter {{enableThreadDumping}} to {{SEPExecutor}} that is only enabled
by {{RequestThreadPoolExecutor}}. When this parameter is set, a {{ThreadDumper}} is instantiated
and {{ThreadDumper.maybeLogThreadDump}} is called when there are blocked requests.

Tested patch with jvisualvm (screenshot [attached|https://issues.apache.org/jira/secure/attachment/12813172/ThreadDumper.png])
and verified that it only logs thread dump once and unsets the flag.

Patch and CI tests below:
||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-11713]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-dtest/lastCompletedBuild/testReport/]|


was (Author: pauloricardomg):
Attaching new patch encapsulating thread dumping capability in {{ThreadDumper}} class, which
registers itself as a {{org.apache.cassandra.utils.ThreadDumper}} mbean and logs a warn in
case it's not able to register the MBean. 

Also added a new parameter {{enableThreadDumping}} to {{SEPExecutor}} that is only enabled
by {{RequestThreadPoolExecutor}}. When this parameter is set, a {{ThreadDumper}} is instantiated
and {{ThreadDumper.maybeLogThreadDump}} is called when there are blocked requests.

Tested patch with jvisualvm (screenshot attached) and checked that it only logs thread dump
once and unsets the flag.

Patch and CI tests below:
||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-11713]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-dtest/lastCompletedBuild/testReport/]|

> Add ability to log thread dump when NTR pool is blocked
> -------------------------------------------------------
>
>                 Key: CASSANDRA-11713
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Observability
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>            Priority: Minor
>         Attachments: ThreadDumper.png
>
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests contention
issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the conditions are
transient and it's hard to catch the exact moment when they happen, so it could be useful
to generate and log them upon user request when certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} that when
enabled via JMX generates and logs a single thread dump on the system log when the thread
pool queue is full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message