Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Sat, 25 Jun 2016 00:41:16 +0000 (UTC)
From: "Paulo Motta (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12965107.1462399195000.26009.1466815276193@Atlassian.JIRA>
In-Reply-To: <JIRA.12965107.1462399195000@Atlassian.JIRA>
References: <JIRA.12965107.1462399195000@Atlassian.JIRA> <JIRA.12965107.1462399195335@arcas>
Subject: [jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump
 when NTR pool is blocked
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 25 Jun 2016 00:41:18 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348955#comment-15348955 ] 

Paulo Motta commented on CASSANDRA-11713:
-----------------------------------------

Attaching new patch encapsulating thread dumping capability in {{ThreadDumper}} class, which registers itself as a {{org.apache.cassandra.utils.ThreadDumper}} mbean and logs a warn in case it's not able to register the MBean. 

Also added a new parameter {{enableThreadDumping}} to {{SEPExecutor}} that is only enabled by {{RequestThreadPoolExecutor}}. When this parameter is set, a {{ThreadDumper}} is instantiated and {{ThreadDumper.maybeLogThreadDump}} is called when there are blocked requests.

Tested patch with jvisualvm (screenshot attached) and checked that it only logs thread dump once and unsets the flag.

Patch and CI tests below:
||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-11713]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-11713-dtest/lastCompletedBuild/testReport/]|

> Add ability to log thread dump when NTR pool is blocked
> -------------------------------------------------------
>
>                 Key: CASSANDRA-11713
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11713
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Observability
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>            Priority: Minor
>
> Thread dumps are very useful for troubleshooting Native-Transport-Requests contention issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the conditions are transient and it's hard to catch the exact moment when they happen, so it could be useful to generate and log them upon user request when certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} that when enabled via JMX generates and logs a single thread dump on the system log when the thread pool queue is full.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)