cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-11713) Add ability to log thread dump when NTR pool is blocked
Date Thu, 05 May 2016 05:40:12 GMT


Jeff Jirsa commented on CASSANDRA-11713:

I know this sounds weird, but we've seen clusters that actually LOOK like they're behaving
DESPITE the high NTR blocks, and I suspect a lot of unsuspecting users who would have been
happy with that type of situation would be hurt severely by this patch because the log output
in those clusters would be significant - for example, we've had 2-3 hour long stress runs
where individual nodes have 7 digit ntr-blocked counters, can you imagine dumping threads
to logs a million times an hour?.

At the very least, I'd personally like to see it guarded with a system property or log spam
protection. As an operator, I'd strongly suggest we not commit this patch until one of those
were available.

> Add ability to log thread dump when NTR pool is blocked
> -------------------------------------------------------
>                 Key: CASSANDRA-11713
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Observability
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>            Priority: Minor
> Thread dumps are very useful for troubleshooting Native-Transport-Requests contention
issues like CASSANDRA-11363 and CASSANDRA-11529.
> While they could be generated externally with {{jstack}}, sometimes the conditions are
transient and it's hard to catch the exact moment when they happen, so it could be useful
to generate and log them upon user request when certain internal condition happens.
> I propose adding a {{logThreadDumpOnNextContention}} flag to {{SEPExecutor}} that when
enabled via JMX generates and logs a single thread dump on the system log when the thread
pool queue is full.

This message was sent by Atlassian JIRA

View raw message