cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10140) Enable GC logging by default
Date Thu, 20 Aug 2015 17:57:46 GMT


Ariel Weisberg commented on CASSANDRA-10140:

[This is what I was thinking of.|]
Very informative article, thank you. We have seen due to IO overload inside Linux. When this
happens, GC log entries show use_time ~0, sys_time ~ 0, and real-time ~ at least 1 second.
We are able to recreate this type of stalls in the lab too. It turns out that deferred writes
to append a file can be blocked for a long time when the write is blocked by journal commit.
Or when dirty_ratio is exceeded. We straced the Java process and could correlate some but
not all of the stalls to GC threads when they write to the gc.log file. If GC threads do not
have park the Java threads running in kernel mode, we are stumped about what else could have
caused the stall (where user_time ~ 0, sys_time ~0). Any other data/traces you would recommend
to help us understand the issue better? Many thanks.

I agree we should be able to turn on GC logging all the time. But it should buffer sufficiently
to private memory and have a dedicated victim thread to flush to files.

> Enable GC logging by default
> ----------------------------
>                 Key: CASSANDRA-10140
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Config
>            Reporter: Chris Lohfink
>            Assignee: Chris Lohfink
>            Priority: Minor
>         Attachments: CASSANDRA-10140.txt
> Overhead for the gc logging is very small (with cycling logs in 7+) and it provides a
ton of useful information. This will open up more for C* diagnostic tools to provide feedback
as well without requiring restarts.

This message was sent by Atlassian JIRA

View raw message