cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10241) Keep a separate production debug log for troubleshooting
Date Tue, 01 Sep 2015 21:12:47 GMT


Ariel Weisberg commented on CASSANDRA-10241:

My goal with this suggestion is to eliminate the conflicting priorities of user visible human
readable logging and more verbose less readable debug/trace logging. We can't eliminate the
overhead of trace logging so we can't log everything. Statements that execute one per operation
or per record and statements that dump large data structures aren't good candidates. 

Once you can be chatty in the log you can start logging state transitions that you almost
never care about, but are helpful when the unexpected happens. Over time you can start to
accrue more useful logging as you work through real failures.

We also really need go/no go measurements for what is logged via this mechanism so we know
what it is costing us. Maybe a JMX metric that reports on the log throughput, statements/second,
bytes/second with some historical window so we can diagnose issues related to poorly designed
logging. Or we can just be careful.

bq. Related: is it time to switch to AsyncAppender?
One thing you can do with AsyncAppending is have error and warn log synchronously so you get
a stronger guarantee errors are visible (such as before voluntarily terminating). That somewhat
mitigates the risk of missing important messages.

I think that switching to async appending is less critical in C* because it doesn't have a
thread per core design. If there are places we log while holding locks, well that's a different
story. As long as the thread doing the logging is the only one impacted and we don't have
a design where for whatever reason we have some special threads everyone may end up waiting
on then it is fine.

Put another way it's 4th or 5th nine issue.

> Keep a separate production debug log for troubleshooting
> --------------------------------------------------------
>                 Key: CASSANDRA-10241
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Config
>            Reporter: Jonathan Ellis
>            Assignee: Brandon Williams
>             Fix For: 2.1.x
> [~aweisberg] had the suggestion to keep a separate debug log for aid in troubleshooting,
not intended for regular human consumption but where we can log things that might help if
something goes wrong.

This message was sent by Atlassian JIRA

View raw message