cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10241) Keep a separate production debug log for troubleshooting
Date Thu, 10 Sep 2015 17:02:46 GMT


Jeff Jirsa commented on CASSANDRA-10241:

Appreciate the ping, [~jbellis]

Pretty torn on this on the operator side.  I have to imagine most production operators (larger
than RF=N=3) probably: 

- Rely on JMX for understanding most day to day state (tracking exceptions, compaction, gc,
etc), and only go to logs when things break, which supports the idea that having more verbose
logs probably would help us when we're debugging something unusual.
- Rely at least partly on {{tail -f}} during those times, so binary log would hurt immediate
access to the logs when they're actually needed (probably more often with the "dozens" of
nodes guys than the "hundreds" / "thousands")
- Would probably disable the extra verbose logging if it hit us with more than a few % performance

If things go bad and the log is there and it helps us find some weird behavior, then it's
great. But if it kills performance we'd turn it off, and if it has to be binary to avoid killing
performance, we wouldn't be able to {{tail -f}} it, which would interrupt at least some of
the normal operator behavior.

I'm not sure there's a happy answer for everyone here on the operations side, so as long as
it can be turned off if it's really a performance hit, and doesn't remove otherwise useful
information from the existing log, I'm sure everyone will adapt over time. 

Personally, I might be tempted to keep a ring buffer of selective debug logged statements
somewhere and allow those to be flushed to disk on command ("nodetool debug"), and then that
can be uploaded to support / jira as requested, or flushed on jvm shutdown for historical
records if desired - no disk hit unless requested, just a small memory penalty? Of course,
that makes it far less useful for looking far into the past, so I'm not sure if that's a win,

> Keep a separate production debug log for troubleshooting
> --------------------------------------------------------
>                 Key: CASSANDRA-10241
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Config
>            Reporter: Jonathan Ellis
>            Assignee: Paulo Motta
>             Fix For: 2.1.x, 2.2.x, 3.0.x
> [~aweisberg] had the suggestion to keep a separate debug log for aid in troubleshooting,
not intended for regular human consumption but where we can log things that might help if
something goes wrong.

This message was sent by Atlassian JIRA

View raw message