cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10241) Keep a separate production debug log for troubleshooting
Date Fri, 11 Sep 2015 14:41:46 GMT


Paulo Motta commented on CASSANDRA-10241:

Given there is no strong opposition against providing a more detailed logging, I propose following
[~benedict]'s summary in the context of this ticket:
* shrinking the current regular INFO (and up) output, to only very high level state changes
and important information
* moving DEBUG statements that cannot be logged as standard to TRACE
* expanding DEBUG with low traffic high information statements, and including everything we
remove from INFO (such as all of the flush / compaction information)
* logging DEBUG (and up) to a separate file

This will provide us the folllowing benefits:
* Sytem.log will be much more accessible to non-power users (manager-friendly), and we can
rely on it to to display optimization tips (such as queries with too much tombstone, or too
small sstable_size_in_mb for LCS) improving the value of system.log. We can even incorporate
a system.log console to Opscenter and other monitoring tools for easier high-level troubleshooting
in the future.
* Debug.log will be able to display more frequent status changes and detailed monitoring of
infrequent operational activities, what can help in debugging production problems and file
JIRA tickets.

Regarding [~aweisberg] concern that class or package-level logging is not the way to go, although
providing a different logger for these activities could solve that, it would introduce additional
complexity and would require developers to remember which logger to call for which situation,
which could hamper the adoption of the guideline. I think we can enable {{DEBUG}} logging
in all packages and use the log levels themselves to handle different detail levels:
* {{INFO}} Very high-level "your-grandma-will-understand" node information and optimization
* {{DEBUG}} More detailed state changes (frequency higher than a second) and details of infrequent
* {{TRACE}} very chatty-stuff, development debugging, very frequent state transitions.

In order to support that in the context of this ticket, I propose downgrading all current
{{DEBUG}} statements to {{TRACE}}, and I would go through the packages mentioned before and
upgrade the relevant statements to {{DEBUG}}. We could later perform grooming in other packages
if necessary. I would also downgrade too frequent {{INFO}} statements (such as compaction
and flushing) to {{DEBUG}}. This way, developers would be able to use the new logging guidelines
with the current logging constructs.

We should also provide a simple way of disabling debug logging if operators desire so, as
[~jjirsa] mentioned.

> Keep a separate production debug log for troubleshooting
> --------------------------------------------------------
>                 Key: CASSANDRA-10241
>                 URL:
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Config
>            Reporter: Jonathan Ellis
>            Assignee: Paulo Motta
>             Fix For: 2.1.x, 2.2.x, 3.0.x
> [~aweisberg] had the suggestion to keep a separate debug log for aid in troubleshooting,
not intended for regular human consumption but where we can log things that might help if
something goes wrong.

This message was sent by Atlassian JIRA

View raw message