cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Jordan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-14318) Debug logging can create massive performance issues
Date Sat, 17 Mar 2018 18:22:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403671#comment-16403671
] 

Jeremiah Jordan edited comment on CASSANDRA-14318 at 3/17/18 6:21 PM:
----------------------------------------------------------------------

CASSANDRA-10241 created that debug.log to be a “production” debug log. If there are things
being logged at DEBUG which cause performance issues we should disable those or move them
to TRACE, not turn off the debug.log.


was (Author: jjordan):
CASSANDRA-10241 creates that debug.log to be a “production” debug log. If there are things
being logged at DEBUG which cause performance issues we should disable those or move them
to TRACE, not turn off the debug.log.

> Debug logging can create massive performance issues
> ---------------------------------------------------
>
>                 Key: CASSANDRA-14318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14318
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alexander Dejanovski
>            Priority: Major
>              Labels: lhf, performance
>             Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
>         Attachments: debuglogging.png, flame22 nodebug sjk svg.png, flame22-nodebug-sjk.svg,
flame22-sjk.svg, flame_graph_snapshot.png
>
>
> Debug logging can involve in many cases (especially very low latency ones) a very important
overhead on the read path in 2.2 as we've seen when upgrading clusters from 2.0 to 2.2.
> The performance impact was especially noticeable on the client side metrics, where p99
could go up to 10 times higher, while ClientRequest metrics recorded by Cassandra didn't show
any overhead.
> Below shows latencies recorded on the client side with debug logging on first, and then
without it :
> !debuglogging.png!  
> We generated a flame graph before turning off debug logging that shows the read call
stack is dominated by debug logging : 
> !flame_graph_snapshot.png!
> I've attached the original flame graph for exploration.
> Once disabled, the new flame graph shows that the read call stack gets extremely thin,
which is further confirmed by client recorded metrics : 
> !flame22 nodebug sjk svg.png!
> The query pager code has been reworked since 3.0 and it looks like log.debug() calls
are gone there, but for 2.2 users and to prevent such issues to appear with default settings,
I really think debug logging should be disabled by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message