cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-14318) Debug logging can create massive performance issues
Date Fri, 30 Mar 2018 15:19:00 GMT


Paulo Motta updated CASSANDRA-14318:
       Resolution: Fixed
         Reviewer: Paulo Motta
    Fix Version/s:     (was: 3.11.x)
                       (was: 4.x)
                       (was: 3.0.x)
                       (was: 2.2.x)
           Status: Resolved  (was: Patch Available)

{quote}I think anyone performing benchmarks for Cassandra changes should be aware that the
predefined mode isn't relevant and that a user defined test should be used (maybe we should
create one that would be used as standard benchmark).
Good find! Can you check if this is the case in trunk, and if so maybe open a lhf ticket to
change that?
{quote}For the record, the same tests on 3.11.2 didn't show any notable performance difference
between debug on and off
Nice to know we managed to handle all debug/verbose log leaks there. It will be easier to
maintain this after CASSANDRA-14326.
{quote}here's the patch if you're willing to review/commit it, and the unit test results in
Thanks for the patch, experiments and analysis! Even though 2.2 is on critical fixes only
mode, 50% is a significant performance hit on throughput for this workload, and since the
patch is pretty simple I don't see a reason not to commit it.

CI looks good. I added a CHANGES.txt not and committed as {{ac77e5e7742548f7c7c25da3923841f59d4b2713}}
to cassandra-2.2 branch.

> Debug logging can create massive performance issues
> ---------------------------------------------------
>                 Key: CASSANDRA-14318
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alexander Dejanovski
>            Assignee: Alexander Dejanovski
>            Priority: Major
>              Labels: lhf, performance
>             Fix For: 2.2.13
>         Attachments: cassandra-2.2-debug.yaml, debuglogging.png, flame22 nodebug sjk
svg.png, flame22-nodebug-sjk.svg, flame22-sjk.svg, flame_graph_snapshot.png
> Debug logging can involve in many cases (especially very low latency ones) a very important
overhead on the read path in 2.2 as we've seen when upgrading clusters from 2.0 to 2.2.
> The performance impact was especially noticeable on the client side metrics, where p99
could go up to 10 times higher, while ClientRequest metrics recorded by Cassandra didn't show
any overhead.
> Below shows latencies recorded on the client side with debug logging on first, and then
without it :
> !debuglogging.png!  
> We generated a flame graph before turning off debug logging that shows the read call
stack is dominated by debug logging : 
> !flame_graph_snapshot.png!
> I've attached the original flame graph for exploration.
> Once disabled, the new flame graph shows that the read call stack gets extremely thin,
which is further confirmed by client recorded metrics : 
> !flame22 nodebug sjk svg.png!
> The query pager code has been reworked since 3.0 and it looks like log.debug() calls
are gone there, but for 2.2 users and to prevent such issues to appear with default settings,
I really think debug logging should be disabled by default.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message