cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xiangzhou xia (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-13736) CASSANDRA-9673 cause atomic batch p99 increase 3x
Date Tue, 01 Aug 2017 03:44:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

xiangzhou xia updated CASSANDRA-13736:
--------------------------------------
    Description: 
When we testing atomic batch in production traffic, we found that p99 latency in atomic batch
write is 2x-3x worse than 2.2. 

After debuging, we found that the regression is causing by CASSANDRA-9673. This patch changed
consistency level in batchlog store from ONE to TWO. [~iamaleksey] think only block for one
batchlog message is a bug in batchlog and change it to block for two in CASSANDRA-9673, I
think it's actually a very good optimization to reduce latency. 

Set the consistency to one will decrease the possibility of slow data node (GC, long message
queue, etc) affect the latency of atomic batch.  In our shadow cluster, when we change consistency
from two to one, we notice a 2x-3x p99 latency drop in atomic batch.   

  was:we notice that changing consistency level from ONE to TWO dramatically increased p99
latency in 3.0 atomic batch. [~iamaleksey] think only block for one batchlog message is a
bug in batchlog and change it to block for two in CASSANDRA-9673, I think it's actually a
very good optimization to reduce latency. Set the consistency to one will decrease the possibility
of slow data node (GC, long message queue, etc) affect the latency of atomic batch.  In our
shadow cluster, when we change consistency from two to one, we notice a 2x-3x p99 latency
drop in atomic batch.   

        Summary: CASSANDRA-9673 cause atomic batch p99 increase 3x  (was: consistency level
change in batchlog send from CASSANDRA-9673 cause atomic batch p99 increase 3x)

> CASSANDRA-9673 cause atomic batch p99 increase 3x
> -------------------------------------------------
>
>                 Key: CASSANDRA-13736
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13736
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: xiangzhou xia
>            Assignee: xiangzhou xia
>
> When we testing atomic batch in production traffic, we found that p99 latency in atomic
batch write is 2x-3x worse than 2.2. 
> After debuging, we found that the regression is causing by CASSANDRA-9673. This patch
changed consistency level in batchlog store from ONE to TWO. [~iamaleksey] think only block
for one batchlog message is a bug in batchlog and change it to block for two in CASSANDRA-9673,
I think it's actually a very good optimization to reduce latency. 
> Set the consistency to one will decrease the possibility of slow data node (GC, long
message queue, etc) affect the latency of atomic batch.  In our shadow cluster, when we change
consistency from two to one, we notice a 2x-3x p99 latency drop in atomic batch.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message