cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan McGuire (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-5932) Speculative read performance data show unexpected results
Date Sat, 24 Aug 2013 00:51:51 GMT
Ryan McGuire created CASSANDRA-5932:

             Summary: Speculative read performance data show unexpected results
                 Key: CASSANDRA-5932
             Project: Cassandra
          Issue Type: Bug
            Reporter: Ryan McGuire
         Attachments: compaction-makes-slow.png, eager-read-looks-promising.png, eager-read-not-consistent.png,

I've done a series of stress tests with eager retries enabled that show undesirable behavior.
I'm grouping these behaviours into one ticket as they are most likely related.

1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but
not consistently.

My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7

My script:
node0 writes some data: stress -d node4 -F 30000000 -n 30000000 -i 5 -l 2 -K 20
node0 reads some data: stress -d node4 -n 30000000 -o read -i 5 -K 20

h3. Examples:

h5. A node going down increases performance:


[Data for this test here|]

h5. Compactions make nodes permanently slow:


The green and orange lines represent trials with eager retry enabled, they never recover their
op-rate from before the compaction as the red and blue lines do.

[Data for this test here|]

h5. Speculative Read tends to lessen the *immediate* impact:


This graph looked the most promising to me, the two trials with eager retry, the green and
orange line, at 450s showed the smallest dip in performance. 

[Data for this test here|]

h5. But not always:


This is a retrial with the same settings as above, yet the 95percentile eager retry (red line)
did poorly this time at 450s.

[Data for this test here|]

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message