cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Boudreault (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10229) bad read performance after a major compaction
Date Wed, 02 Sep 2015 21:49:48 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728089#comment-14728089
] 

Alan Boudreault commented on CASSANDRA-10229:
---------------------------------------------

So, just finished to test again this on ec2 (i2.2xlarge).

Before major compaction: 37500 op/s.
After major compaction run 1: 27000 op/s.
After major compaction run 2: 28800 op/s.
After major compaction run 3:  29000 op/s.
After major compaction run 4:   28700 op/s.

So, its better than the 3x regression I got the other day..... but curious where this 37k
comes from if running it after multiple times doesn't help.

[~enigmacurry]  [~jjordan] I know this ticket seems strange and/or invalid... but I would
like to understand what's going on. Should we investigate more or close this ticket?

[~mambocab] want to give it a try on ec2?

> bad read performance after a major compaction
> ---------------------------------------------
>
>                 Key: CASSANDRA-10229
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10229
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alan Boudreault
>              Labels: perfomance
>         Attachments: users-caching.yaml
>
>
> I am trying to understand what I am seeing. My scenario is very basic, it's a simple
users table with key cache and row cache disabled. I write 50M then read 5M random elements.
The read performance is not that bad BEFORE a major compaction of the data. I see a ~3x performance
regression AFTER I run a major compaction. 
> Here's the read performance numbers for my scenario: 
> {code}
> 3.0 before a major compaction (Key cache and row cache disabled), note that this is the
numbers from 50M,  I see the same with 5M
> ==================================================================================
> Results:
> op rate                   : 9149 [read:9149]
> partition rate            : 9149 [read:9149]
> row rate                  : 9149 [read:9149]
> latency mean              : 32.8 [read:32.8]
> latency median            : 31.2 [read:31.2]
> latency 95th percentile   : 47.2 [read:47.2]
> latency 99th percentile   : 55.0 [read:55.0]
> latency 99.9th percentile : 66.4 [read:66.4]
> latency max               : 305.4 [read:305.4]
> Total partitions          : 50000000 [read:50000000]
> Total errors              : 0 [read:0]
> total gc count            : 0
> total gc mb               : 0
> total gc time (s)         : 0
> avg gc time(ms)           : NaN
> stdev gc time(ms)         : 0
> Total operation time      : 01:31:05
> END
> -rw-rw-r-- 1 aboudreault aboudreault  4.7G Aug 31 08:51 ma-1024-big-Data.db
> -rw-rw-r-- 1 aboudreault aboudreault  4.9G Aug 31 09:08 ma-1077-big-Data.db
> 3.0 after a major compaction (Key cache and row cache disabled), note that this is the
numbers from 50M, I see the same with 5M
> ================================================================================
> Results:
> op rate                   : 3275 [read:3275]
> partition rate            : 3275 [read:3275]
> row rate                  : 3275 [read:3275]
> latency mean              : 91.6 [read:91.6]
> latency median            : 88.8 [read:88.8]
> latency 95th percentile   : 107.2 [read:107.2]
> latency 99th percentile   : 116.0 [read:116.0]
> latency 99.9th percentile : 125.5 [read:125.5]
> latency max               : 249.0 [read:249.0]
> Total partitions          : 50000000 [read:50000000]
> Total errors              : 0 [read:0]
> total gc count            : 0
> total gc mb               : 0
> total gc time (s)         : 0
> avg gc time(ms)           : NaN
> stdev gc time(ms)         : 0
> Total operation time      : 04:14:26
> END
> -rw-rw-r-- 1 aboudreault aboudreault 9.5G Aug 31 09:40 ma-1079-big-Data.db
> 2.1 before major compaction (Key cache and row cache disabled)
> ==============================================================
> Results:
> op rate                   : 21348 [read:21348]
> partition rate            : 21348 [read:21348]
> row rate                  : 21348 [read:21348]
> latency mean              : 14.1 [read:14.1]
> latency median            : 8.0 [read:8.0]
> latency 95th percentile   : 38.5 [read:38.5]
> latency 99th percentile   : 60.8 [read:60.8]
> latency 99.9th percentile : 99.2 [read:99.2]
> latency max               : 229.2 [read:229.2]
> Total partitions          : 5000000 [read:5000000]
> Total errors              : 0 [read:0]
> total gc count            : 0
> total gc mb               : 0
> total gc time (s)         : 0
> avg gc time(ms)           : NaN
> stdev gc time(ms)         : 0
> Total operation time      : 00:03:54
> END
> 2.1 after major compaction (Key cache and row cache disabled)
> =============================================================
> Results:
> op rate                   : 5262 [read:5262]
> partition rate            : 5262 [read:5262]
> row rate                  : 5262 [read:5262]
> latency mean              : 57.0 [read:57.0]
> latency median            : 55.5 [read:55.5]
> latency 95th percentile   : 69.4 [read:69.4]
> latency 99th percentile   : 83.3 [read:83.3]
> latency 99.9th percentile : 197.4 [read:197.4]
> latency max               : 1169.0 [read:1169.0]
> Total partitions          : 5000000 [read:5000000]
> Total errors              : 0 [read:0]
> total gc count            : 0
> total gc mb               : 0
> total gc time (s)         : 0
> avg gc time(ms)           : NaN
> stdev gc time(ms)         : 0
> Total operation time      : 00:15:50
> END
> {code}
> I can reproduce that read performance regression on EC2 and locally. To reproduce:
> 1. Launch a 1 node cluster (2.1, 2.2 or 3.0)
> 2. Set the compaction thoughput at 0. (need a restart IIRC)
> 3. Write 50M elements (so we get the same sstable size for the test). The yaml profile
is attached in this ticket. Ensure you are using stress from apache/cassandra-3.0, trunk is
broken at the moment.
> {code}
> cassandra-stress user profile=`pwd`/users-caching.yaml ops\(insert=1\) n=50M -rate threads=100
> {code}
> 4. Wait for the auto-compaction to finish  and also flush the memtables. You should get
around 2-6 sstables when it's done.
> 5. Read 5M elements
> {code}
> cassandra-stress user profile=/path/to/users-caching.yaml ops\(read=1\) n=5M -rate threads=300
> {code}
> 6. Restart C*, then start a major compaction and wait it's finish.
> {code}
> ccm stop && ccm start
> ccm nodetool compact
> {code}
> 7. Read 5M elements
> {code}
> cassandra-stress user profile=/path/to/users-caching.yaml ops\(read=1\) n=5M -rate threads=300
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message