incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Brophy <cbro...@zulily.com>
Subject Re: Running repair negatively impacts read performance?
Date Fri, 28 Sep 2012 14:20:00 GMT
Odd indeed.

1) It is observable after the compactions are through and the system has
"settled"
2) We're using SizeTiered strategy
3) CentOS 6 & Oracle JVM 1.6.31

I'll do a repair and get some before/after stats to answer your remaining
questions.

Thanks Aaron

On Wed, Sep 26, 2012 at 2:51 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Sounds very odd.
>
> Is read performance degrading _after_ repair and compactions that normally
> result have completed ?
> What Compaction Strategy ?
> What OS and JVM ?
>
> What are are the bloom filter false positive stats from cf stats ?
>
> Do you have some read latency numbers from cfstats ?
> Also, could you take a look at cfhistograms  ?
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/09/2012, at 3:05 AM, Charles Brophy <cbrophy@zulily.com> wrote:
>
> Hey guys,
>
> I've begun to notice that read operations take a performance nose-dive
> after a standard (full) repair of a fairly large column family: ~11 million
> records. Interestingly, I've then noticed that read performance returns to
> normal after a full scrub of the column family. Is it possible that the
> repair operation is not correctly establishing the bloom filter afterwards?
> I've noticed an interesting note of the scrub operation is that it will
> "rebuild sstables with correct bloom filters" which is what is leading me
> to this conclusion. Does this make sense?
>
> I'm using 1.1.3 and Oracle JDK 1.6.31
> The column family is a stanard type and I've noticed this exact behavior
> regardless of the key/column/value serializers in use.
>
> Charles
>
>
>

Mime
View raw message