cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Surbhi Gupta <surbhi.gupt...@gmail.com>
Subject Re: Overload because of hint pressure + MVs
Date Mon, 10 Feb 2020 05:42:41 GMT
Thanks a lot Jon..
Will try the recommendations and let you know the results....

On Fri, Feb 7, 2020 at 10:52 AM Jon Haddad <jon@jonhaddad.com> wrote:

> There's a few things you can do here that might help.
>
> First off, if you're using the default heap settings, that's a serious
> problem.  If you've got the head room, my recommendation is to use 16GB
> heap with 12 GB new gen and pin your memtable heap space to 2GB.  Set your
> max tenuring threshold to 6 and your survivor ratio to 6.  You don't need a
> lot of old gen space with cassandra, almost everything that will show up
> there is memtable related, and we allocate a *lot* whenever we read data
> off disk.
>
> Most folks use the default disk read ahead setting of 128KB.  You can
> check this setting using blockdev --report, under the RA column.  You'll
> see 256 there, that's in 512 byte sectors.  MVs rely on a read before a
> write, so for every read off disk you do, you'll pull additional 128KB into
> your page cache.  This is usually a waste and puts WAY too much pressure on
> your disk.  On SSD, I always change this to 4KB.
>
> Next, be sure you're setting your compression rate accordingly.  I wrote a
> long post on the topic here:
> https://thelastpickle.com/blog/2018/08/08/compression_performance.html.
> Our default compression is very unfriendly for read heavy workloads if
> you're reading small rows.  If your records are small, 4KB compression
> chunk length is your friend.
>
> I have some slides showing pretty good performance improvements from the
> above 2 changes.  Specifically, I went from 16K reads a second at 180ms p99
> latency up to 63K reads / second at 21ms p99.  Disk usage dropped by a
> factor of 10.  Throw in those JVM changes I recommended and things should
> improve even further.
>
> Generally speaking, I recommend avoiding MVs, as they can be a giant mine
> if you aren't careful.  They're not doing any magic behind the scenes that
> makes scaling easier, and in a lot of cases they're a hinderance.  You
> still need to understand the underlying data and how it's laid out to use
> them properly, which is 99% of the work.
>
> Jon
>
> On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler <michael@pbandjelly.org>
> wrote:
>
>> That JIRA still says Open, so no, it has not been fixed (unless there's
>> a fixed duplicate in JIRA somewhere).
>>
>> For clarification, you could update that ticket with a comment including
>> your environmental details, usage of MV, etc. I'll bump the priority up
>> and include some possible branchX fixvers.
>>
>> Michael
>>
>> On 2/7/20 10:53 AM, Surbhi Gupta wrote:
>> > Hi,
>> >
>> > We are getting hit by the below bug.
>> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other work
>> > around ?
>> >
>> > https://issues.apache.org/jira/browse/CASSANDRA-13810
>> >
>> > Any idea if it got fixed in later version.
>> > We are on Open source Cassandra 3.11.1  .
>> >
>> > Thanks
>> > Surbhi
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>

Mime
View raw message