cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Haddad <...@jonhaddad.com>
Subject Re: Overload because of hint pressure + MVs
Date Fri, 07 Feb 2020 18:52:15 GMT
There's a few things you can do here that might help.

First off, if you're using the default heap settings, that's a serious
problem.  If you've got the head room, my recommendation is to use 16GB
heap with 12 GB new gen and pin your memtable heap space to 2GB.  Set your
max tenuring threshold to 6 and your survivor ratio to 6.  You don't need a
lot of old gen space with cassandra, almost everything that will show up
there is memtable related, and we allocate a *lot* whenever we read data
off disk.

Most folks use the default disk read ahead setting of 128KB.  You can check
this setting using blockdev --report, under the RA column.  You'll see 256
there, that's in 512 byte sectors.  MVs rely on a read before a write, so
for every read off disk you do, you'll pull additional 128KB into your page
cache.  This is usually a waste and puts WAY too much pressure on your
disk.  On SSD, I always change this to 4KB.

Next, be sure you're setting your compression rate accordingly.  I wrote a
long post on the topic here:
https://thelastpickle.com/blog/2018/08/08/compression_performance.html.
Our default compression is very unfriendly for read heavy workloads if
you're reading small rows.  If your records are small, 4KB compression
chunk length is your friend.

I have some slides showing pretty good performance improvements from the
above 2 changes.  Specifically, I went from 16K reads a second at 180ms p99
latency up to 63K reads / second at 21ms p99.  Disk usage dropped by a
factor of 10.  Throw in those JVM changes I recommended and things should
improve even further.

Generally speaking, I recommend avoiding MVs, as they can be a giant mine
if you aren't careful.  They're not doing any magic behind the scenes that
makes scaling easier, and in a lot of cases they're a hinderance.  You
still need to understand the underlying data and how it's laid out to use
them properly, which is 99% of the work.

Jon

On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler <michael@pbandjelly.org>
wrote:

> That JIRA still says Open, so no, it has not been fixed (unless there's
> a fixed duplicate in JIRA somewhere).
>
> For clarification, you could update that ticket with a comment including
> your environmental details, usage of MV, etc. I'll bump the priority up
> and include some possible branchX fixvers.
>
> Michael
>
> On 2/7/20 10:53 AM, Surbhi Gupta wrote:
> > Hi,
> >
> > We are getting hit by the below bug.
> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other work
> > around ?
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-13810
> >
> > Any idea if it got fixed in later version.
> > We are on Open source Cassandra 3.11.1  .
> >
> > Thanks
> > Surbhi
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>

Mime
View raw message