cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Cassandra rate dropping over long term test
Date Fri, 03 Aug 2018 13:58:46 GMT
Probably Compaction

Cassandra data files are immutable

The write path first appends to a commitlog, then puts data into the memtable. When the memtable
hits a threshold, it’s flushed to data files on disk (let’s call the first one “1”,
second “2” and so on)

Over time we build up multiple data files on disk - when Cassandra reads, it will merge data
in those files to give you the result you expect, choosing the latest value for each column

But it’s usually wasteful to lots of files around, and that merging is expensive, so compaction
combines those data files behind the scenes in a background thread.

By default they’re combined when 4 or more files are approximately the same size, so if
your write rate is such that you fill and flush the memtable every 5 minutes, compaction will
likely happen at least every 20 minutes (sometimes more). This is called size tiered compaction;
there are 4 strategies but size tiered is default and easiest to understand.

You’re seeing mostly writes because the reads are likely in page cache (the kernel doesn’t
need to go to disk to read the files, it’s got them in memory for serving normal reads).

-- 
Jeff Jirsa


> On Aug 3, 2018, at 12:30 AM, Mihai Stanescu <mihai.stanescu@gmail.com> wrote:
> 
> Hi all,
> 
> I am perftesting cassandra over a longrun in a cluster of 8 nodes and i noticed the rate
of service drops. 
> Most of the nodes have the CPU between 40-65% however one of the nodes has a higher CPU
and also started performing a lot of read IOPS as seen in the image. (green is read IOPS)
> 
> My test has a mixed rw scenario.
> 1. insert row
> 2. after 60 seconds read row
> 3. delete row. 
> 
> The rate of inserts is bigger than the rate of deletes so some delete will not happen.
> 
> I have checked the client it it does not accumulate RAM, GC is a straight line so o don't
understand whats going on.
> 
> Any hints?
> 
> Regards,
> MIhai
> 
> <image.png>
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message