cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Cassandra crashes....
Date Tue, 22 Aug 2017 14:57:53 GMT
You typically don't want to set the eden space when you're using G1

-- 
Jeff Jirsa


> On Aug 22, 2017, at 7:42 AM, Thakrar, Jayesh <jthakrar@conversantmedia.com> wrote:
> 
> Hi All,
>  
> We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping the user group
for their experiences.
>  
> Our usage profile is  batch jobs that load millions of rows to Cassandra every hour.
> And there are similar period batch jobs that read millions of rows and do some processing,
outputting the result to HDFS (no issues with HDFS).
>  
> We often seen Cassandra daemons crash.
> Key points of our environment are:
> Pretty good servers: 54 cores (with hyperthreading), 256 GB RAM, 3.2 TB SSD drive
> Compaction: TWCS compaction with 7 day windows as the data retention period is limited
- about 120 days.
> JDK: Java 1.8.0.71 and G1 GC
> Heap Size: 16 GB
> Large SSTables: 50 GB to 300+ GB
> 
> We see the daemons crash after some back-to-back long GCs (1.5 to 3.5 seconds).
> Note that we had set the target for GC pauses to be 200 ms
>  
> We have been somewhat able to tame the crashes by updating the TWCS compaction properties
> to have min/max compaction sstables = 4 and by drastically reducing the size of the New/Eden
space (to 5% of heap space = 800 MB).
> Its been about 12 hours and our stop-the-world gc pauses are under 90 ms.
> Since the servers have more than sufficient resources, we are not seeing any noticeable
performance impact.
>  
> Is this kind of tuning normal/expected?
>  
> Thanks,
> Jayesh
>  

Mime
View raw message