Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 4 May 2015 17:48:12 +0000 (UTC)
From: "Anuj (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12749305.1413826661000.71844.1430761691987@Atlassian.JIRA>
In-Reply-To: <JIRA.12749305.1413826661000@Atlassian.JIRA>
References: <JIRA.12749305.1413826661000@Atlassian.JIRA>
 <JIRA.12749305.1413826661876@arcas>
Subject: [jira] [Commented] (CASSANDRA-8150) Revaluate Default JVM tuning
 parameters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526919#comment-14526919 ] 

Anuj commented on CASSANDRA-8150:
---------------------------------

We have write heavy workload and used to face promotion failures/long gc pauses with Cassandra 2.0.x. I am not into code yet but I think that memtable and compaction related objects have mid-life and write heavy workload is not suitable for generation collection by default. So, we tuned JVM to make sure that minimum objects are promoted to Old Gen and achieved great success in that:

MAX_HEAP_SIZE="12G"
HEAP_NEWSIZE="3G"
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=20
-XX:CMSInitiatingOccupancyFraction=70
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=20"
JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions"
JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity"
JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs"
JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=32768"
JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"
JVM_OPTS="$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=30000"
JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=2000"
JVM_OPTS="$JVM_OPTS -XX:+CMSEdenChunksRecordAlways"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelInitialMarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"

We also think that default total_space_in_mb=1/4 heap is too much for write heavy loads. By default, young gen is also 1/4 heap.We reduced it to 1000mb in order to make sure that memtable related objects dont stay in memory for too long. Combining this with SurvivorRatio=2 and MaxTenuringThreshold=20 did the job well. GC was very consistent. No Full GC observed.

Environment: 3 node cluster with each node having 24cores,64G RAM and SSDs in RAID5.

We are making around 12k writes/sec in 5 cf (one with 4 sec index) and 2300 reads/sec on each node of 3 node cluster. 2 CFs have wide rows with max data of around 100mb per row

> Revaluate Default JVM tuning parameters
> ---------------------------------------
>
>                 Key: CASSANDRA-8150
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Config
>            Reporter: Matt Stump
>            Assignee: Ryan McGuire
>         Attachments: upload.png
>
>
> It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used.
> Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)