cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anuj (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
Date Mon, 04 May 2015 17:49:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526919#comment-14526919
] 

Anuj edited comment on CASSANDRA-8150 at 5/4/15 5:48 PM:
---------------------------------------------------------

We have write heavy workload and used to face promotion failures/long gc pauses with Cassandra
2.0.x. I am not into code yet but I think that memtable and compaction related objects have
mid-life and write heavy workload is not suitable for generation collection by default. So,
we tuned JVM to make sure that minimum objects are promoted to Old Gen and achieved great
success in that:

MAX_HEAP_SIZE="12G"
HEAP_NEWSIZE="3G"
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=20
-XX:CMSInitiatingOccupancyFraction=70
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=20"
JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions"
JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity"
JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs"
JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=32768"
JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"
JVM_OPTS="$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=30000"
JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=2000"
JVM_OPTS="$JVM_OPTS -XX:+CMSEdenChunksRecordAlways"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelInitialMarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"

We also think that default total_memtable_space_in_mb=1/4 heap is too much for write heavy
loads. By default, young gen is also 1/4 heap.We reduced it to 1000mb in order to make sure
that memtable related objects dont stay in memory for too long. Combining this with SurvivorRatio=2
and MaxTenuringThreshold=20 did the job well. GC was very consistent. No Full GC observed.

Environment: 3 node cluster with each node having 24cores,64G RAM and SSDs in RAID5.

We are making around 12k writes/sec in 5 cf (one with 4 sec index) and 2300 reads/sec on each
node of 3 node cluster. 2 CFs have wide rows with max data of around 100mb per row


was (Author: eanujwa):
We have write heavy workload and used to face promotion failures/long gc pauses with Cassandra
2.0.x. I am not into code yet but I think that memtable and compaction related objects have
mid-life and write heavy workload is not suitable for generation collection by default. So,
we tuned JVM to make sure that minimum objects are promoted to Old Gen and achieved great
success in that:

MAX_HEAP_SIZE="12G"
HEAP_NEWSIZE="3G"
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=20
-XX:CMSInitiatingOccupancyFraction=70
JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=20"
JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions"
JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity"
JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs"
JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=32768"
JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"
JVM_OPTS="$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=30000"
JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=2000"
JVM_OPTS="$JVM_OPTS -XX:+CMSEdenChunksRecordAlways"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelInitialMarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"

We also think that default total_space_in_mb=1/4 heap is too much for write heavy loads. By
default, young gen is also 1/4 heap.We reduced it to 1000mb in order to make sure that memtable
related objects dont stay in memory for too long. Combining this with SurvivorRatio=2 and
MaxTenuringThreshold=20 did the job well. GC was very consistent. No Full GC observed.

Environment: 3 node cluster with each node having 24cores,64G RAM and SSDs in RAID5.

We are making around 12k writes/sec in 5 cf (one with 4 sec index) and 2300 reads/sec on each
node of 3 node cluster. 2 CFs have wide rows with max data of around 100mb per row

> Revaluate Default JVM tuning parameters
> ---------------------------------------
>
>                 Key: CASSANDRA-8150
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Config
>            Reporter: Matt Stump
>            Assignee: Ryan McGuire
>         Attachments: upload.png
>
>
> It's been found that the old twitter recommendations of 100m per core up to 800m is harmful
and should no longer be used.
> Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is
debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better
for releases greater than 2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message