incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ηŽ‹δΈ€ι”‹" <wangyif...@aspire-tech.com>
Subject Frequent crashes
Date Wed, 14 Jul 2010 09:38:39 GMT
Hi, 

Has anyboy done any memory usage analysis for cassandra?

How much memory does cassandra need to manager 300G of data load? How much extra memory will
be needed when doing compaction?

Regarding mmap, memory usage will be determined by the OS so it has nothing to do with the
heap size of JVM, am I right?

I have a cassandra cluster of 13 nodes, each with 200~300g data.
JVM settings
JVM_OPTS=" \
        -ea \
        -Xms6G \
        -Xmx6G \
        -XX:TargetSurvivorRatio=90 \
        -XX:+AggressiveOpts \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:+HeapDumpOnOutOfMemoryError \
        -XX:SurvivorRatio=128 \
        -XX:MaxTenuringThreshold=0 \
        -XX:+PrintGC -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \
        -Dcom.sun.management.jmxremote.port=4993 \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.authenticate=false"
KeysCache settings for 3 column families are 5,000,000  1,000,000  1,000,000

some nodes run for 1 to 2 days, and then gets very slow, due to bad gc performance, then crashed.
This happed quite a lot, almost every day. 
Here is a fragment of the gc.log

 (concurrent mode failure): 6014591K->6014591K(6014592K), 25.4846400 secs] 6289343K->6282274K(6289344K),
[CMS Perm : 17290K->17287K(28988K)], 25.4848970 secs] [Times: user=37.76 sys=0.12, real=25.49
secs] 
69695.771: [Full GC 69695.771: [CMS: 6014591K->6014591K(6014592K), 21.0911470 secs] 6289343K->6282177K(6289344K),
[CMS Perm : 17287K->17287K(28988K)], 21.0913910 secs] [Times: user=21.01 sys=0.12, real=21.09
secs] 
69716.902: [GC [1 CMS-initial-mark: 6014591K(6014592K)] 6287620K(6289344K), 0.2759980 secs]
[Times: user=0.28 sys=0.00, real=0.28 secs] 
69717.178: [CMS-concurrent-mark-start]
69717.203: [Full GC 69717.203: [CMS69721.345: [CMS-concurrent-mark: 4.152/4.167 secs] [Times:
user=16.64 sys=0.01, real=4.17 secs] 
 (concurrent mode failure): 6014592K->6014591K(6014592K), 25.3649330 secs] 6289343K->6282200K(6289344K),
[CMS Perm : 17287K->17287K(28988K)], 25.3651670 secs] [Times: user=37.67 sys=0.13, real=25.37
secs] 
69742.598: [Full GC 69742.598: [CMS: 6014591K->6014592K(6014592K), 21.0942430 secs] 6289343K->6282398K(6289344K),
[CMS Perm : 17290K->17287K(28988K)], 21.0944950 secs] [Times: user=21.00 sys=0.12, real=21.10
secs] 
69763.721: [Full GC 69763.721: [CMS: 6014592K->6014591K(6014592K), 21.0978230 secs] 6289343K->6282553K(6289344K),
[CMS Perm : 17290K->17287K(28988K)], 21.0980600 secs] [Times: user=20.99 sys=0.12, real=21.09
secs] 
69784.830: [GC [1 CMS-initial-mark: 6014591K(6014592K)] 6287995K(6289344K), 0.2765360 secs]
[Times: user=0.28 sys=0.00, real=0.28 secs] 
69785.107: [CMS-concurrent-mark-start]
69785.123: [Full GC 69785.123: [CMS69789.244: [CMS-concurrent-mark: 4.132/4.136 secs] [Times:
user=16.49 sys=0.03, real=4.13 secs] 
 (concurrent mode failure): 6014591K->6014591K(6014592K), 26.0883660 secs] 6289343K->6282549K(6289344K),
[CMS Perm : 17290K->17287K(28988K)], 26.0886060 secs] [Times: user=38.28 sys=0.15, real=26.09
secs] 

Anybody got an idea?

Mime
View raw message