incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Yves LEBLEU <jleb...@gmail.com>
Subject Out of Memory every 2 weeks
Date Mon, 14 Mar 2011 15:27:05 GMT
Sorry to create a new thread about Out of Memory problem, but I
checked all other threads and did not find the answer.

We have a running cluster of 2 cassandra nodes replication factor = 2
on red hat 4.8 32 bits with 4 G of memory  where we run periodicaly
out of memory (every 2 weeks)  and both nodes are crashing (trace at
the end of the file).

We are running 0.6.12 (do not had the time to upgrade to 0.7.3) and we
followed the memory configuration from the wiki :

memtable_throughput_in_mb * 3 * number of hot CFs + 1G + internal caches

memtable_throughput_in_mb = 16
We have 11 column families

So if my calculation is right, I need  16*3*11 + 1G + internal caches
: 528 Mbytes + 1G + internal caches, so more than 1,5 G of heap memory

We start with 2G

Here is the cassandra.in.sh parameters

JVM_OPTS=" \
        -ea \
        -Xms2000M \
        -Xmx2000M \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:SurvivorRatio=8 \
        -XX:MaxTenuringThreshold=1 \
        -XX:CMSInitiatingOccupancyFraction=75 \
        -XX:+UseCMSInitiatingOccupancyOnly \
        -XX:+HeapDumpOnOutOfMemoryError \


The question is I don't really understand the configuration problem,
if some body have any clue of what we could do, or what we should
monitor to avoid the problem, as it is very difficult to reproduce the
problem as it does not happen very often.

Thanks for any help.
Jean-Yves


<---------------------below the stack trace
------------------------------------------>


INFO [FLUSH-WRITER-POOL:1] 2011-03-04 17:59:45,952 Memtable.java (line 152)
Writing Memtable-HintsColumnFamily@15830055(2280 bytes, 240 operations)
 INFO [FLUSH-WRITER-POOL:1] 2011-03-04 17:59:45,972 Memtable.java (line 166)
Completed flushing
/opt/database/data/system/HintsColumnFamily-164-Data.db (255 bytes)
 INFO [FLUSH-TIMER] 2011-03-04 18:17:47,802 ColumnFamilyStore.java (line 478)
VoiceMail has reached its threshold; switching in a fresh Memtable at
CommitLogContext(file='/opt/database/commitlog/CommitLog-1297775500410.log',
position=10407973)
 INFO [FLUSH-TIMER] 2011-03-04 18:17:47,803 ColumnFamilyStore.java (line 748)
Enqueuing flush of Memtable-VoiceMail@29468188(3000 bytes, 120 operations)
 INFO [FLUSH-WRITER-POOL:1] 2011-03-04 18:17:47,803 Memtable.java (line 152)
Writing Memtable-VoiceMail@29468188(3000 bytes, 120 operations)
ERROR [FLUSH-WRITER-POOL:1] 2011-03-04 18:17:47,824 CassandraDaemon.java (line
87) Uncaught exception in thread Thread[FLUSH-WRITER-POOL:1,5,main]
java.lang.OutOfMemoryError
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:209)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:246)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:389)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:365)
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
        at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064)
        at org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:94)
        at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:162)
        at org.apache.cassandra.db.Memtable.access$000(Memtable.java:46)
        at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:178)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
 INFO [main] 2011-03-07 17:46:09,454 CLibrary.java (line 47) JNA not found.
Native methods will be disabled.
 INFO [main] 2011-03-07 17:46:09,768 DatabaseDescriptor.java (line 277)
DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard

Mime
View raw message