What consistencyLevel are you inserting the elements?  If you do ./bin/nodeprobe -host localhost tpstats on each machine do you see one metric that has a lot of pending items?

On Sun, Dec 13, 2009 at 8:14 AM, Brian Burruss <bburruss@real.com> wrote:
another OOM exception.  the only thing interesting about my testing is that there are 2 servers, RF=2, W=1, R=1 ... there is 248G of data on each server.  I have -Xmx3G assigned to each server

2009-12-12 22:04:37,436 ERROR [pool-1-thread-309] [Cassandra.java:734] Internal error processing get
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
       at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:523)
       at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:373)
       at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
       at org.apache.cassandra.service.CassandraServer.multigetColumns(CassandraServer.java:265)
       at org.apache.cassandra.service.CassandraServer.multigetInternal(CassandraServer.java:320)
       at org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:253)
       at org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:724)
       at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:712)
       at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
       at java.lang.Thread.run(Thread.java:636)

________________________________________
From: Brian Burruss
Sent: Saturday, December 12, 2009 7:45 AM
To: cassandra-user@incubator.apache.org
Subject: OOM Exception

this happened after cassandra was running for a couple of days.  I have -Xmx3G on JVM.

is there any other info you need so this makes sense?

thx!


2009-12-11 21:38:37,216 ERROR [HINTED-HANDOFF-POOL:1] [DebuggableThreadPoolExecutor.java:157] Error in ThreadPoolExecutor
java.lang.OutOfMemoryError: Java heap space
       at org.apache.cassandra.io.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
       at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
       at org.apache.cassandra.io.SSTableWriter.<init>(SSTableWriter.java:53)
       at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:911)
       at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:855)
       at org.apache.cassandra.db.ColumnFamilyStore.doMajorCompactionInternal(ColumnFamilyStore.java:698)
       at org.apache.cassandra.db.ColumnFamilyStore.doMajorCompaction(ColumnFamilyStore.java:670)
       at org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:190)
       at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:75)
       at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:249)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
       at java.lang.Thread.run(Thread.java:636)



--
Dan Di Spaltro