cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stu Hood" <stuart.h...@rackspace.com>
Subject Re: OOM Exception
Date Sun, 13 Dec 2009 18:39:41 GMT
PS: If this turns out to actually be the problem, I'll open a ticket for it.

Thanks,
Stu

-----Original Message-----
From: "Stu Hood" <stuart.hood@rackspace.com>
Sent: Sunday, December 13, 2009 12:28pm
To: cassandra-user@incubator.apache.org
Subject: Re: OOM Exception

With 248G per box, you probably have slightly more than 1/2 billion items?

One current implementation detail in Cassandra is that it loads 128th of the index into memory
for faster lookups. This means you might have something like 4.5 million keys in memory at
the moment.

The '128' value is a constant at SSTable.INDEX_INTERVAL. You should be able to recompile with
'1024' to allow for an 8 times larger database, but understand that this will have a negative
effect on your read performance.

Thanks,
Stu

-----Original Message-----
From: "Dan Di Spaltro" <dan.dispaltro@gmail.com>
Sent: Sunday, December 13, 2009 12:06pm
To: cassandra-user@incubator.apache.org
Subject: Re: OOM Exception

What consistencyLevel are you inserting the elements?  If you do
./bin/nodeprobe -host localhost tpstats on each machine do you see one
metric that has a lot of pending items?

On Sun, Dec 13, 2009 at 8:14 AM, Brian Burruss <bburruss@real.com> wrote:

> another OOM exception.  the only thing interesting about my testing is that
> there are 2 servers, RF=2, W=1, R=1 ... there is 248G of data on each
> server.  I have -Xmx3G assigned to each server
>
> 2009-12-12 22:04:37,436 ERROR [pool-1-thread-309] [Cassandra.java:734]
> Internal error processing get
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.OutOfMemoryError: Java heap space
>        at
> org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:523)
>        at
> org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:373)
>        at
> org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
>        at
> org.apache.cassandra.service.CassandraServer.multigetColumns(CassandraServer.java:265)
>        at
> org.apache.cassandra.service.CassandraServer.multigetInternal(CassandraServer.java:320)
>        at
> org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:253)
>        at
> org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:724)
>        at
> org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:712)
>        at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>
> ________________________________________
> From: Brian Burruss
> Sent: Saturday, December 12, 2009 7:45 AM
> To: cassandra-user@incubator.apache.org
> Subject: OOM Exception
>
> this happened after cassandra was running for a couple of days.  I have
> -Xmx3G on JVM.
>
> is there any other info you need so this makes sense?
>
> thx!
>
>
> 2009-12-11 21:38:37,216 ERROR [HINTED-HANDOFF-POOL:1]
> [DebuggableThreadPoolExecutor.java:157] Error in ThreadPoolExecutor
> java.lang.OutOfMemoryError: Java heap space
>        at
> org.apache.cassandra.io.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>        at
> org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>        at
> org.apache.cassandra.io.SSTableWriter.<init>(SSTableWriter.java:53)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:911)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:855)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.doMajorCompactionInternal(ColumnFamilyStore.java:698)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.doMajorCompaction(ColumnFamilyStore.java:670)
>        at
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:190)
>        at
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:75)
>        at
> org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:249)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
>



-- 
Dan Di Spaltro





Mime
View raw message