cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Ran into an issue where Cassandra Crashed when running out of heap space
Date Tue, 20 Jul 2010 19:09:15 GMT
> CassandraDaemon.java (line 83) Uncaught exception in thread
> Thread[pool-1-thread-37895,5,main]
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
>         at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1116)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)

Did someone send garbage on the wrong port, causing thrift to try to
read some huge string in the RPC layer? There is a bug filed about
this upstream with thrift but I couldn't find it now.

> Is there a problem with Garbage Collection? Should I restart my
> servers every few days?

No. The CMS collector will be subject to some delayed slow growth as a
result of fragmentation in old space (just like malloc would), but in
general you should expect to see memory use stabilize. Likely, you're
either causing an out of memory condition by some kind of explosive
memory use (such as the garbage-on-thrift-port, or some humongous
mutation request, etc) or you are legitimately using too much memory
in which case you may look into adjusting cache sizes and memtable
flushing thresholds.

If your version of cassandra logs GC:s (I'm not sure if 0.6.x does
it), legitimate heap growth should be obvious from GC messages in the
cassandra system log. You can also run with -XX:+PrintGC and
-XX:+PrintGCDetails to get GC logs from the JVM on stdout, and with
-Xloggc:path/to/log to redirect said GC output to file.

You may want to use something like VisualVM or JConsole to attach to
cassandra on monitor memory usage if you prefer that to looking at the
log output.

-- 
/ Peter Schuller

Mime
View raw message