incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aram Ayazyan <ayaz...@gmail.com>
Subject OutOfMemory exceptions w/ Cassandra 0.6.8
Date Thu, 02 Dec 2010 00:28:02 GMT
Hi,

We have a small cluster of 3 Cassandra servers running w/ full
replication. Every once in a while we get an OutOfMemory exception and
have to restart servers. Sometimes just restarting doesn’t do it and
we have to clean the commitlog or data directory.

We are running Cassandra 0.6.8. There is only 1 keyspace and 3 column
families. There are less than 1000 keys across all column families.
There is roughly 1 write request per second and 1 read request. Each
server is allocated 1GB.  Size of all files in data directory of the
only column family is ~300MB. MemtableThroughputInMB is throttled way
down to 2 and BinaryMemtableThroughputInMB to 8 (w/ higher values we
were running out of memory extremely fast, this way it works for a
couple of days w/o crashing).

Last time this issue happened, I didn’t clear the commitlog/data
folders, enabled gc logging and restarted Cassandra. It crashes really
fast, but what is really strange is that it seems like it still has
plenty of memory when the error happens, last 3 lines from gc log:
21.408: [GC 437098K->436592K(1046464K), 0.0986800 secs]
21.520: [GC 453616K->453117K(1046464K), 0.0967770 secs]
21.629: [GC 470141K->469436K(1046464K), 0.0383520 secs]
The full log is here: http://pastebin.com/XGRSRcBd

I’ve tried increasing the memory up to 1.5GB, but it still doesn’t start.

Any ideas what might be the problem here?

Thank you,
Aram

Mime
View raw message