incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "olek.stasiak@gmail.com" <olek.stas...@gmail.com>
Subject OOM while reading key cache
Date Fri, 08 Nov 2013 10:31:53 GMT
Hello,
I'm facing OOM on reading key_cache
Cluster conf is as follows:
-6 machines which 8gb RAM each and three 150GB disks each
-default heap configuration
-deafult key cache configuration
-the biggest keyspace has abt 500GB size (RF: 2, so in fact there is
250GB of raw data).

After upgrading first of the machines from 1.2.11 to 2.0.2 i've recieved error:
 INFO [main] 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114)
reading saved cache
/home/synat/nosql_filesystem/cassandra/data/saved_caches/production_storage-METADATA-KeyCache-b.db
ERROR [main] 2013-11-08 10:53:16,895 CassandraDaemon.java (line 478)
Exception encountered during startup
java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
        at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:352)
        at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
        at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:264)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:409)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:381)
        at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
        at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)


Error appears every start, so I've decided to disable key cache (this
was not helpful) and temporarily moved key cache out of cache folder
(file was of size 13M). That helps in starting node, but this is only
workaround and it's not demanded configuration. Anyone has any idea
what is the real cause of problem with oom?
best regards
Aleksander
ps. I've still 5 nodes to upgrade, I'll inform if problem apperas on the rest.

Mime
View raw message