I'm having the same problem, after upgrading from 1.2.3 to 1.2.10. 

I can remember this was a bug that was solved in the 1.0 or 1.1 version some time ago, but apparently it got back.
A workaround is to delete the contents of the saved_caches directory before starting up.


Tom


On Tue, Nov 12, 2013 at 5:15 AM, Aaron Morton <aaron@thelastpickle.com> wrote:
-6 machines which 8gb RAM each and three 150GB disks each
-default heap configuration
With 8GB the default heap is 2GB, try kicking that up to 4GB and a 600 to 800 MB new heap. 

I would guess for the data load  you have 2GB is not enough. 

hope that helps. 

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting

On 8/11/2013, at 11:31 pm, olek.stasiak@gmail.com wrote:

Hello,
I'm facing OOM on reading key_cache
Cluster conf is as follows:
-6 machines which 8gb RAM each and three 150GB disks each
-default heap configuration
-deafult key cache configuration
-the biggest keyspace has abt 500GB size (RF: 2, so in fact there is
250GB of raw data).

After upgrading first of the machines from 1.2.11 to 2.0.2 i've recieved error:
INFO [main] 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114)
reading saved cache
/home/synat/nosql_filesystem/cassandra/data/saved_caches/production_storage-METADATA-KeyCache-b.db
ERROR [main] 2013-11-08 10:53:16,895 CassandraDaemon.java (line 478)
Exception encountered during startup
java.lang.OutOfMemoryError: Java heap space
       at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
       at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
       at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:352)
       at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
       at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:264)
       at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:409)
       at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:381)
       at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
       at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
       at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
       at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274)
       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)


Error appears every start, so I've decided to disable key cache (this
was not helpful) and temporarily moved key cache out of cache folder
(file was of size 13M). That helps in starting node, but this is only
workaround and it's not demanded configuration. Anyone has any idea
what is the real cause of problem with oom?
best regards
Aleksander
ps. I've still 5 nodes to upgrade, I'll inform if problem apperas on the rest.