incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom van den Berge <...@drillster.com>
Subject Re: OutOfMemory Java Heap Space error on startup...
Date Wed, 04 Dec 2013 19:39:29 GMT
To start up your node again, you could delete the stored key caches (
/var/lib/cassandra/saved_caches/*).

Regards,
Tom


On Wed, Dec 4, 2013 at 7:46 PM, Krishna Chaitanya <bnsk1990rulz@gmail.com>wrote:

> Hey Nate,
>          Thanks for the reply. The link was really good...!!! Looking
> forward to making the necessary changes and trying this approach.
>
> Thanks.
>
> Regards,
> BNSK.
>
>
> On Wed, Dec 4, 2013 at 9:00 AM, Nate McCall <nate@thelastpickle.com>wrote:
>
>> For a limited memory environment, take a look at the following:
>>
>> http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/
>>
>>
>>
>>
>> On Wed, Dec 4, 2013 at 11:05 AM, Krishna Chaitanya <
>> bnsk1990rulz@gmail.com> wrote:
>>
>>> Hello,
>>>          I am currently using Cassandra-2.0.0 on OpenSuse for storing
>>> netflow packets that are seen on my ethernet interface. I deliberately
>>> tried to test Cassandra with heavy data and it ran fine for about 30 mins
>>> after which it crashed  with OutOfMemory error. I set up a two-node cluster
>>> to which this data is getting stored with replication_factor 1. Now,
>>> Cassandra is not even starting up. The log is given below for your
>>> reference.
>>>                                                             Can I solve
>>> this problem by tweaking JVM OPTS? If yes, which all and how? How can I be
>>> sure that it is not someother issue like corrupted commit log headers, etc.
>>> so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5
>>> machine with 4G RAM.
>>>
>>> Here is the output when I try to start Cassandra:-
>>>
>>> linux-0cpn:~/bnsk/
>>> experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f &
>>> [1] 984
>>> linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin #
>>> ./../conf/cassandra-env.sh: line 137: elseError:: command not found
>>> xss =  -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
>>> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M
>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k
>>>  INFO 03:46:43,178 Logging initialized
>>>  INFO 03:46:43,188 32bit JVM detected.  It is recommended to run
>>> Cassandra on a 64bit JVM for better performance.
>>>  INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server
>>> VM/1.7.0_45
>>>  INFO 03:46:43,189 Heap size: 1046937600/1046937600
>>>  INFO 03:46:43,189 Classpath:
>>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar
>>>  INFO 03:46:43,191 JNA not found. Native methods will be disabled.
>>>  INFO 03:46:43,199 Loading settings from
>>> file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml
>>>  INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data]
>>>  INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog
>>>  INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard,
>>> indexAccessMode is standard
>>>  INFO 03:46:43,418 disk_failure_policy is stop
>>>  INFO 03:46:43,422 Global memtable threshold is enabled at 332MB
>>>  INFO 03:46:43,510 Not using multi-threaded compaction
>>>  INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs.
>>>  INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds
>>> (going to save all keys).
>>>  INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs
>>>  INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to
>>> save all keys).
>>>  INFO 03:46:43,755 Initializing system.schema_triggers
>>>  INFO 03:46:43,768 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104
>>> (57 bytes)
>>>  INFO 03:46:43,768 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105
>>> (57 bytes)
>>>  INFO 03:46:43,769 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103
>>> (57 bytes)
>>>  INFO 03:46:43,790 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db
>>>  INFO 03:46:43,798 Initializing system.batchlog
>>>  INFO 03:46:43,800 Initializing system.peer_events
>>>  INFO 03:46:43,804 Initializing system.compactions_in_progress
>>>  INFO 03:46:43,805 Opening
>>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22
>>> (177 bytes)
>>>  INFO 03:46:43,805 Opening
>>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23
>>> (42 bytes)
>>>  INFO 03:46:43,809 Initializing system.hints
>>>  INFO 03:46:43,812 Initializing system.schema_keyspaces
>>>  INFO 03:46:43,815 Opening
>>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493
>>> (554 bytes)
>>>  INFO 03:46:43,816 Opening
>>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494
>>> (473 bytes)
>>>  INFO 03:46:43,818 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
>>>  INFO 03:46:43,821 Initializing system.range_xfers
>>>  INFO 03:46:43,824 Initializing system.schema_columnfamilies
>>>  INFO 03:46:43,826 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535
>>> (77 bytes)
>>>  INFO 03:46:43,827 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534
>>> (75 bytes)
>>>  INFO 03:46:43,833 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533
>>> (8589 bytes)
>>>  INFO 03:46:43,841 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid984.hprof ...
>>> Heap dump file created [15493326 bytes in 0.068 secs]
>>> ERROR 03:46:43,995 Exception encountered during startup
>>> java.lang.OutOfMemoryError: Java heap space
>>>     at
>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>>>     at
>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>>>     at
>>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>>>     at
>>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>>>     at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>>>     at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>>>     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>>>     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>>>     at
>>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>>> java.lang.OutOfMemoryError: Java heap space
>>>     at
>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>>>     at
>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>>>     at
>>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>>>     at
>>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>>>     at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>>>     at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>>>     at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>>>     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>>>     at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>>>     at
>>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>>>     at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>>> Exception encountered during startup: Java heap space
>>>
>>>
>>> Thanks in advance...
>>> --
>>> Regards,
>>> BNSK
>>> *. *
>>>
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>
>
> --
> Regards,
> BNSK
> *. *
>



-- 

Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands

+31 30 755 5330

Open your free account at www.drillster.com

Mime
View raw message