incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Hill <jasonhill...@gmail.com>
Subject Java heap space on Cassandra start up version 1.0.10
Date Fri, 06 Jul 2012 23:00:39 GMT
Hello friends,

I'm getting a:

ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.OutOfMemoryError: Java heap space

error when I start Cassandra. This node was running fine and after
some server work/upgrades it started throwing this error when I start
the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to
see if it would help, but I get the same error. I've removed some of
the column families from my keyspace directory to see if I can get it
to start without the heap space error and with some combinations it
will run. However, I'd like to get it running with all my colFams and
wonder if someone could give me some advice on what might be causing
my error. It doesn't seem to be related to compaction, if I am reading
the log correctly, and most of the help I've found on this topic deals
with compaction. I'm thinking that my 2 column families should not be
enough to fill my heap, but I am at a loss as to what I should try
next?

Thanks for your consideration.

output.log:

 INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
VM/1.6.0_26
 INFO 22:50:26,322 Heap size: 5905580032/5905580032
 INFO 22:50:26,322 Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
 INFO 22:50:28,586 JNA mlockall successful
 INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml
DEBUG 22:50:28,677 Syncing log with a period of 10000
 INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
 INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB
DEBUG 22:50:28,761 setting auto_bootstrap to true
<snip>
DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data
DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog
DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches
DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,818 Removing compacted SSTable files from
Versions.76657273696f6e (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,827 Removing compacted SSTable files from
HintsColumnFamily (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:28,833 Initializing system.NodeIdInfo
DEBUG 22:50:28,839 Starting CFS NodeIdInfo
DEBUG 22:50:28,868 Creating IntervalNode from []
DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1
DEBUG 22:50:28,871 Initializing system.Versions
DEBUG 22:50:28,873 Starting CFS Versions
 INFO 22:50:28,877 Opening
/var/lib/cassandra/data/system/Versions-hd-5 (248 bytes)
DEBUG 22:50:28,879 Load metadata for
/var/lib/cassandra/data/system/Versions-hd-5
 INFO 22:50:28,880 Opening
/var/lib/cassandra/data/system/Versions-hd-6 (248 bytes)
DEBUG 22:50:28,880 Load metadata for
/var/lib/cassandra/data/system/Versions-hd-6
DEBUG 22:50:28,897 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Versions-hd-5: 20 ms.
DEBUG 22:50:28,898 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Versions-hd-6: 18 ms.
DEBUG 22:50:28,899 key cache contains 0/0 keys
DEBUG 22:50:28,899 key cache contains 0/0 keys
 INFO 22:50:28,899 Opening
/var/lib/cassandra/data/system/Versions-hd-7 (248 bytes)
DEBUG 22:50:28,900 Load metadata for
/var/lib/cassandra/data/system/Versions-hd-7
DEBUG 22:50:28,901 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Versions-hd-7: 2 ms.
DEBUG 22:50:28,901 key cache contains 0/0 keys
DEBUG 22:50:28,903 Creating IntervalNode from
[Interval(DecoratedKey(93220794208128599841715671226150005828,
746872696674), DecoratedKey(135285944860343992175601105924967452217,
63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828,
746872696674), DecoratedKey(135285944860343992175601105924967452217,
63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828,
746872696674), DecoratedKey(135285944860343992175601105924967452217,
63716c))]
DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-5
to list of files tracked for system.Versions
DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-6
to list of files tracked for system.Versions
DEBUG 22:50:28,906 adding /var/lib/cassandra/data/system/Versions-hd-7
to list of files tracked for system.Versions
DEBUG 22:50:28,906 KeyCache capacity for Versions is 3
DEBUG 22:50:28,906 Initializing system.IndexInfo
DEBUG 22:50:28,908 Starting CFS IndexInfo
DEBUG 22:50:28,910 Creating IntervalNode from []
DEBUG 22:50:28,910 KeyCache capacity for IndexInfo is 1
DEBUG 22:50:28,910 Initializing system.Schema
DEBUG 22:50:28,912 Starting CFS Schema
 INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-46
(3831 bytes)
DEBUG 22:50:28,916 Load metadata for /var/lib/cassandra/data/system/Schema-hd-46
DEBUG 22:50:28,918 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Schema-hd-46: 3 ms.
 INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-45
(187275 bytes)
DEBUG 22:50:28,918 Load metadata for /var/lib/cassandra/data/system/Schema-hd-45
DEBUG 22:50:28,918 key cache contains 0/0 keys
DEBUG 22:50:28,920 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Schema-hd-45: 5 ms.
DEBUG 22:50:28,921 key cache contains 0/0 keys
DEBUG 22:50:28,922 Creating IntervalNode from
[Interval(DecoratedKey(124804735337540159479107746638263794797,
4c617374204d6967726174696f6e),
DecoratedKey(168749730613100185197104044460479972233,
62626139343461302d633734632d313165312d303030302d353662363634353961646666)),
Interval(DecoratedKey(2957058266746302108707016693211230853,
64323438663637302d633438662d313165312d303030302d366637313036623962396666),
DecoratedKey(167756402517845603382936953195099512584,
65306532376534302d633734622d313165312d303030302d353662363634353961646666))]
DEBUG 22:50:28,922 adding /var/lib/cassandra/data/system/Schema-hd-46
to list of files tracked for system.Schema
DEBUG 22:50:28,923 adding /var/lib/cassandra/data/system/Schema-hd-45
to list of files tracked for system.Schema
DEBUG 22:50:28,923 KeyCache capacity for Schema is 2
DEBUG 22:50:28,924 Initializing system.Migrations
DEBUG 22:50:28,926 Starting CFS Migrations
 INFO 22:50:28,929 Opening
/var/lib/cassandra/data/system/Migrations-hd-45 (323713 bytes)
DEBUG 22:50:28,930 Load metadata for
/var/lib/cassandra/data/system/Migrations-hd-45
 INFO 22:50:28,931 Opening
/var/lib/cassandra/data/system/Migrations-hd-46 (7945 bytes)
DEBUG 22:50:28,931 Load metadata for
/var/lib/cassandra/data/system/Migrations-hd-46
DEBUG 22:50:28,932 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Migrations-hd-45: 3 ms.
DEBUG 22:50:28,933 INDEX LOAD TIME for
/var/lib/cassandra/data/system/Migrations-hd-46: 2 ms.
DEBUG 22:50:28,933 key cache contains 0/0 keys
DEBUG 22:50:28,933 key cache contains 0/0 keys
DEBUG 22:50:28,935 Creating IntervalNode from
[Interval(DecoratedKey(159559397954378837828954138596956659794,
4d6967726174696f6e73204b6579),
DecoratedKey(159559397954378837828954138596956659794,
4d6967726174696f6e73204b6579)),
Interval(DecoratedKey(159559397954378837828954138596956659794,
4d6967726174696f6e73204b6579),
DecoratedKey(159559397954378837828954138596956659794,
4d6967726174696f6e73204b6579))]
DEBUG 22:50:28,936 adding
/var/lib/cassandra/data/system/Migrations-hd-45 to list of files
tracked for system.Migrations
DEBUG 22:50:28,937 adding
/var/lib/cassandra/data/system/Migrations-hd-46 to list of files
tracked for system.Migrations
DEBUG 22:50:28,937 KeyCache capacity for Migrations is 2
DEBUG 22:50:28,938 Initializing system.LocationInfo
DEBUG 22:50:28,940 Starting CFS LocationInfo
 INFO 22:50:28,943 Opening
/var/lib/cassandra/data/system/LocationInfo-hd-17 (357 bytes)
DEBUG 22:50:28,944 Load metadata for
/var/lib/cassandra/data/system/LocationInfo-hd-17
DEBUG 22:50:28,947 INDEX LOAD TIME for
/var/lib/cassandra/data/system/LocationInfo-hd-17: 4 ms.
DEBUG 22:50:28,948 key cache contains 0/0 keys
DEBUG 22:50:28,948 Creating IntervalNode from
[Interval(DecoratedKey(61078635599166706937511052402724559481, 4c),
DecoratedKey(136380791207214638541884810421800860241,
426f6f747374726170))]
DEBUG 22:50:28,949 adding
/var/lib/cassandra/data/system/LocationInfo-hd-17 to list of files
tracked for system.LocationInfo
DEBUG 22:50:28,949 KeyCache capacity for LocationInfo is 1
DEBUG 22:50:28,949 Initializing system.HintsColumnFamily
DEBUG 22:50:28,951 Starting CFS HintsColumnFamily
DEBUG 22:50:28,953 Creating IntervalNode from []
DEBUG 22:50:28,954 KeyCache capacity for HintsColumnFamily is 1
DEBUG 22:50:28,959 collectTimeOrderedData
DEBUG 22:50:28,975 collectTimeOrderedData
 INFO 22:50:28,977 Loading schema version bba944a0-c74c-11e1-0000-56b66459adff
DEBUG 22:50:28,978 collectAllData
DEBUG 22:50:28,983 collecting 0 of 2147483647:
Avro/Schema:false:2799@1341566817771
DEBUG 22:50:28,983 collecting 1 of 2147483647: Innovari:false:788@1341566817771
DEBUG 22:50:29,143 Adding
org.apache.cassandra.config.CFMetaData@37e55794[cfId=1017,ksName=Innovari,cfName=Temps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@31d8d237,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>]
to cfIdMap
DEBUG 22:50:29,144 Adding
org.apache.cassandra.config.CFMetaData@3d46e381[cfId=1004,ksName=Innovari,cfName=TICStatusCurrent,cfType=Standard,comparator=org.apache.cassandra.db.marshal.AsciiType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@79024994,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>]
to cfIdMap
DEBUG 22:50:29,145 Removing compacted SSTable files from NodeIdInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,147 Removing compacted SSTable files from Versions (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,149 Removing compacted SSTable files from
Versions.76657273696f6e (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,150 Removing compacted SSTable files from IndexInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,151 Removing compacted SSTable files from Schema (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,153 Removing compacted SSTable files from Migrations
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,155 Removing compacted SSTable files from LocationInfo
(see http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,157 Removing compacted SSTable files from
HintsColumnFamily (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,158 Removing compacted SSTable files from Temps (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,160 Removing compacted SSTable files from
TICStatusCurrent (see
http://wiki.apache.org/cassandra/MemtableSSTable)
DEBUG 22:50:29,161 opening keyspace system
DEBUG 22:50:29,161 opening keyspace Innovari
DEBUG 22:50:29,162 Configured datacenter replicas are datacenter1:1
DEBUG 22:50:29,162 Initializing Innovari.Temps
DEBUG 22:50:29,164 Starting CFS Temps
 INFO 22:50:29,166 Opening
/var/lib/cassandra/data/Innovari/Temps-g-117 (156 bytes)
DEBUG 22:50:29,166 Load metadata for
/var/lib/cassandra/data/Innovari/Temps-g-117
 INFO 22:50:29,166 Opening
/var/lib/cassandra/data/Innovari/Temps-g-118 (156 bytes)
DEBUG 22:50:29,167 Load metadata for
/var/lib/cassandra/data/Innovari/Temps-g-118
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/lib/cassandra/java_1341615025.hprof ...
Heap dump file created [47294429 bytes in 0.418 secs]
ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.OutOfMemoryError: Java heap space
	at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:222)
	at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:204)
	at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:194)
	at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:155)
	at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Service exit with a return value of 100

Mime
View raw message