cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Java heap space on Cassandra start up version 1.0.10
Date Tue, 10 Jul 2012 06:27:05 GMT
You may have a corrupt metadata/statistics sstable component.  You can
try deleting those and restarting.  Cassandra can rebuild that
component if it is missing.

On Fri, Jul 6, 2012 at 6:00 PM, Jason Hill <jasonhillpdx@gmail.com> wrote:
> Hello friends,
>
> I'm getting a:
>
> ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main]
> java.lang.OutOfMemoryError: Java heap space
>
> error when I start Cassandra. This node was running fine and after
> some server work/upgrades it started throwing this error when I start
> the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to
> see if it would help, but I get the same error. I've removed some of
> the column families from my keyspace directory to see if I can get it
> to start without the heap space error and with some combinations it
> will run. However, I'd like to get it running with all my colFams and
> wonder if someone could give me some advice on what might be causing
> my error. It doesn't seem to be related to compaction, if I am reading
> the log correctly, and most of the help I've found on this topic deals
> with compaction. I'm thinking that my 2 column families should not be
> enough to fill my heap, but I am at a loss as to what I should try
> next?
>
> Thanks for your consideration.
>
> output.log:
>
>  INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server
> VM/1.6.0_26
>  INFO 22:50:26,322 Heap size: 5905580032/5905580032
>  INFO 22:50:26,322 Classpath:
> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
>  INFO 22:50:28,586 JNA mlockall successful
>  INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml
> DEBUG 22:50:28,677 Syncing log with a period of 10000
>  INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap,
> indexAccessMode is mmap
>  INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB
> DEBUG 22:50:28,761 setting auto_bootstrap to true
> <snip>
> DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data
> DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog
> DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches
> DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,818 Removing compacted SSTable files from
> Versions.76657273696f6e (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,827 Removing compacted SSTable files from
> HintsColumnFamily (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:28,833 Initializing system.NodeIdInfo
> DEBUG 22:50:28,839 Starting CFS NodeIdInfo
> DEBUG 22:50:28,868 Creating IntervalNode from []
> DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1
> DEBUG 22:50:28,871 Initializing system.Versions
> DEBUG 22:50:28,873 Starting CFS Versions
>  INFO 22:50:28,877 Opening
> /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes)
> DEBUG 22:50:28,879 Load metadata for
> /var/lib/cassandra/data/system/Versions-hd-5
>  INFO 22:50:28,880 Opening
> /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes)
> DEBUG 22:50:28,880 Load metadata for
> /var/lib/cassandra/data/system/Versions-hd-6
> DEBUG 22:50:28,897 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Versions-hd-5: 20 ms.
> DEBUG 22:50:28,898 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Versions-hd-6: 18 ms.
> DEBUG 22:50:28,899 key cache contains 0/0 keys
> DEBUG 22:50:28,899 key cache contains 0/0 keys
>  INFO 22:50:28,899 Opening
> /var/lib/cassandra/data/system/Versions-hd-7 (248 bytes)
> DEBUG 22:50:28,900 Load metadata for
> /var/lib/cassandra/data/system/Versions-hd-7
> DEBUG 22:50:28,901 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Versions-hd-7: 2 ms.
> DEBUG 22:50:28,901 key cache contains 0/0 keys
> DEBUG 22:50:28,903 Creating IntervalNode from
> [Interval(DecoratedKey(93220794208128599841715671226150005828,
> 746872696674), DecoratedKey(135285944860343992175601105924967452217,
> 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828,
> 746872696674), DecoratedKey(135285944860343992175601105924967452217,
> 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828,
> 746872696674), DecoratedKey(135285944860343992175601105924967452217,
> 63716c))]
> DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-5
> to list of files tracked for system.Versions
> DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-6
> to list of files tracked for system.Versions
> DEBUG 22:50:28,906 adding /var/lib/cassandra/data/system/Versions-hd-7
> to list of files tracked for system.Versions
> DEBUG 22:50:28,906 KeyCache capacity for Versions is 3
> DEBUG 22:50:28,906 Initializing system.IndexInfo
> DEBUG 22:50:28,908 Starting CFS IndexInfo
> DEBUG 22:50:28,910 Creating IntervalNode from []
> DEBUG 22:50:28,910 KeyCache capacity for IndexInfo is 1
> DEBUG 22:50:28,910 Initializing system.Schema
> DEBUG 22:50:28,912 Starting CFS Schema
>  INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-46
> (3831 bytes)
> DEBUG 22:50:28,916 Load metadata for /var/lib/cassandra/data/system/Schema-hd-46
> DEBUG 22:50:28,918 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Schema-hd-46: 3 ms.
>  INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-45
> (187275 bytes)
> DEBUG 22:50:28,918 Load metadata for /var/lib/cassandra/data/system/Schema-hd-45
> DEBUG 22:50:28,918 key cache contains 0/0 keys
> DEBUG 22:50:28,920 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Schema-hd-45: 5 ms.
> DEBUG 22:50:28,921 key cache contains 0/0 keys
> DEBUG 22:50:28,922 Creating IntervalNode from
> [Interval(DecoratedKey(124804735337540159479107746638263794797,
> 4c617374204d6967726174696f6e),
> DecoratedKey(168749730613100185197104044460479972233,
> 62626139343461302d633734632d313165312d303030302d353662363634353961646666)),
> Interval(DecoratedKey(2957058266746302108707016693211230853,
> 64323438663637302d633438662d313165312d303030302d366637313036623962396666),
> DecoratedKey(167756402517845603382936953195099512584,
> 65306532376534302d633734622d313165312d303030302d353662363634353961646666))]
> DEBUG 22:50:28,922 adding /var/lib/cassandra/data/system/Schema-hd-46
> to list of files tracked for system.Schema
> DEBUG 22:50:28,923 adding /var/lib/cassandra/data/system/Schema-hd-45
> to list of files tracked for system.Schema
> DEBUG 22:50:28,923 KeyCache capacity for Schema is 2
> DEBUG 22:50:28,924 Initializing system.Migrations
> DEBUG 22:50:28,926 Starting CFS Migrations
>  INFO 22:50:28,929 Opening
> /var/lib/cassandra/data/system/Migrations-hd-45 (323713 bytes)
> DEBUG 22:50:28,930 Load metadata for
> /var/lib/cassandra/data/system/Migrations-hd-45
>  INFO 22:50:28,931 Opening
> /var/lib/cassandra/data/system/Migrations-hd-46 (7945 bytes)
> DEBUG 22:50:28,931 Load metadata for
> /var/lib/cassandra/data/system/Migrations-hd-46
> DEBUG 22:50:28,932 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Migrations-hd-45: 3 ms.
> DEBUG 22:50:28,933 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/Migrations-hd-46: 2 ms.
> DEBUG 22:50:28,933 key cache contains 0/0 keys
> DEBUG 22:50:28,933 key cache contains 0/0 keys
> DEBUG 22:50:28,935 Creating IntervalNode from
> [Interval(DecoratedKey(159559397954378837828954138596956659794,
> 4d6967726174696f6e73204b6579),
> DecoratedKey(159559397954378837828954138596956659794,
> 4d6967726174696f6e73204b6579)),
> Interval(DecoratedKey(159559397954378837828954138596956659794,
> 4d6967726174696f6e73204b6579),
> DecoratedKey(159559397954378837828954138596956659794,
> 4d6967726174696f6e73204b6579))]
> DEBUG 22:50:28,936 adding
> /var/lib/cassandra/data/system/Migrations-hd-45 to list of files
> tracked for system.Migrations
> DEBUG 22:50:28,937 adding
> /var/lib/cassandra/data/system/Migrations-hd-46 to list of files
> tracked for system.Migrations
> DEBUG 22:50:28,937 KeyCache capacity for Migrations is 2
> DEBUG 22:50:28,938 Initializing system.LocationInfo
> DEBUG 22:50:28,940 Starting CFS LocationInfo
>  INFO 22:50:28,943 Opening
> /var/lib/cassandra/data/system/LocationInfo-hd-17 (357 bytes)
> DEBUG 22:50:28,944 Load metadata for
> /var/lib/cassandra/data/system/LocationInfo-hd-17
> DEBUG 22:50:28,947 INDEX LOAD TIME for
> /var/lib/cassandra/data/system/LocationInfo-hd-17: 4 ms.
> DEBUG 22:50:28,948 key cache contains 0/0 keys
> DEBUG 22:50:28,948 Creating IntervalNode from
> [Interval(DecoratedKey(61078635599166706937511052402724559481, 4c),
> DecoratedKey(136380791207214638541884810421800860241,
> 426f6f747374726170))]
> DEBUG 22:50:28,949 adding
> /var/lib/cassandra/data/system/LocationInfo-hd-17 to list of files
> tracked for system.LocationInfo
> DEBUG 22:50:28,949 KeyCache capacity for LocationInfo is 1
> DEBUG 22:50:28,949 Initializing system.HintsColumnFamily
> DEBUG 22:50:28,951 Starting CFS HintsColumnFamily
> DEBUG 22:50:28,953 Creating IntervalNode from []
> DEBUG 22:50:28,954 KeyCache capacity for HintsColumnFamily is 1
> DEBUG 22:50:28,959 collectTimeOrderedData
> DEBUG 22:50:28,975 collectTimeOrderedData
>  INFO 22:50:28,977 Loading schema version bba944a0-c74c-11e1-0000-56b66459adff
> DEBUG 22:50:28,978 collectAllData
> DEBUG 22:50:28,983 collecting 0 of 2147483647:
> Avro/Schema:false:2799@1341566817771
> DEBUG 22:50:28,983 collecting 1 of 2147483647: Innovari:false:788@1341566817771
> DEBUG 22:50:29,143 Adding
> org.apache.cassandra.config.CFMetaData@37e55794[cfId=1017,ksName=Innovari,cfName=Temps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@31d8d237,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class
> org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>]
> to cfIdMap
> DEBUG 22:50:29,144 Adding
> org.apache.cassandra.config.CFMetaData@3d46e381[cfId=1004,ksName=Innovari,cfName=TICStatusCurrent,cfType=Standard,comparator=org.apache.cassandra.db.marshal.AsciiType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@79024994,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class
> org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>]
> to cfIdMap
> DEBUG 22:50:29,145 Removing compacted SSTable files from NodeIdInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,147 Removing compacted SSTable files from Versions (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,149 Removing compacted SSTable files from
> Versions.76657273696f6e (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,150 Removing compacted SSTable files from IndexInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,151 Removing compacted SSTable files from Schema (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,153 Removing compacted SSTable files from Migrations
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,155 Removing compacted SSTable files from LocationInfo
> (see http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,157 Removing compacted SSTable files from
> HintsColumnFamily (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,158 Removing compacted SSTable files from Temps (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,160 Removing compacted SSTable files from
> TICStatusCurrent (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
> DEBUG 22:50:29,161 opening keyspace system
> DEBUG 22:50:29,161 opening keyspace Innovari
> DEBUG 22:50:29,162 Configured datacenter replicas are datacenter1:1
> DEBUG 22:50:29,162 Initializing Innovari.Temps
> DEBUG 22:50:29,164 Starting CFS Temps
>  INFO 22:50:29,166 Opening
> /var/lib/cassandra/data/Innovari/Temps-g-117 (156 bytes)
> DEBUG 22:50:29,166 Load metadata for
> /var/lib/cassandra/data/Innovari/Temps-g-117
>  INFO 22:50:29,166 Opening
> /var/lib/cassandra/data/Innovari/Temps-g-118 (156 bytes)
> DEBUG 22:50:29,167 Load metadata for
> /var/lib/cassandra/data/Innovari/Temps-g-118
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to /var/lib/cassandra/java_1341615025.hprof ...
> Heap dump file created [47294429 bytes in 0.418 secs]
> ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main]
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:222)
>         at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:204)
>         at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:194)
>         at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:155)
>         at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Service exit with a return value of 100



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message