cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Javier Sotelo <javier.a.sot...@gmail.com>
Subject Re: Cassandra 1.1.1 Fails to Start
Date Fri, 08 Jun 2012 16:07:36 GMT
Looks like someone beat me to it,
https://issues.apache.org/jira/browse/CASSANDRA-4321

On Fri, Jun 8, 2012 at 9:06 AM, Javier Sotelo <javier.a.sotelo@gmail.com>wrote:

> Different node same hardware now gets the stack overflow error but I found
> the part of the stack trace that is more interesting:
>
>
>         at
> com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
>         at
> com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
>         at
> com.google.common.collect.Iterators$5.hasNext(Iterators.java:517)
>         at
> com.google.common.collect.Iterators$3.hasNext(Iterators.java:114)
>         at
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
>         at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>         at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>         at com.google.common.collect.Iterators.size(Iterators.java:129)
>         at com.google.common.collect.Sets$3.size(Sets.java:670)
>         at com.google.common.collect.Iterables.size(Iterables.java:80)
>         at
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557)
>         at
> org.apache.cassandra.db.compaction.CompactionController.<init>(CompactionController.java:79)
>         at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105)
>         at
> org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>
> Is it time for a JIRA ticket?
>
>
> On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo <javier.a.sotelo@gmail.com>wrote:
>
>> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small
>> keyspace with no compression, about 250MB. The rest taken by the second
>> keyspace with leveled compaction and snappy compressed.
>>
>> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM.
>>
>>
>> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>>> How much data do you have on the node ?
>>> Was this a previously running system that was upgraded ?
>>>
>>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>>> error on SSTableBatchOpen thread
>>> Do you have the stack trace from the log ?
>>>
>>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>>> AbstractCassandraDaemon.java (line 134) Exception in thread
>>> Thread[CompactionExecutor:6,1,main]
>>> > java.lang.StackOverflowError
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> Was there more to this stack trace ?
>>> What were the log messages before this error ?
>>>
>>>
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 122) Heap size: 1525415936/1525415936
>>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have
>>> some data to load I would not be surprised if it failed to start.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote:
>>>
>>> > Hi All,
>>> >
>>> > On SuSe Linux blade with 6GB of RAM.
>>> >
>>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed
>>> error on SSTableBatchOpen thread. cat /proc/<pid>/maps shows a peak of
>>> 53521 right before it dies. vm.max_map_count = 1966080 and
>>> /proc/<pid>/limits shows unlimited locked memory.
>>> >
>>> > with disk_access_mode standard, the node does start up but I see the
>>> repeated error:
>>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772
>>> AbstractCassandraDaemon.java (line 134) Exception in thread
>>> Thread[CompactionExecutor:6,1,main]
>>> > java.lang.StackOverflowError
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> >         at com.google.common.collect.Sets$1.iterator(Sets.java:578)
>>> > ...
>>> >
>>> > I'm not sure the second error is related to the first. I prefer to run
>>> with full mmap but I have run out of ideas. Is there anything else I can do
>>> to debug this?
>>> >
>>> > Here's startup settings from debug log:
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31
>>> >  INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java
>>> (line 122) Heap size: 1525415936/1525415936
>>> >  ...
>>> >  INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA
>>> mlockall successful
>>> >  ...
>>> >  INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line
>>> 191) DiskAccessMode is standard, indexAccessMode is standard
>>> >  INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line
>>> 247) Global memtable threshold is enabled at 484MB
>>> >  INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96)
>>> Initializing key cache with capacity of 72 MBs.
>>> >  INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107)
>>> Scheduling key cache save to each 14400 seconds (going to save all keys).
>>> >  INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121)
>>> Initializing row cache with capacity of 0 MBs and provider
>>> org.apache.cassandra.cache.SerializingCacheProvider
>>> >  INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133)
>>> Scheduling row cache save to each 0 seconds (going to save all keys).
>>> >
>>> > Thanks In Advance,
>>> > Javier
>>>
>>>
>>
>

Mime
View raw message