cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shotaro Kamio (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2626) stack overflow while compacting
Date Wed, 25 May 2011 13:28:47 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039115#comment-13039115
] 

Shotaro Kamio commented on CASSANDRA-2626:
------------------------------------------

I was able to reproduce the stack overflow problem on cassandra 0.8.0 trunk.
The DataTracker creates unmodifiableSet of 'Set<SSTableReader> compacting' on switching
memtable and other operations. It creates unmodifiableSet one level deeper every time memtables
is switched. When the number of switching memtable reaches some level and its size() is called,
the stack overflow exception occurs if the call stack exceeds stack limit.

Stack trace below (dumped by simple wrapper class MyUnmodifiableSet) shows size() is called
by constructor of HashSet in View.markCompacting() (DataTracker.java).
If the nesting is unavoidable, a solution is to use UnmodifiableSet in apache commons collections
library. It doesn't create nests of unmodifiable collection.


Steps to reproduce the stack overflow:
1. Create single cassandra node with standard column families.
2. Disable compaction and set MemtableThroughputInMB and MemtableOperationsInMillions to small
value in order to flush memtables frequently.
3. Insert many data for many keys.
4. When number of sstables exceeds 2000 (this parameter may vary in environments), run "nodetool
compact". The error will be logged.


-------
* Change for debug in DataTracker.java:
{quote}
        public View(Memtable memtable, Set<Memtable> pendingFlush, Set<SSTableReader>
sstables, Set<SSTableReader> compacting)
        \{
            this.memtable = memtable;
  //             this.memtablesPendingFlush = Collections.unmodifiableSet(pendingFlush);
  //             this.sstables = Collections.unmodifiableSet(sstables);
  //             this.compacting = Collections.unmodifiableSet(compacting);

            this.memtablesPendingFlush = new MyUnmodifiableSet(pendingFlush);
            this.sstables              = new MyUnmodifiableSet(sstables);
            this.compacting            = new MyUnmodifiableSet(compacting);
        \}
{quote}
-------
* Stacktrace by Thread.dumpStack() in MyUnmodifiableSet.

        at java.lang.Thread.dumpStack(Thread.java:1249)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:35)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
   ......
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.HashSet.<init>(HashSet.java:99)
        at org.apache.cassandra.db.DataTracker$View.markCompacting(DataTracker.java:495)
        at org.apache.cassandra.db.DataTracker.markCompacting(DataTracker.java:188)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:312)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

------

> stack overflow while compacting
> -------------------------------
>
>                 Key: CASSANDRA-2626
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2626
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Documentation & website
>    Affects Versions: 0.8 beta 1
>            Reporter: Terje Marthinussen
>
> This is a trunk build from May 3.
> After adding  CASSANDRA-2401, I have gotten the following on several nodes.
> I am not 100% sure right now if it is related to 2401 but it may seem likely.
> Unfortunately, as often is the case with stack overflows, I don't see the start of the
stack
> ERROR [CompactionExecutor:17] 2011-05-09 07:56:32,479 AbstractCassandraDaemon.java (line
112) Fatal exception in thread Thread[CompactionExecutor:17,1,main]
> java.lang.StackOverflowError
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message