Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 25 May 2011 13:28:47 +0000 (UTC)
From: "Shotaro Kamio (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1417716302.42240.1306330127445.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <2045847489.31717.1304917563238.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (CASSANDRA-2626) stack overflow while compacting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039115#comment-13039115 ] 

Shotaro Kamio commented on CASSANDRA-2626:
------------------------------------------

I was able to reproduce the stack overflow problem on cassandra 0.8.0 trunk.
The DataTracker creates unmodifiableSet of 'Set<SSTableReader> compacting' on switching memtable and other operations. It creates unmodifiableSet one level deeper every time memtables is switched. When the number of switching memtable reaches some level and its size() is called, the stack overflow exception occurs if the call stack exceeds stack limit.

Stack trace below (dumped by simple wrapper class MyUnmodifiableSet) shows size() is called by constructor of HashSet in View.markCompacting() (DataTracker.java).
If the nesting is unavoidable, a solution is to use UnmodifiableSet in apache commons collections library. It doesn't create nests of unmodifiable collection.


Steps to reproduce the stack overflow:
1. Create single cassandra node with standard column families.
2. Disable compaction and set MemtableThroughputInMB and MemtableOperationsInMillions to small value in order to flush memtables frequently.
3. Insert many data for many keys.
4. When number of sstables exceeds 2000 (this parameter may vary in environments), run "nodetool compact". The error will be logged.


-------
* Change for debug in DataTracker.java:
{quote}
        public View(Memtable memtable, Set<Memtable> pendingFlush, Set<SSTableReader> sstables, Set<SSTableReader> compacting)
        \{
            this.memtable = memtable;
  //             this.memtablesPendingFlush = Collections.unmodifiableSet(pendingFlush);
  //             this.sstables = Collections.unmodifiableSet(sstables);
  //             this.compacting = Collections.unmodifiableSet(compacting);

            this.memtablesPendingFlush = new MyUnmodifiableSet(pendingFlush);
            this.sstables              = new MyUnmodifiableSet(sstables);
            this.compacting            = new MyUnmodifiableSet(compacting);
        \}
{quote}
-------
* Stacktrace by Thread.dumpStack() in MyUnmodifiableSet.

        at java.lang.Thread.dumpStack(Thread.java:1249)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:35)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
   ......
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
        at org.apache.cassandra.db.MyUnmodifiableSet.size(MyUnmodifiableSet.java:36)
        at java.util.HashSet.<init>(HashSet.java:99)
        at org.apache.cassandra.db.DataTracker$View.markCompacting(DataTracker.java:495)
        at org.apache.cassandra.db.DataTracker.markCompacting(DataTracker.java:188)
        at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:312)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

------

> stack overflow while compacting
> -------------------------------
>
>                 Key: CASSANDRA-2626
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2626
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Documentation & website
>    Affects Versions: 0.8 beta 1
>            Reporter: Terje Marthinussen
>
> This is a trunk build from May 3.
> After adding  CASSANDRA-2401, I have gotten the following on several nodes.
> I am not 100% sure right now if it is related to 2401 but it may seem likely.
> Unfortunately, as often is the case with stack overflows, I don't see the start of the stack
> ERROR [CompactionExecutor:17] 2011-05-09 07:56:32,479 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:17,1,main]
> java.lang.StackOverflowError
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)
>         at java.util.Collections$UnmodifiableCollection.size(Collections.java:998)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira