incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ike Walker <ike.wal...@flite.com>
Subject Long running nodetool move operation
Date Tue, 10 Sep 2013 02:08:13 GMT
I've been using nodetool move to rebalance my cluster. Most of the moves take under an hour,
or a few hours at most. The current move has taken 4+ days so I'm afraid it will never complete.
What's the best way to cancel it and try again?

I'm running a cluster of 12 nodes at AWS. Each node runs Cassandra 1.2.5 on an m1.xlarge EC2
instance, and they are spread across 3 availability zones within a single region.

I've seen some of these errors in the log. I'm not sure if it's related or not:

ERROR [CompactionExecutor:4092] 2013-09-10 01:31:49,783 CassandraDaemon.java (line 175) Exception
in thread Thread[CompactionExecutor:4092,1,main]
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
        at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:128)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
        at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:219)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
        at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
        at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
        at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:134)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

Here's the status of the cluster as reported by nodetool ring, showing the one node in "Moving"
state:

Datacenter: us-east
==========
Address         Rack        Status State   Load            Owns                Token     
                                 
                                                                               127605887595351923798765477786913079290
    
10.xxx.xxx.xxx  1c          Up     Normal  224.53 GB       25.00%              0         
                                 
10.xxx.xxx.xxx  1d          Up     Moving  297.46 GB       2.44%               4150051970709140963435425752946440221
      
10.xxx.xxx.xxx  1d          Up     Normal  107.75 GB       5.89%               14178431955039102644307275309657008810
     
10.xxx.xxx.xxx  1e          Up     Normal  82.75 GB        8.33%               28356863910078205288614550619314017620
     
10.xxx.xxx.xxx  1e          Up     Normal  173.83 GB       2.99%               33451586107772559423309548485325625873
     
10.xxx.xxx.xxx  1c          Up     Normal  64.4 GB         5.34%               42535295865117307932921825928971026430
     
10.xxx.xxx.xxx  1d          Up     Normal  56.46 GB        8.33%               56713727820156410577229101238628035240
     
10.xxx.xxx.xxx  1e          Up     Normal  57.88 GB        8.33%               70892159775195513221536376548285044050
     
10.xxx.xxx.xxx  1c          Up     Normal  95.18 GB        8.33%               85070591730234615865843651857942052860
     
10.xxx.xxx.xxx  1d          Up     Normal  169.37 GB       8.33%               99249023685273718510150927167599061670
     
10.xxx.xxx.xxx  1e          Up     Normal  109.99 GB       8.33%               113427455640312821154458202477256070480
    
10.xxx.xxx.xxx  1c          Up     Normal  98.64 GB        8.33%               127605887595351923798765477786913079290
    


Mime
View raw message