cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ike Walker <>
Subject Long running nodetool move operation
Date Tue, 10 Sep 2013 02:08:13 GMT
I've been using nodetool move to rebalance my cluster. Most of the moves take under an hour,
or a few hours at most. The current move has taken 4+ days so I'm afraid it will never complete.
What's the best way to cancel it and try again?

I'm running a cluster of 12 nodes at AWS. Each node runs Cassandra 1.2.5 on an m1.xlarge EC2
instance, and they are spread across 3 availability zones within a single region.

I've seen some of these errors in the log. I'm not sure if it's related or not:

ERROR [CompactionExecutor:4092] 2013-09-10 01:31:49,783 (line 175) Exception
in thread Thread[CompactionExecutor:4092,1,main]
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(
        at org.apache.cassandra.db.AbstractColumnContainer.addColumn(
        at org.apache.cassandra.db.ColumnFamily.addAtom(
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(
        at org.apache.cassandra.db.compaction.PrecompactedRow.merge(
        at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(
        at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
        at org.apache.cassandra.db.compaction.CompactionManager$
        at java.util.concurrent.Executors$
        at java.util.concurrent.FutureTask$Sync.innerRun(
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$

Here's the status of the cluster as reported by nodetool ring, showing the one node in "Moving"

Datacenter: us-east
Address         Rack        Status State   Load            Owns                Token     
                                                                               127605887595351923798765477786913079290  1c          Up     Normal  224.53 GB       25.00%              0         
                          1d          Up     Moving  297.46 GB       2.44%               4150051970709140963435425752946440221  1d          Up     Normal  107.75 GB       5.89%               14178431955039102644307275309657008810  1e          Up     Normal  82.75 GB        8.33%               28356863910078205288614550619314017620  1e          Up     Normal  173.83 GB       2.99%               33451586107772559423309548485325625873  1c          Up     Normal  64.4 GB         5.34%               42535295865117307932921825928971026430  1d          Up     Normal  56.46 GB        8.33%               56713727820156410577229101238628035240  1e          Up     Normal  57.88 GB        8.33%               70892159775195513221536376548285044050  1c          Up     Normal  95.18 GB        8.33%               85070591730234615865843651857942052860  1d          Up     Normal  169.37 GB       8.33%               99249023685273718510150927167599061670  1e          Up     Normal  109.99 GB       8.33%               113427455640312821154458202477256070480  1c          Up     Normal  98.64 GB        8.33%               127605887595351923798765477786913079290

View raw message