I am running a move on one node in a 5 node cluster. There are no writes to
the cluster during the move.
I am seeing an exception on one of the nodes (not the node which I am doing
the move on).
The exception stack is
ERROR [CompactionExecutor:1] 2011-02-04 08:10:46,855 PrecompactedRow.java
(line 82) Skipping row DecoratedKey(656517988577125179070965247963445,
555345524e414d452e6a6f746173696c766573747265) in
/var/lib/cassandra/data/Wenzani/UUID_UUID_SUPER-e-408-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:138)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Output from nodetool ring.
Address Status State Load Owns Token
105916716988735575505223832861775432335
1.1.1.2 Up Normal 34.29 GB 45.36%
12956529933298582072612274413196299151
1.1.1.3 Up Normal 34.46 GB 11.41%
32366675628954067180152712803029297247
1.1.1.4 Up Normal 48.96 GB 11.40%
51756081624280481651195537730585467204
1.1.1.5 Up Normal 22 GB 22.78%
90515859237527157456212262236145255573
1.1.1.6 Up Leaving 13.34 GB 9.05%
105916716988735575505223832861775432335
1.1.1.6 is the node which I executed the move on. It seems to be locked in
the Leaving state. Is this normal until the move completes?
There is almost no activity in the logs and very little cpu usage across the
cluster.
Is this expected for a move?
Cheers
Stu
|