incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Colby <jonathan.co...@gmail.com>
Subject Is the repair still going on or did it fail because of exceptions?
Date Fri, 08 Apr 2011 14:19:04 GMT
It seems on my cluster there are a few unserializable Rows.  I'm trying to run a repair on
the nodes, but it also seems that the replica nodes have unreadable or unserializable rows.
   The problem is, I cannot determine if the repair is still going on, or if was interrupted
because of these errors.   It is unclear because nothing else related to the repair show up
in the logs.  It's been about 5 hours and I also don't see anything happening when I perform
a "nodetool netstats" on the nodes.  The nodetool repair command is still blocking from the
console.

On the node I'm trying to repair, I see this after launching a "repair":

...
 INFO [manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc] 2011-04-08 11:41:55,520 AntiEntropyService.java
(line 770) Waiting for repair requests: [#<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-7
9caafd7d3cc, /10.46.108.102, (DFS,main)>, #<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc,
/10.46.108.101, (DFS,main)>, #<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc
, /10.46.108.100, (DFS,main)>, #<TreeRequest manual-repair-6160b400-2c82-4ccb-9451-79caafd7d3cc,
/10.47.108.101, (DFS,main)>]
...

In the log of the node "10.46.108.102" where the repair tries to compare the replica data,
  I see a couple of the below exceptions a few minutes later.    Are the exceptions bad enough
to cause the repair to fail?


ERROR [CompactionExecutor:1] 2011-04-08 11:43:01,177 PrecompactedRow.java (line 82) Skipping
row DecoratedKey(1782314446006375058060694305099335169, 4d657373616765456e726963686d656e743a31343236)
in /va
r/lib/cassandra/data/DFS/main-f-177-Data.db
java.io.EOFException
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
        at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
        at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
       at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
        at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
        at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
        at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
        at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
        at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
        at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
        at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
        at org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
        at org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
        at org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
ERROR [CompactionExecutor:1] 2011-04-08 11:43:53,762 PrecompactedRow.java (line 82) Skipping
row DecoratedKey(8073554114801607394928746621229606383, 34393734663734382d316330302d346164372d613333372d3162
34303866613933333832) in /var/lib/cassandra/data/DFS/main-f-177-Data.db
java.io.EOFException
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
        at org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
:

nodetool netstats reports:

Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name                    Active   Pending      Completed
Commands                        n/a         0         526207
Responses                       n/a         0        1747991 
Mime
View raw message