incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Hiramoto <k...@hiramoto.org>
Subject nodetool repair hung in 0.7.3
Date Tue, 08 Mar 2011 15:30:09 GMT
I never saw this before upgrading to 0.7.3 but now I do nodetool repair 
and it sits there for hours.  Previously it took about 20 minutes per 
node (about 10GB of data per node).

I had some OOM crashes, but haven't seen them since I increased the heap 
size and decreased the key cache.


In the nodes logs I see:


  INFO [manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a] 2011-03-08 
15:15:41,856 AntiEntropyService.java (line 752) No neighbors to repair 
with: manual-repair-dbf777bf-12ff-4172-8d7f-a4ed6d9ccc6a completed.
  INFO [manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd] 2011-03-08 
15:15:41,862 AntiEntropyService.java (line 770) Waiting for repair 
requests: [#<TreeRequest 
manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /10.224.4.10, 
(OfferRequest,Offer)>, #<TreeRequest 
manual-repair-0d4e4e0d-0641-4e25-8e20-baabbce344dd, /10.224.4.15, 
(OfferRequest,Offer)>]
  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 
ColumnFamilyStore.java (line 695) switching in a fresh Memtable for 
Offer at 
CommitLogContext(file='/home/cassandra/commitlog/CommitLog-1299596902726.log', 
position=110125920)
  INFO [CompactionExecutor:1] 2011-03-08 15:15:41,862 
ColumnFamilyStore.java (line 1006) Enqueuing flush of 
Memtable-Offer@185502190(41184536 bytes, 109934 operations)
  INFO [FlushWriter:1] 2011-03-08 15:15:41,862 Memtable.java (line 157) 
Writing Memtable-Offer@185502190(41184536 bytes, 109934 operations)
  INFO [FlushWriter:1] 2011-03-08 15:15:42,334 Memtable.java (line 164) 
Completed flushing 
/home/cassandra/data/OfferRequest/Offer-f-5616-Data.db (38695861 bytes)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,835 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070595676245833572375607193993401386, 
383236424d374449494a47) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
java.io.EOFException
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
         at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
         at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
         at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,837 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070596351011236998883565851954247305, 
4332383230413943313234) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,841 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070619433259740737929970212513100751, 
314d414244334c374d4c4e) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:43,843 
AbstractCassandraDaemon.java (line 114) Fatal exception in thread 
Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
         at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
         at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
         at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doValidationCompaction(CompactionManager.java:803)
         at 
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.java:56)
         at 
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:358)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
         ... 21 more
  INFO [CompactionExecutor:1] 2011-03-08 15:15:43,844 
CompactionManager.java (line 396) Compacting 
[SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5565-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5569-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db'),SSTableReader(path='/home/cassandra/data/OfferRequest/Offer-f-5590-Data.db')]
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,179 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070595676245833572375607193993401386, 
383236424d374449494a47) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
java.io.EOFException
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
         at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:268)
         at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
         at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:267)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,180 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070596351011236998883565851954247305, 
4332383230413943313234) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,184 
PrecompactedRow.java (line 82) Skipping row 
DecoratedKey(85070619433259740737929970212513100751, 
314d414244334c374d4c4e) in 
/home/cassandra/data/OfferRequest/Offer-f-5571-Data.db
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid 
column name length 0
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
         at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:176)
         at 
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
         at 
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
         at 
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
ERROR [CompactionExecutor:1] 2011-03-08 15:15:45,186 
AbstractCassandraDaemon.java (line 114) Fatal exception in thread 
Thread[CompactionExecutor:1,1,main]
java.io.IOError: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
         at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
         at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39)
         at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
         at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
         at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
         at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
         at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
         at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
         at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
         at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
         at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:449)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
         at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
         at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
         at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.EOFException
         at 
org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
         at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:109)
         ... 21 more

Mime
View raw message