incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: recurring EOFException exception in 0.7.4
Date Fri, 15 Apr 2011 19:55:18 GMT
Try running nodetool scrub on the cf: its pretty good at detecting and
fixing most corruption problems.

Dan


-----Original Message-----
From: Jonathan Colby [mailto:jonathan.colby@gmail.com] 
Sent: April-15-11 15:41
To: user@cassandra.apache.org
Subject: recurring EOFException exception in 0.7.4

I've been struggling with these kinds of exceptions for some time now.  I
thought it might have been a one-time thing, so on the 2 nodes where I saw
this problem I pulled in fresh data with a repair on an empty data
directory.

Unfortunately, this problem is now coming up on a new node that has, up
until now, not had this problem.

What could be causing this?  Could it be related to encoding?   Why are
these rows not readable?   

This exception prevents cassandra from doing repairs, and even minor
compactions.  It also messes up memtable management (with a normal load of
25GB,  disk goes to almost 100% full on a 500 GB hd).

This is incredibly frustrating.  This is the only pain-point I have had with
cassandra so far.   By the way, this node was never upgraded - it was 0.7.4
from the start, so that eliminates format compatibility problems.

ERROR [CompactionExecutor:1] 2011-04-15 21:31:23,479 PrecompactedRow.java
(line 82) Skipping row DecoratedKey(105452551814086725777389040553659117532,
4d657373616765456e726963686d656e743a313032343937) in
/var/lib/cassandra/data/DFS/main-f-91-Data.db
java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
	at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRand
omAccessFile.java:270)
	at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
	at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java
:272)
	at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9
4)
	at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:3
5)
	at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFami
lySerializer.java:129)
	at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithC
olumns(SSTableIdentityIterator.java:176)
	at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
	at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterato
r.java:147)
	at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:108)
	at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:43)
	at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav
a:73)
	at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)
	at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)
	at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(Filter
Iterator.java:183)
	at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterat
or.java:94)
	at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.jav
a:449)
	at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
	at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
	at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
	at java.lang.Thread.run(Thread.java:662)

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.894 / Virus Database: 271.1.1/3574 - Release Date: 04/15/11
02:34:00


Mime
View raw message