Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of dan.hendry.junk@gmail.com
 designates 209.85.213.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:to:references:in-reply-to:subject:date:message-id:mime-version
         :content-type:content-transfer-encoding:x-mailer:content-language
         :thread-index;
        b=dM3kK/ahu4dK6oaAD1RKBAk60WgWeXFyNLDeLLSjefuy1ulEaGcvoXQToTTCjHWVmO
         CNCFW/zRpzWCfrnypvOvp86YttIjVtrbjdB8yMzLY5miiRZ5jpiV5jZuA+Sb2f1bvy7G
         VBnn90dWYfSvhmWhQOOBJxPrmqnQJjUkadqEs=
From: "Dan Hendry" <dan.hendry.junk@gmail.com>
To: <user@cassandra.apache.org>
References: <83AC64F5-8E30-47BD-9D65-C67E44BA75B2@gmail.com>
In-Reply-To: <83AC64F5-8E30-47BD-9D65-C67E44BA75B2@gmail.com>
Subject: RE: recurring EOFException exception in 0.7.4
Date: Fri, 15 Apr 2011 15:55:18 -0400
Message-ID: <4da8a2b9.94a5e70a.679b.77ae@mx.google.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-language: en-ca
Thread-index: Acv7pR3mnmoaP0YMS8eWsj/lFnSr9gAAUeAQ

Try running nodetool scrub on the cf: its pretty good at detecting and
fixing most corruption problems.

Dan


-----Original Message-----
From: Jonathan Colby [mailto:jonathan.colby@gmail.com] 
Sent: April-15-11 15:41
To: user@cassandra.apache.org
Subject: recurring EOFException exception in 0.7.4

I've been struggling with these kinds of exceptions for some time now.  I
thought it might have been a one-time thing, so on the 2 nodes where I saw
this problem I pulled in fresh data with a repair on an empty data
directory.

Unfortunately, this problem is now coming up on a new node that has, up
until now, not had this problem.

What could be causing this?  Could it be related to encoding?   Why are
these rows not readable?   

This exception prevents cassandra from doing repairs, and even minor
compactions.  It also messes up memtable management (with a normal load of
25GB,  disk goes to almost 100% full on a 500 GB hd).

This is incredibly frustrating.  This is the only pain-point I have had with
cassandra so far.   By the way, this node was never upgraded - it was 0.7.4
from the start, so that eliminates format compatibility problems.

ERROR [CompactionExecutor:1] 2011-04-15 21:31:23,479 PrecompactedRow.java
(line 82) Skipping row DecoratedKey(105452551814086725777389040553659117532,
4d657373616765456e726963686d656e743a313032343937) in
/var/lib/cassandra/data/DFS/main-f-91-Data.db
java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
	at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRand
omAccessFile.java:270)
	at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
	at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java
:272)
	at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9
4)
	at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:3
5)
	at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFami
lySerializer.java:129)
	at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithC
olumns(SSTableIdentityIterator.java:176)
	at
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
	at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterato
r.java:147)
	at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:108)
	at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:43)
	at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav
a:73)
	at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)
	at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)
	at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(Filter
Iterator.java:183)
	at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterat
or.java:94)
	at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.jav
a:449)
	at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
	at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
	at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
	at java.lang.Thread.run(Thread.java:662)

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.894 / Virus Database: 271.1.1/3574 - Release Date: 04/15/11
02:34:00