Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.colby@gmail.com
 designates 209.85.214.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=aE7+QsLjC/psobIX88nkDSsgRrxzmORGvwRs+Cm7z3waqcsDSBv0R1dnOhk6f2P+pe
         MSqBJI3w5E7rCJ+pYTe9AF0cNlEsNlH23+4ZMRlZ8Lckp0xcWVeb8xo3rDDYQ2T7/y3J
         a+DCPYMgKWWFC19m5R1Os5ZD9Al+LZplRRGYk=
From: Jonathan Colby <jonathan.colby@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: recurring EOFException exception in 0.7.4
Date: Fri, 15 Apr 2011 21:40:54 +0200
Message-Id: <83AC64F5-8E30-47BD-9D65-C67E44BA75B2@gmail.com>
To: user@cassandra.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)

I've been struggling with these kinds of exceptions for some time now.  =
I thought it might have been a one-time thing, so on the 2 nodes where I =
saw this problem I pulled in fresh data with a repair on an empty data =
directory.

Unfortunately, this problem is now coming up on a new node that has, up =
until now, not had this problem.

What could be causing this?  Could it be related to encoding?   Why are =
these rows not readable?  =20

This exception prevents cassandra from doing repairs, and even minor =
compactions.  It also messes up memtable management (with a normal load =
of 25GB,  disk goes to almost 100% full on a 500 GB hd).

This is incredibly frustrating.  This is the only pain-point I have had =
with cassandra so far.   By the way, this node was never upgraded - it =
was 0.7.4 from the start, so that eliminates format compatibility =
problems.

ERROR [CompactionExecutor:1] 2011-04-15 21:31:23,479 =
PrecompactedRow.java (line 82) Skipping row =
DecoratedKey(105452551814086725777389040553659117532, =
4d657373616765456e726963686d656e743a313032343937) in =
/var/lib/cassandra/data/DFS/main-f-91-Data.db
java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
	at =
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRa=
ndomAccessFile.java:270)
	at =
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
	at =
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja=
va:272)
	at =
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java=
:94)
	at =
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java=
:35)
	at =
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa=
milySerializer.java:129)
	at =
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWit=
hColumns(SSTableIdentityIterator.java:176)
	at =
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
	at =
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionItera=
tor.java:147)
	at =
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.j=
ava:108)
	at =
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.j=
ava:43)
	at =
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.j=
ava:73)
	at =
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat=
or.java:136)
	at =
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1=
31)
	at =
org.apache.commons.collections.iterators.FilterIterator.setNextObject(Filt=
erIterator.java:183)
	at =
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIter=
ator.java:94)
	at =
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.j=
ava:449)
	at =
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:12=
4)
	at =
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94=
)
	at =
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.=
java:886)
	at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:908)
	at java.lang.Thread.run(Thread.java:662)