Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.colby@gmail.com
 designates 209.85.161.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=dMewTaHV5ckoqcPDASe1IXuzss+bcS9MEQpZg4460NBpJbXn6qwZxgfhklHbFeTvSy
         FLLKCEXWjSyXWMJAeyxeXyXo6dL3bNbtSD6MBMzKwZKOPMzM1MDXDwjCL2ZXKJe/aRMY
         IGLXX3QHpezQlluhu06pJ4p0nM2q27GrIxBKw=
From: Jonathan Colby <jonathan.colby@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: unrepairable sstable data rows
Date: Sun, 10 Apr 2011 14:15:51 +0200
Message-Id: <0D174140-3132-4DCA-9BB3-A6BEB2FB9BCE@gmail.com>
To: user@cassandra.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)

It appears we have several unserializable or unreadable rows.  These =
were not fixed even after doing a "scrub"  on all nodes -  even though =
the scrub seemed to have completed successfully.

I trying to fix these by doing a "repair", but these exceptions are =
thrown exactly when doing a repair.   Anyone run into this issue?  =
What's the best way to fix this? =20

I was thinking that flushing and reloading the data with a move (reusing =
the same token) might be a way to get out of this.


Exception seem multiple times for different keys during a repair:

ERROR [CompactionExecutor:1] 2011-04-10 14:05:55,528 =
PrecompactedRow.java (line 82) Skipping row =
DecoratedKey(58054163627659284217684165071269705317, =
64396663313763662d383432622d343439652d623761312d643164663936333738306565) =
in /var/lib/cassandra/data/DFS/main-f-232-Data.db
java.io.EOFException
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
	at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
	at =
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRa=
ndomAccessFile.java:268)
	at =
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:310)
	at =
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja=
va:267)
	at =
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java=
:94)
	at =
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java=
:35)
	at =
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa=
milySerializer.java:129)
	at =
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWit=
hColumns(SSTableIdentityIterator.java:176)
	at =
org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:78)
	at =
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionItera=
tor.java:139)
	at =
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.j=
ava:108)
	at =
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.j=
ava:43)
	at =
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.j=
ava:73)
	at =
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterat=
or.java:136)
	at =
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:1=
31)
	at =
org.apache.commons.collections.iterators.FilterIterator.setNextObject(Filt=
erIterator.java:183)
	at =
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIter=
ator.java:94)
	at =
org.apache.cassandra.db.CompactionManager.doValidationCompaction(Compactio=
nManager.java:803)
	at =
org.apache.cassandra.db.CompactionManager.access$800(CompactionManager.jav=
a:56)
	at =
org.apache.cassandra.db.CompactionManager$6.call(CompactionManager.java:35=
8)
	at =
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.=
java:886)
	at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:908)
	at java.lang.Thread.run(Thread.java:662)


This WARN also seems to come up often during a repair.  Not sure if it =
related to this problem:

 WARN [ScheduledTasks:1] 2011-04-10 14:10:24,991 GCInspector.java (line =
149) Heap is 0.8675910480028087 full.  You may need to reduce memtable =
and/or cache sizes.  Cassandra will now flush up to the two largest =
memtables to free up memory.  Adjust flush_largest_memtables_at =
threshold in cassandra.yaml if you don't want Cassandra to do this =
automatically
 WARN [ScheduledTasks:1] 2011-04-10 14:10:24,992 StorageService.java =
(line 2206) Flushing ColumnFamilyStore(table=3D'DFS', =
columnFamily=3D'main') to relieve memory pressure
 INFO [ScheduledTasks:1] 2011-04-10 14:10:24,992 ColumnFamilyStore.java =
(line 695) switching in a fresh Memtable for main at =
CommitLogContext(file=3D'/var/lib/cassandra/commitlog/CommitLog-1302435708=
131.log', position=3D28257053)