Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 98222 invoked from network); 26 Jan 2011 20:40:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 26 Jan 2011 20:40:42 -0000 Received: (qmail 66320 invoked by uid 500); 26 Jan 2011 20:40:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 66278 invoked by uid 500); 26 Jan 2011 20:40:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 66270 invoked by uid 99); 26 Jan 2011 20:40:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 20:40:39 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dan.hendry.junk@gmail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qw0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 20:40:31 +0000 Received: by qwi2 with SMTP id 2so1434977qwi.31 for ; Wed, 26 Jan 2011 12:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:subject:date:message-id:mime-version :content-type:x-mailer:content-language:thread-index; bh=11VaaAPc4tuFesRISOeIwY21A8JZeXnrNN6VTDHRa38=; b=vLZJE7s5Xukbk9uqbAvmUotEpuycUJ5iagBh8znWj9Q3G5B6UnNg8n113zq5lkn+Pv Zidd6VYpD9cwZrTdE65omv/USQW3cjquKL8EKdb9UewoNJupMlrLxBk7e/TPYy0YUG/4 AHf9+iSfT+hf+L88kcUEJKEPQ00RIQtijwNkY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:message-id:mime-version:content-type:x-mailer :content-language:thread-index; b=aa7ktwwM1hR2lc9+0IZ0ouyOEbJdKnZuXg8tncrS47dXWW7ENENrCePxt7AQ1JZVx/ LS1sJsKlvyHU9Fr8TkZXBHbMU1S2r/dL9K+k64oeEwwP7VrMeJmsgrZSXY21kD61Nzec YO/zkjywOLtguFgP5RFFwRCF7PfFF1aiyPDVU= Received: by 10.224.2.213 with SMTP id 21mr780272qak.248.1296074410495; Wed, 26 Jan 2011 12:40:10 -0800 (PST) Received: from DHTABLET (kik.client.acceleratorcentre.net [38.121.79.182]) by mx.google.com with ESMTPS id g28sm11263159qck.25.2011.01.26.12.40.09 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 26 Jan 2011 12:40:09 -0800 (PST) From: "Dan Hendry" To: Subject: My new nemesis: EOFException (0.7.0) Date: Wed, 26 Jan 2011 15:40:04 -0500 Message-ID: <4d4086a9.9c4de50a.0264.ffffb303@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0045_01CBBD6F.4DE16DF0" X-Mailer: Microsoft Office Outlook 12.0 Content-language: en-ca Thread-index: Acu9mTXKqq78dM5dTYKctpJ0t0g1+A== This is a multi-part message in MIME format. ------=_NextPart_000_0045_01CBBD6F.4DE16DF0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I am having yet another issue on one of my Cassandra nodes. Last night, one of my nodes ran out of memory and crashed after flooding the logs with the same type of errors I am seeing below. After restarting, they are popping up again. My solution has been to drop the consistency from ALL to ONE for the query which seems to be causing this problem so my service using Cassandra starts working again but its a terrible solution at best. Is there any thought as to what the root cause of this issue is or thoughts on how to fix it? These errors seem to be popping up when reading from the same column family which I have been having other problems with. Recently, I drained the node and shutdown, deleted all on disk files for this column family, then ran a repair (which caused the node to run out of disk space as I have detailed in a previous email). Could the repair somehow have corrupted the nodes new data? Why is this error only appearing on one node given the data now is nearly guaranteed to have come from another replica? I have triple checked and all nodes are running the same, release version of 0.7 Are there any suggestions for tools to check over a systems hardware (Ubuntu 10.04)? SMART info for the disk shows nothing alarming and there is nothing in /var/log/messages. ERROR [ReadStage:10] 2011-01-26 12:15:59,607 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor java.lang.RuntimeException: java.io.EOFException at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:124) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:47) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableS liceIterator.java:108) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIter ator.java:283) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIt erator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIte rator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav a:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQ ueryFilter.java:118) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilte r.java:142) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1230) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1107) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1077) at org.apache.cassandra.db.Table.getRow(Table.java:384) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.jav a:63) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63 ) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9 4) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 64) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 13) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetche r.getNextBlock(IndexedSliceReader.java:180) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:119) ... 22 more ERROR [ReadStage:10] 2011-01-26 12:15:59,608 AbstractCassandraDaemon.java (line 91) Fatal exception in thread Thread[ReadStage:10,5,main] java.lang.RuntimeException: java.io.EOFException at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:124) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:47) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableS liceIterator.java:108) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIter ator.java:283) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIt erator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIte rator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav a:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQ ueryFilter.java:118) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilte r.java:142) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1230) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1107) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1077) at org.apache.cassandra.db.Table.getRow(Table.java:384) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.jav a:63) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63 ) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9 4) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 64) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 13) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetche r.getNextBlock(IndexedSliceReader.java:180) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:119) ... 22 more ERROR [ReadStage:2] 2011-01-26 12:16:09,732 DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor java.lang.RuntimeException: java.io.EOFException at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:124) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:47) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableS liceIterator.java:108) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIter ator.java:283) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIt erator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIte rator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav a:68) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator .java:136) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131 ) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQ ueryFilter.java:118) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilte r.java:142) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1230) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1107) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1077) at org.apache.cassandra.db.Table.getRow(Table.java:384) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.jav a:63) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63 ) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383) at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9 4) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 64) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:3 13) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetche r.getNextBlock(IndexedSliceReader.java:180) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe dSliceReader.java:119) ... 22 more ------=_NextPart_000_0045_01CBBD6F.4DE16DF0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I am = having yet another issue on one of my Cassandra nodes. Last night, one = of my nodes ran out of memory and crashed after flooding the logs with = the same type of errors I am seeing below. After restarting, they are = popping up again. My solution has been to drop the consistency from ALL = to ONE for the query which seems to be causing this problem so my = service using Cassandra starts working again but its a terrible solution = at best. Is there any thought as to what the root cause of this issue is = or thoughts on how to fix it?

 

These errors = seem to be popping up when reading from the same column family which I = have been having other problems with.  Recently, I drained the node = and shutdown, deleted all on disk files for this column family, then ran = a repair (which caused the node to run out of disk space as I have = detailed in a previous email). Could the repair somehow have corrupted = the nodes new data? Why is this error only appearing on one node given = the data now is nearly guaranteed to have come from another replica? I = have triple checked and all nodes are running the same, release version = of 0.7

 

Are there any suggestions for tools to check over a = systems hardware (Ubuntu 10.04)? SMART info for the disk shows nothing = alarming and there is nothing in /var/log/messages.

 

 

ERROR = [ReadStage:10] 2011-01-26 12:15:59,607 DebuggableThreadPoolExecutor.java = (line 103) Error in ThreadPoolExecutor

java.lang.RuntimeException: = java.io.EOFException

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:124)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:47)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTab= leSliceIterator.java:108)

        at = org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI= terator.java:283)

        at = org.apache.commons.collections.iterators.CollatingIterator.least(Collatin= gIterator.java:326)

        at = org.apache.commons.collections.iterators.CollatingIterator.next(Collating= Iterator.java:230)

        at = org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.= java:68)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(Sli= ceQueryFilter.java:118)

        at = org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFi= lter.java:142)

        at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamily= Store.java:1230)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1107)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1077)

        at = org.apache.cassandra.db.Table.getRow(Table.java:384)

        at = org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.= java:63)

        at = org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)

        at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :63)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908)

     =    at = java.lang.Thread.run(Thread.java:662)

Caused by: java.io.EOFException

        at = java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)<= /p>

        at = org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280= )

        at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.jav= a:94)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:364)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:313)

     =    at = org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFet= cher.getNextBlock(IndexedSliceReader.java:180)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:119)

        ... 22 = more

ERROR [ReadStage:10] 2011-01-26 = 12:15:59,608 AbstractCassandraDaemon.java (line 91) Fatal exception in = thread Thread[ReadStage:10,5,main]

java.lang.RuntimeException: = java.io.EOFException

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:124)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:47)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTab= leSliceIterator.java:108)

        at = org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI= terator.java:283)

        at = org.apache.commons.collections.iterators.CollatingIterator.least(Collatin= gIterator.java:326)

        at = org.apache.commons.collections.iterators.CollatingIterator.next(Collating= Iterator.java:230)

        at = org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.= java:68)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(Sli= ceQueryFilter.java:118)

        at = org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFi= lter.java:142)

        at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamily= Store.java:1230)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1107)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1077)

        at = org.apache.cassandra.db.Table.getRow(Table.java:384)

        at = org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.= java:63)

        at = org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)

        at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :63)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908)

        at = java.lang.Thread.run(Thread.java:662)

Caused by: java.io.EOFException

        at = java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)<= /p>

        at = org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280= )

        at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.jav= a:94)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:364)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:313)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFet= cher.getNextBlock(IndexedSliceReader.java:180)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:119)

        ... 22 = more

ERROR [ReadStage:2] 2011-01-26 = 12:16:09,732 DebuggableThreadPoolExecutor.java (line 103) Error in = ThreadPoolExecutor

java.lang.RuntimeException: = java.io.EOFException

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:124)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:47)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTab= leSliceIterator.java:108)

        at = org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI= terator.java:283)

        at = org.apache.commons.collections.iterators.CollatingIterator.least(Collatin= gIterator.java:326)

        at = org.apache.commons.collections.iterators.CollatingIterator.next(Collating= Iterator.java:230)

        at = org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.= java:68)

        at = com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136)

        at = com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131)

        at = org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(Sli= ceQueryFilter.java:118)

        at = org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFi= lter.java:142)

        at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamily= Store.java:1230)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1107)

        at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilySto= re.java:1077)

        at = org.apache.cassandra.db.Table.getRow(Table.java:384)

        at = org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.= java:63)

        at = org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)

        at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :63)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886)

        at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908)

        at = java.lang.Thread.run(Thread.java:662)

Caused by: java.io.EOFException

        at = java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)<= /p>

        at = org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280= )

        at = org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.jav= a:94)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:364)

        at = org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.jav= a:313)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFet= cher.getNextBlock(IndexedSliceReader.java:180)

        at = org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Ind= exedSliceReader.java:119)

        ... 22 = more

------=_NextPart_000_0045_01CBBD6F.4DE16DF0--