Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7FA30CE2A for ; Mon, 9 Sep 2013 13:15:47 +0000 (UTC) Received: (qmail 22260 invoked by uid 500); 9 Sep 2013 13:15:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22181 invoked by uid 500); 9 Sep 2013 13:15:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22168 invoked by uid 99); 9 Sep 2013 13:15:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Sep 2013 13:15:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dmcnelis@gmail.com designates 209.85.215.42 as permitted sender) Received: from [209.85.215.42] (HELO mail-la0-f42.google.com) (209.85.215.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Sep 2013 13:15:29 +0000 Received: by mail-la0-f42.google.com with SMTP id ep20so4983470lab.29 for ; Mon, 09 Sep 2013 06:15:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=FLhwmrsFVDaKy+pSIkUhpMznhUbwq5CLLFpELW9SFog=; b=cEV8qGKKqZ135T06+ZLYLYPB7DihdXInG5ZLC4J8HcYZnGnilEQUROH2EEjNyBsxAJ TQHPowD1p/8Bx0OmV8ZLAiYX/Ru6pDrTdGxj5YHQd6CRQfFs/Z4AMXxr4kR7t7u9dEuk dWqMsdvrAehFQ1Jlp4vjjLR9VKFdQh5UpGuHahbMHoRQpEIQ2ioXuI7SI/WslL2oRpzJ uI1FKFfXStTVJvE03x456mbix5cvbl77SlMmbbZt+yiR5XXIE+7R4KY1gTNqaOLB01t/ gDdQzfacaZMrKNbExhhpBlfJWzlAaEYAhaaae4c+JxikFtaa8auncL9+AI2OMGaVwawZ in/g== MIME-Version: 1.0 X-Received: by 10.112.28.109 with SMTP id a13mr16470583lbh.3.1378732508606; Mon, 09 Sep 2013 06:15:08 -0700 (PDT) Received: by 10.112.170.39 with HTTP; Mon, 9 Sep 2013 06:15:08 -0700 (PDT) Date: Mon, 9 Sep 2013 09:15:08 -0400 Message-ID: Subject: FSReadError From: David McNelis To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1133f7266a75ac04e5f32d2d X-Virus-Checked: Checked by ClamAV on apache.org --001a1133f7266a75ac04e5f32d2d Content-Type: text/plain; charset=ISO-8859-1 Morning, I'm getting the following error (21 node cluster running 1.2.8) FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:93) at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312) at org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.java:98) at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:73) at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1424) at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1476) at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:46) at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.io.IOException: Input/output error at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) at sun.nio.ch.IOUtil.read(IOUtil.java:198) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:108) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85) ... 11 more This starts during compaction and compaction ends up stuck, the node eventually winds up locked up with this error repeating throughout the logs, and requires a restart. On restart I tried running a nodetool scrub against the offending column family, which itself failed. Any suggestions on taking care of this? Should I just delete the sstable and repair? The node keeps starting compactions and running into this after a period of time. David --001a1133f7266a75ac04e5f32d2d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Morning,

I'm getting the following = error (21 node cluster running 1.2.8)




FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-D=
ata.db

  at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer=
(CompressedRandomAccessReader.java:93)

	at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader=
.java:312)

	at org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.ja=
va:98)

	at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFacto=
ry.java:73)

	at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnF=
amilyStore.java:1424)

	at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilySto=
re.java:1476)

	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(Range=
SliceVerbHandler.java:46)

	at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVer=
bHandler.java:58)

	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.ja=
va:56)

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja=
va:1145)

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j=
ava:615)

	at java.lang.Thread.run(Thread.java:724)
Caused =
by: java.io.IOException: Input/output error


	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)=



	at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherI=
mpl.java:46)


	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java=
:225)


	at sun.nio.ch.IOUtil.read(IOUtil.java:198)

	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)

	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompres=
sChunk(CompressedRandomAccessReader.java:108)

	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(=
CompressedRandomAccessReader.java:85)

	... 11 more

This starts during compact=
ion and compaction ends up stuck, the node eventually winds up locked up wi=
th this error repeating throughout the logs, and requires a restart.


On restart I tried running a nodetool scrub against the offending colum=
n family, which itself failed.

Any suggestions on taking care of thi=
s? Should I just delete the sstable and repair? The node keeps starting com=
pactions and running into this after a period of time.



David
--001a1133f7266a75ac04e5f32d2d--