Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9ECCFCC3E for ; Sat, 19 May 2012 17:16:10 +0000 (UTC) Received: (qmail 92044 invoked by uid 500); 19 May 2012 17:16:07 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 92019 invoked by uid 500); 19 May 2012 17:16:07 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 92011 invoked by uid 99); 19 May 2012 17:16:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 May 2012 17:16:07 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bk0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 May 2012 17:16:02 +0000 Received: by bkty8 with SMTP id y8so3626141bkt.31 for ; Sat, 19 May 2012 10:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=J4HrWyZ/ZQJKJWMC4PuAP+62HSHOZQzRu15nq89rkCs=; b=snbVODxNKSSpPFlWQrsLUtQUBs+oxdqzyAlqoyO/TkQ5f6yQKll9B/aF9JBcBcCMjD O0fvpiXXUOiimdmqD2rFzJ5+/4CFoGWsTIirmqoznChCrpGtsu/uNwSldB3hMCyVk1vY loyEDwgWE816ywfVtaNYdUOmjfA+PacHx4wN8xagk/fc30eThMyjlIYELKSkYkdJCx+j yR6gRcwliRFJj20UpNFJ/CVEWpX29XcvVoL0DncE/L4qEkNwcv75fTbvS8FFp6d4U8+Y DpxfV8/KMXa5AjuUzWjTTXzrOFJLsGnRp4WaawMFEFmHwSRr/ShRx8wuwgoyG2zMmNMy n/bw== Received: by 10.204.133.200 with SMTP id g8mr5412241bkt.110.1337447741391; Sat, 19 May 2012 10:15:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.33.203 with HTTP; Sat, 19 May 2012 10:15:20 -0700 (PDT) In-Reply-To: References: From: Jonathan Ellis Date: Sat, 19 May 2012 12:15:20 -0500 Message-ID: Subject: Re: while compaction occur EOFException To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Looks like sstable corruption to me. Bad memory can often cause this. You should upgrade to the latest 0.7 release and run nodetool scrub. I don't think the 0.7.3 scrub was very robust. On Thu, May 17, 2012 at 1:36 AM, Preston Cheung wro= te: > While doing compaction, cassandra occured an EOFException, and it seems t= hat > compaction failed. > > I wonder whether my sstables are corrupt or it is a bug? Thanks all help! > > Our cassandra is 0.7.3. > CentOS 5.4 > jdk1.7.0 > > This is the log: > > INFO [CompactionExecutor:1] 2012-05-17 10:42:18,095 CompactionManager.jav= a > (line 452) Compacting > [SSTableReader(path=3D'/data00/data/picasso/value-f-63129-Dat > a.db'),SSTableReader(path=3D'/data01/data/picasso/value-f-63893-Data.db')= ,SSTableReader(path=3D'/data01/data/picasso/value-f-63989-Data.db'),SSTable= Reader(path=3D' > /data00/data/picasso/value-f-63691-Data.db'),SSTableReader(path=3D'/data0= 0/data/picasso/value-f-61779-Data.db'),SSTableReader(path=3D'/data00/data/p= icasso/value- > f-61916-Data.db'),SSTableReader(path=3D'/data00/data/picasso/value-f-6187= 5-Data.db'),SSTableReader(path=3D'/data00/data/picasso/value-f-63296-Data.d= b'),SSTableRe > ader(path=3D'/data00/data/picasso/value-f-62139-Data.db'),SSTableReader(p= ath=3D'/data00/data/picasso/value-f-63821-Data.db')] > ERROR [CompactionExecutor:1] 2012-05-17 10:42:24,306 > AbstractCassandraDaemon.java (line 114) Fatal exception in thread > Thread[CompactionExecutor:1,1,main] > java.io.IOError: java.io.EOFException > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIde= ntityIterator.java:117) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIde= ntityIterator.java:67) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S= STableScanner.java:179) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S= STableScanner.java:144) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:1= 36) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:3= 9) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI= terator.java:284) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.commons.collections.iterators.CollatingIterator.least(Collatin= gIterator.java:326) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.commons.collections.iterators.CollatingIterator.next(Collating= Iterator.java:230) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.= java:68) > =A0=A0=A0=A0=A0=A0=A0 at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:136) > =A0=A0=A0=A0=A0=A0=A0 at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 131) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.commons.collections.iterators.FilterIterator.setNextObject(Fil= terIterator.java:183) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIte= rator.java:94) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.= java:505) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:2= 56) > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > =A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.FutureTask.run(FutureTask.j= ava:166) > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0=A0=A0=A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.EOFException > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65= ) > =A0=A0=A0=A0=A0=A0=A0 at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIde= ntityIterator.java:109) > =A0=A0=A0=A0=A0=A0=A0 ... 20 more > > thx > -- > by Preston Cheung > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com