Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 82A85711B for ; Sat, 27 Aug 2011 16:42:30 +0000 (UTC) Received: (qmail 40660 invoked by uid 500); 27 Aug 2011 16:42:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 40423 invoked by uid 500); 27 Aug 2011 16:42:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 40415 invoked by uid 99); 27 Aug 2011 16:42:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Aug 2011 16:42:27 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-ew0-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Aug 2011 16:42:20 +0000 Received: by ewy19 with SMTP id 19so2043202ewy.31 for ; Sat, 27 Aug 2011 09:42:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=C1ar7nhZrsCwTn/k+Myq3iwOMWS8lX+++PnRmv8eark=; b=rXaw23/TiEDTlyLB5Qx/Q/7y4mXUQWq/JDuD2gblp7V4FZy/VCAVAO42WbE25KiMJ9 YvB3zd+k1G2IYp+wqxnBY+nwwNEd3cnJRYyQhTrmFe4t0+anXaVaLn6biH7wYA5ppYTS iAcH0Y2NxHorQGF18cyLumE3S5qUEkvhBquWo= Received: by 10.213.28.17 with SMTP id k17mr1414802ebc.109.1314463318853; Sat, 27 Aug 2011 09:41:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.108.83 with HTTP; Sat, 27 Aug 2011 09:41:38 -0700 (PDT) In-Reply-To: <023E9AB0-F13A-4E01-9CC6-817D3847CB9A@gmail.com> References: <023E9AB0-F13A-4E01-9CC6-817D3847CB9A@gmail.com> From: Jonathan Ellis Date: Sat, 27 Aug 2011 11:41:38 -0500 Message-ID: Subject: Re: mmap segment underflow To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org My guess would be CASSANDRA-2675. Upgrade to the latest version in your release series, if you haven't. On Sat, Aug 27, 2011 at 9:53 AM, Chad Johnson wr= ote: > Hi, > > We periodically get the following exception in our Cassandra cluster. Wha= t I mean by periodically is once or twice a week. It always affects the sam= e column family ( containing super columns). =A0The frequency of the proble= ms seems to center on the same couple nodes, but not always. I've been gett= ing past the problem by performing a scrub on the column family, which corr= ects the problem by eliminating one unreadable row. > > =A0WARN [CompactionExecutor:1] 2011-08-27 04:08:42,916 CompactionManager.= java (line 654) Unable to recover 1 rows that were skipped. =A0You can atte= mpt manual recovery from the pre-scrub snapshot. =A0You can also run nodeto= ol repair to transfer the data from a healthy replica, if any > > Next time I have the opportunity, I'll throw the node into debug mode bef= ore scrubbing, perhaps more information will be available. > > I have a copy of the SSTables if anybody has an interest in dissecting it= . I'm not a Cassandra developer, but currently I've been able to open the S= STable using the SSTableReader class, but not sure where to go from there. = Maybe I should be using the CompactionManager.scrubOne method or maybe use = the SSTableScanner. Any tips from developers on the cassandra list in this = regard would also be much appreciated. Specifically, I'd like to dissect th= e sstable and to figure out what the key is to the bad row and what is wron= g with the columns/supercolumns in that row. > > The only issue I've found WRT to mmap segment underflow is CASSANDRA-2315= , which doesn't seem to apply in this case. > > Thanks for your help. > > Chad > > ERROR [ReadStage:2772] 2011-08-27 03:42:08,606 AbstractCassandraDaemon.ja= va (line 113) Fatal exception in thread Thread[ReadStage:2772,5,main] > java.io.IOError: java.io.IOException: mmap segment underflow; remaining i= s 2612010 but 692929875 requested > =A0 =A0 =A0 =A0at org.apache.cassandra.io.util.ColumnIterator.deserialize= Next(ColumnSortedMap.java:252) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.util.ColumnIterator.next(Column= SortedMap.java:268) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.util.ColumnIterator.next(Column= SortedMap.java:227) > =A0 =A0 =A0 =A0at java.util.concurrent.ConcurrentSkipListMap.buildFromSor= ted(ConcurrentSkipListMap.java:1521) > =A0 =A0 =A0 =A0at java.util.concurrent.ConcurrentSkipListMap.(Concu= rrentSkipListMap.java:1471) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.SuperColumnSerializer.deseriali= ze(SuperColumn.java:379) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.SuperColumnSerializer.deseriali= ze(SuperColumn.java:362) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.SuperColumnSerializer.deseriali= ze(SuperColumn.java:322) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.columniterator.SimpleSliceReade= r.computeNext(SimpleSliceReader.java:79) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.columniterator.SimpleSliceReade= r.computeNext(SimpleSliceReader.java:40) > =A0 =A0 =A0 =A0at com.google.common.collect.AbstractIterator.tryToCompute= Next(AbstractIterator.java:136) > =A0 =A0 =A0 =A0at com.google.common.collect.AbstractIterator.hasNext(Abst= ractIterator.java:131) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.columniterator.SSTableSliceIter= ator.hasNext(SSTableSliceIterator.java:108) > =A0 =A0 =A0 =A0at org.apache.commons.collections.iterators.CollatingItera= tor.set(CollatingIterator.java:283) > =A0 =A0 =A0 =A0at org.apache.commons.collections.iterators.CollatingItera= tor.least(CollatingIterator.java:326) > =A0 =A0 =A0 =A0at org.apache.commons.collections.iterators.CollatingItera= tor.next(CollatingIterator.java:230) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ReducingIterator.computeNext= (ReducingIterator.java:69) > =A0 =A0 =A0 =A0at com.google.common.collect.AbstractIterator.tryToCompute= Next(AbstractIterator.java:136) > =A0 =A0 =A0 =A0at com.google.common.collect.AbstractIterator.hasNext(Abst= ractIterator.java:131) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.filter.SliceQueryFilter.collect= ReducedColumns(SliceQueryFilter.java:116) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.filter.QueryFilter.collectColla= tedColumns(QueryFilter.java:130) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelCo= lumns(ColumnFamilyStore.java:1390) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.getColumnFami= ly(ColumnFamilyStore.java:1267) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.getColumnFami= ly(ColumnFamilyStore.java:1195) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.getRow(Table.java:324) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.SliceFromReadCommand.getRow(Sli= ceFromReadCommand.java:63) > =A0 =A0 =A0 =A0at org.apache.cassandra.service.StorageProxy$LocalReadRunn= able.runMayThrow(StorageProxy.java:451) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.runWorker(Threa= dPoolExecutor.java:1110) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:603) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.IOException: mmap segment underflow; remaining is 2612= 010 but 692929875 requested > =A0 =A0 =A0 =A0at org.apache.cassandra.io.util.MappedFileDataInput.readBy= tes(MappedFileDataInput.java:118) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBuff= erUtil.java:356) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.ByteBufferUtil.readWithLengt= h(ByteBufferUtil.java:318) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnSerializer.deserialize(Co= lumnSerializer.java:81) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.util.ColumnIterator.deserialize= Next(ColumnSortedMap.java:248) > =A0 =A0 =A0 =A0... 30 more > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com