From user-return-21252-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Sun Oct 2 23:20:53 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3DA3C78F7 for ; Sun, 2 Oct 2011 23:20:53 +0000 (UTC) Received: (qmail 27768 invoked by uid 500); 2 Oct 2011 23:20:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 27739 invoked by uid 500); 2 Oct 2011 23:20:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 27729 invoked by uid 99); 2 Oct 2011 23:20:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Oct 2011 23:20:51 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a78.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Oct 2011 23:20:44 +0000 Received: from homiemail-a78.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a78.g.dreamhost.com (Postfix) with ESMTP id 5588915C059 for ; Sun, 2 Oct 2011 16:20:21 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=NZPH4QeLgP DftMz+/ixex+cAoYMM/uIZFx4LtR26xSHl8qFQORiWPEyTRF/JHm8SU/JxwLTTk/ 0M2r8duUxU4uQ/x7h4It1ZitAUQB6Espr1otpGG+RiSotidnzPGlZWmVtFWdcj/N vHo6WmJm7hHl8K1gSA+42YkkZe0sSXpts= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=W6pGNOM8/26QVMWl IeqePL7Ycq8=; b=kjdE1h5gP87teQupmJ1KBHGWngMY0j7UuoOJ+pOI5P6VWB2h QVqek+vreh06Pu0c4f2Ku1rQNG/sTwYwlrc+C3epkyK07T3nYNyxNiuZ5JtGvfor /2h+oud4fdP1UJP8geGnnWIpJVeRSH+BILkPsRvs+RsoJWFWu83GXtSI5jw= Received: from [172.16.1.4] (222-152-100-77.jetstream.xtra.co.nz [222.152.100.77]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a78.g.dreamhost.com (Postfix) with ESMTPSA id 8A21F15C058 for ; Sun, 2 Oct 2011 16:20:20 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_80A423D1-7109-4AA3-9EE4-C86AC2C5CA5B" Subject: Re: mmap segment underflow Date: Mon, 3 Oct 2011 12:20:16 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1244.3) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_80A423D1-7109-4AA3-9EE4-C86AC2C5CA5B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 tmp files are temporary in nature, if they fail to live up to their name = then they are against nature.=20 If they are still been written to then something is happening, either a = flush or a compaction. Check nodetool compactstats They may indicate errors during the scrub (or compact), check the logs = for ERRORs. Note that scrub may log a lot non crashing errors that it = could read a row, so read the error message carefully.=20 When the server restarts it will clear tmp files. =20 =20 So if you unnerved at the prospect of unnatural temp files, exorcise = them casting a restart spell.=20 Hope that helps.=20 ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 2/10/2011, at 11:41 AM, Eric Czech wrote: > Scrub seems to have worked. Thanks again! >=20 > Will a major compaction delete the "tmp" sstables genereated though? = Scrub seems to have generated a lot of them and they're taking up an = unnerving amount of disk space. >=20 > On Mon, Sep 19, 2011 at 5:34 PM, Eric Czech = wrote: > Ok then I'll shutdown the server, change the access mode, restart, and > run scrub (and then change the access mode back). >=20 > Thanks for the pointers and I'll let you know how it goes one way or = the other. >=20 > On Tue, Sep 20, 2011 at 12:29 AM, aaron morton = wrote: > > I've also found it useful to disable memmapped file access until the = scrub is complete by adding this to the yaml > > > > disk_access_mode: standard > > > > Cheers > > > > ----------------- > > Aaron Morton > > Freelance Cassandra Developer > > @aaronmorton > > http://www.thelastpickle.com > > > > On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote: > > > >> You should start with scrub. > >> > >> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech = wrote: > >>> I'm getting a lot of errors that look something like = "java.io.IOError: > >>> java.io.IOException: mmap segment underflow; remaining is = 348268797 > >>> but 892417075 requested" on one node in a 10 node cluster. I'm > >>> currently running version 0.8.4 but this is data that was carried = over > >>> from much earlier versions. Should I try to run scrub or are = there > >>> any other general guidelines for dealing with this sort of error? > >>> > >>> Thanks everyone! > >>> > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra = support > >> http://www.datastax.com > > > > >=20 --Apple-Mail=_80A423D1-7109-4AA3-9EE4-C86AC2C5CA5B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 tmp = files are temporary in nature, if they fail to live up to their name = then they are against nature. 

If they are still = been written to then something is happening, either a flush or a = compaction. Check nodetool compactstats

They = may indicate errors during the scrub (or compact), check the logs for = ERRORs. Note that scrub may log a lot non crashing errors that it could = read a row, so read the error message = carefully. 

When the server restarts it = will clear tmp files.   
 
So if you = unnerved at the prospect of unnatural temp files, exorcise them casting = a restart spell. 

Hope that = helps. 

http://www.thelastpickle.com

On 2/10/2011, at 11:41 AM, Eric Czech wrote:

Scrub = seems to have worked.  Thanks again!

Will a = major compaction delete the "tmp" sstables genereated though? =  Scrub seems to have generated a lot of them and they're taking up = an unnerving amount of disk space.

On Mon, Sep 19, 2011 at 5:34 PM, Eric = Czech <eric@nextbigsound.com>= wrote:
Ok then I'll shutdown the server, change the access mode, restart, = and
run scrub (and then change the access mode back).

Thanks for the pointers and I'll let you know how it goes one way or the = other.

On Tue, Sep 20, 2011 at 12:29 AM, aaron morton <aaron@thelastpickle.com> = wrote:
> I've also found it useful to disable memmapped file access until = the scrub is complete by adding this to the yaml
>
> disk_access_mode: standard
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/09/2011, at 6:55 AM, Jonathan Ellis wrote:
>
>> You should start with scrub.
>>
>> On Mon, Sep 19, 2011 at 1:04 PM, Eric Czech <eric@nextbigsound.com> = wrote:
>>> I'm getting a lot of errors that look something like = "java.io.IOError:
>>> java.io.IOException: mmap segment underflow; remaining is = 348268797
>>> but 892417075 requested" on one node in a 10 node cluster. =  I'm
>>> currently running version 0.8.4 but this is data that was = carried over
>>> from much earlier versions.  Should I try to run scrub = or are there
>>> any other general guidelines for dealing with this sort of = error?
>>>
>>> Thanks everyone!
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra = support
>> http://www.datastax.com
>
>


= --Apple-Mail=_80A423D1-7109-4AA3-9EE4-C86AC2C5CA5B--