Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 561D47685 for ; Wed, 14 Sep 2011 08:54:20 +0000 (UTC) Received: (qmail 13749 invoked by uid 500); 14 Sep 2011 08:54:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13727 invoked by uid 500); 14 Sep 2011 08:54:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13718 invoked by uid 99); 14 Sep 2011 08:54:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Sep 2011 08:54:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of springrider@gmail.com designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Sep 2011 08:54:12 +0000 Received: by eye4 with SMTP id 4so869130eye.31 for ; Wed, 14 Sep 2011 01:53:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=dtIbuvySnT87JkFcxN8hcgkhNXZQ9f44D6FJbBOb0/Y=; b=rUxHwR+XYJzy/fGAatmRhQETTCjrktMJuZbQCZgMrPpfuN5D/d5FSfVt6+OKN4AI5u c+CROnh9o0tsIOzJjNEuiwQ/nsCyIhWd72VyQKOrif6qeZQhMMwvNdIOFrennL24z7Cs SDlQeMkjb0PDKhOIBpr+YBuXdiOw5YoCXUrGo= Received: by 10.213.104.143 with SMTP id p15mr94235ebo.21.1315990432279; Wed, 14 Sep 2011 01:53:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.32.199 with HTTP; Wed, 14 Sep 2011 01:53:32 -0700 (PDT) In-Reply-To: References: From: Yan Chunlu Date: Wed, 14 Sep 2011 16:53:32 +0800 Message-ID: Subject: Re: what's the difference between repair CF separately and repair the entire node? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0015174c3ffe3e6e9104ace2e64e X-Virus-Checked: Checked by ClamAV on apache.org --0015174c3ffe3e6e9104ace2e64e Content-Type: text/plain; charset=ISO-8859-1 thanks a lot for the help! I have read the post and think 0.8 might be good enough for me, especially 0.8.5. also change gc_grace_seconds is a acceptable solution. On Wed, Sep 14, 2011 at 4:03 PM, Sylvain Lebresne wrote: > On Wed, Sep 14, 2011 at 9:27 AM, Yan Chunlu wrote: > > is 0.8 ready for production use? > > some related discussion here: > http://www.mail-archive.com/user@cassandra.apache.org/msg17055.html > but my personal answer is yes. > > > as I know currently many companies including reddit.com are using 0.7, > how > > does they get rid of the repair problem? > > Repair problems in 0.7 don't hit everyone equally. For some people, it > works > relatively well even if not in the most efficient ways. Also, for some > workload > (if you don't do much deletes for instance), you can set a big > gc_grace_seconds > value (say a month) and only run repair that often, which can make repair > inefficiencies more bearable. > That being said, I can't speak for "many companies", but I do advise > evaluating > an upgrade to 0.8. > > -- > Sylvain > > > > > On Wed, Sep 14, 2011 at 2:47 PM, Sylvain Lebresne > > wrote: > >> > >> On Wed, Sep 14, 2011 at 2:38 AM, Yan Chunlu > wrote: > >> > me neither don't want to repair one CF at the time. > >> > the "node repair" took a week and still running, compactionstats and > >> > netstream shows nothing is running on every node, and also no error > >> > message, no exception, really no idea what was it doing, > >> > >> To add to the list of things repair does wrong in 0.7, we'll have to add > >> that > >> if one of the node participating in the repair (so any node that share a > >> range > >> with the node on which repair was started) goes down (even for a short > >> time), > >> then the repair will simply hang forever doing nothing. And no specific > >> error message will be logged. That could be what happened. Again, recent > >> releases of 0.8 fix that too. > >> > >> -- > >> Sylvain > >> > >> > I stopped yesterday. maybe I should run repair again while disable > >> > compaction on all nodes? > >> > thanks! > >> > > >> > On Wed, Sep 14, 2011 at 6:57 AM, Peter Schuller > >> > wrote: > >> >> > >> >> > I think it is a serious problem since I can not "repair"..... I am > >> >> > using cassandra on production servers. is there some way to fix it > >> >> > without upgrade? I heard of that 0.8.x is still not quite ready in > >> >> > production environment. > >> >> > >> >> It is a serious issue if you really need to repair one CF at the > time. > >> >> However, looking at your original post it seems this is not > >> >> necessarily your issue. Do you need to, or was your concern rather > the > >> >> overall time repair took? > >> >> > >> >> There are other things that are improved in 0.8 that affect 0.7. In > >> >> particular, (1) in 0.7 compaction, including validating compactions > >> >> that are part of repair, is non-concurrent so if your repair starts > >> >> while there is a long-running compaction going it will have to wait, > >> >> and (2) semi-related is that the merkle tree calculation that is part > >> >> of repair/anti-entropy may happen "out of synch" if one of the nodes > >> >> participating happen to be busy with compaction. This in turns causes > >> >> additional data to be sent as part of repair. > >> >> > >> >> That might be why your immediately following repair took a long time, > >> >> but it's difficult to tell. > >> >> > >> >> If you're having issues with repair and large data sets, I would > >> >> generally say that upgrading to 0.8 is recommended. However, if > you're > >> >> on 0.7.4, beware of > >> >> https://issues.apache.org/jira/browse/CASSANDRA-3166 > >> >> > >> >> -- > >> >> / Peter Schuller (@scode on twitter) > >> > > >> > > > > > > --0015174c3ffe3e6e9104ace2e64e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable thanks a lot for the help!=A0

=A0I have read the post an= d think 0.8 might be good enough for me, especially 0.8.5.

also change gc_grace_seconds is a acceptable solution.


On Wed, Sep 14, 2011 at 4:03 = PM, Sylvain Lebresne <sylvain@datastax.com> wrote:
On Wed, Sep 14, 2011 at 9:27 AM, Yan Chunlu <springrider@gmail.com> wrote:
> is 0.8 ready for production use?

some related discussion here:
http://www.mail-archive.com/user@cassandra.apache.or= g/msg17055.html
but my personal answer is yes.

> =A0as I know currently many companies including reddit.com are using 0.7, how
> does they get rid of the repair problem?

Repair problems in 0.7 don't hit everyone equally. For some peopl= e, it works
relatively well even if not in the most efficient ways. Also, for some work= load
(if you don't do =A0much deletes for instance), you can set a big gc_gr= ace_seconds
value (say a month) and only run repair that often, which can make repair inefficiencies more bearable.
That being said, I can't speak for "many companies", but I do= advise evaluating
an upgrade to 0.8.

--
Sylvain

>
> On Wed, Sep 14, 2011 at 2:47 PM, Sylvain Lebresne <sylvain@datastax.com>
> wrote:
>>
>> On Wed, Sep 14, 2011 at 2:38 AM, Yan Chunlu <springrider@gmail.com> wrote:
>> > me neither don't want to repair one CF at the time.
>> > the "node repair" took a week and still running, co= mpactionstats and
>> > netstream shows nothing is running on every node, =A0and also= no error
>> > message, no exception, really no idea what was it doing,
>>
>> To add to the list of things repair does wrong in 0.7, we'll h= ave to add
>> that
>> if one of the node participating in the repair (so any node that s= hare a
>> range
>> with the node on which repair was started) goes down (even for a s= hort
>> time),
>> then the repair will simply hang forever doing nothing. And no spe= cific
>> error message will be logged. That could be what happened. Again, = recent
>> releases of 0.8 fix that too.
>>
>> --
>> Sylvain
>>
>> > I stopped yesterday. =A0maybe I should run repair again while= disable
>> > compaction on all nodes?
>> > thanks!
>> >
>> > On Wed, Sep 14, 2011 at 6:57 AM, Peter Schuller
>> > <peter.schu= ller@infidyne.com> wrote:
>> >>
>> >> > I think it is a serious problem since I can not &quo= t;repair"..... =A0I am
>> >> > using cassandra on production servers. is there some= way to fix it
>> >> > without upgrade? =A0I heard of that 0.8.x is still n= ot quite ready in
>> >> > production environment.
>> >>
>> >> It is a serious issue if you really need to repair one CF= at the time.
>> >> However, looking at your original post it seems this is n= ot
>> >> necessarily your issue. Do you need to, or was your conce= rn rather the
>> >> overall time repair took?
>> >>
>> >> There are other things that are improved in 0.8 that affe= ct 0.7. In
>> >> particular, (1) in 0.7 compaction, including validating c= ompactions
>> >> that are part of repair, is non-concurrent so if your rep= air starts
>> >> while there is a long-running compaction going it will ha= ve to wait,
>> >> and (2) semi-related is that the merkle tree calculation = that is part
>> >> of repair/anti-entropy may happen "out of synch"= ; if one of the nodes
>> >> participating happen to be busy with compaction. This in = turns causes
>> >> additional data to be sent as part of repair.
>> >>
>> >> That might be why your immediately following repair took = a long time,
>> >> but it's difficult to tell.
>> >>
>> >> If you're having issues with repair and large data se= ts, I would
>> >> generally say that upgrading to 0.8 is recommended. Howev= er, if you're
>> >> on 0.7.4, beware of
>> >> https://issues.apache.org/jira/browse/CASSANDRA-3= 166
>> >>
>> >> --
>> >> / Peter Schuller (@scode on twitter)
>> >
>> >
>
>

--0015174c3ffe3e6e9104ace2e64e--