Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 70AFB11645 for ; Tue, 1 Jul 2014 22:54:25 +0000 (UTC) Received: (qmail 85452 invoked by uid 500); 1 Jul 2014 22:54:22 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 85411 invoked by uid 500); 1 Jul 2014 22:54:22 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 85401 invoked by uid 99); 1 Jul 2014 22:54:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jul 2014 22:54:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of philburresseme@gmail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qa0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jul 2014 22:54:20 +0000 Received: by mail-qa0-f44.google.com with SMTP id hw13so8363857qab.31 for ; Tue, 01 Jul 2014 15:53:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=oU0gDAuLiamNgWg6u/c7kIHchvpC9hiFt8ri2519RFs=; b=MZ/v+/5RYRv3jRUNZj//6qp1iIXrtEtI6dex39M9DlMxTLhG9mlw6UTARWBccB7hJk /qks38ck+/kZAaSw2J2BSSo8FgmwnOiVc+vy+4NGz8przfbgYT7Pj/p7c0bKrVXdT+MP GytdXYxVbrg67ZRwDXFnegcnBJCq+ctO9J6pmxRun3rg+N15YzLUiBKRZAJnqh1nr9SJ 2w6XaZ9HvHIIxsyiC1ya68Hc7jQQiYNVshnZL6vg3e1A3yvrGtbMXf9N4l2kk0tHM81A /wo7fwasAtLU3Jk4kiwrqsGwuJdDf8FqmQFM2Uaz19uNk/75qslewu7h2/ZykZ83XwL2 XQ/Q== MIME-Version: 1.0 X-Received: by 10.140.27.108 with SMTP id 99mr49522359qgw.77.1404255236013; Tue, 01 Jul 2014 15:53:56 -0700 (PDT) Received: by 10.140.109.135 with HTTP; Tue, 1 Jul 2014 15:53:55 -0700 (PDT) In-Reply-To: References: Date: Tue, 1 Jul 2014 18:53:55 -0400 Message-ID: Subject: Re: nodetool repair -snapshot option? From: Phil Burress To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c146e484437304fd29a6ce X-Virus-Checked: Checked by ClamAV on apache.org --001a11c146e484437304fd29a6ce Content-Type: text/plain; charset=UTF-8 Thanks! We retrieved all the ranges and started running repair on them. We ran through all of them but found one single range which brought the ENTIRE cluster down. All of the other ranges ran quickly and smoothly. This one problematic range reliably brings it down every time we try to run repair on it. Any thoughts on why one specific range would be a troublemaker? On Tue, Jul 1, 2014 at 11:44 AM, Ken Hancock wrote: > I also expanded on a script originally written by Matt Stump @ Datastax. > The readme has the reasoning behind requiring sub-range repairs. > > https://github.com/hancockks/cassandra_range_repair > > > > > On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress > wrote: > >> @Paulo, this is very cool! Thanks very much for the link! >> >> >> On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes < >> paulo.motta@chaordicsystems.com> wrote: >> >>> If you find it useful, I created a tool where you input the node IP, >>> keyspace, column family, and optionally the number of partitions (default: >>> 32K), and it outputs the list of subranges for that node, CF, partition >>> size: https://github.com/pauloricardomg/cassandra-list-subranges >>> >>> So you can basically iterate over the output of that and do subrange >>> repair for each node and cf, maybe in parallel. :) >>> >>> >>> On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress >> > wrote: >>> >>>> One last question. Any tips on scripting a subrange repair? >>>> >>>> >>>> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress >>> > wrote: >>>> >>>>> We are running repair -pr. We've tried subrange manually and that >>>>> seems to work ok. I guess we'll go with that going forward. Thanks for all >>>>> the info! >>>>> >>>>> >>>>> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia < >>>>> chovatia.jaydeep@gmail.com> wrote: >>>>> >>>>>> Are you running full repair or on subset? If you are running full >>>>>> repair then try running on sub-set of ranges which means less data to worry >>>>>> during repair and that would help JAVA heap in general. You will have to do >>>>>> multiple iterations to complete entire range but at-least it will work. >>>>>> >>>>>> -jaydeep >>>>>> >>>>>> >>>>>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli >>>>>> wrote: >>>>>> >>>>>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita >>>>>>> wrote: >>>>>>> >>>>>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt). >>>>>>>> >>>>>>> >>>>>>> As a general meta comment, the process by which operationally >>>>>>> important defaults change in Cassandra seems ad-hoc and sub-optimal. >>>>>>> >>>>>>> For to record, my view was that this change, which makes repair even >>>>>>> slower than it previously was, was probably overly optimistic. >>>>>>> >>>>>>> It's also weird in that it changes default behavior which has been >>>>>>> unchanged since the start of Cassandra time and is therefore probably >>>>>>> automated against. Why was it so critically important to switch to snapshot >>>>>>> repair that it needed to be shotgunned as a new default in 2.0.2? >>>>>>> >>>>>>> =Rob >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> *Paulo Motta* >>> >>> Chaordic | *Platform* >>> *www.chaordic.com.br * >>> +55 48 3232.3200 >>> >> >> > > > -- > *Ken Hancock *| System Architect, Advanced Advertising > SeaChange International > 50 Nagog Park > Acton, Massachusetts 01720 > ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC > > Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com > | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image: > LinkedIn] > > [image: SeaChange International] > This e-mail and any attachments may contain > information which is SeaChange International confidential. The information > enclosed is intended only for the addressees herein and may not be copied > or forwarded without permission from SeaChange International. > --001a11c146e484437304fd29a6ce Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks! We retrieved all the ranges and started running re= pair on them. We ran through all of them but found one single range which b= rought the ENTIRE cluster down. All of the other ranges ran quickly and smo= othly. This one problematic range reliably brings it down every time we try= to run repair on it. Any thoughts on why one specific range would be a tro= ublemaker?


On Tue, Jul 1= , 2014 at 11:44 AM, Ken Hancock <ken.hancock@schange.com> wrote:
I also expanded on a script= originally written by Matt Stump @ Datastax. The readme has the reasoning = behind requiring sub-range repairs.

https://github.com/hancockks/cassandra_range_repair




On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress <philburresseme@gmail.com> wrote:
@Paulo, this is very cool! Thanks very= much for the link!


On Mon, Jun 3= 0, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes <paulo.motta@c= haordicsystems.com> wrote:
If you find it useful,= I created a tool where you input the node IP, keyspace, column family, and= optionally the number of partitions (default: 32K), and it outputs the lis= t of subranges for that node, CF, partition size:=C2=A0https:= //github.com/pauloricardomg/cassandra-list-subranges

So you can basically iterate over the output of that and do subran= ge repair for each node and cf, maybe in parallel. :)


On Mon, Jun 3= 0, 2014 at 10:26 PM, Phil Burress <philburresseme@gmail.com>= wrote:
One last question. Any tips= on scripting a subrange repair?


On Mon, Jun 30, 2014 at 7:12 PM, Phil Bu= rress <philburresseme@gmail.com> wrote:
We are running repair -pr. = We've tried subrange manually and that seems to work ok. I guess we'= ;ll go with that going forward. Thanks for all the info!


On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <chovatia.jayde= ep@gmail.com> wrote:
Are you running full repair or on subset? If you are runni= ng full repair then try running on sub-set of ranges which means less data = to worry during repair and that would help JAVA heap in general. You will h= ave to do multiple iterations to complete entire range but at-least it will= work.

-jaydeep


On Mon, Jun 30, 2014 at 3:22 = PM, Robert Coli <rcoli@eventbrite.com> wrote:
=
On Mon, Jun 30, 2014 at 3:08 PM, Yuki Moris= hita <mor.yuki@gmail.com> wrote:
Repair uses snapshot option by default since 2.= 0.2 (see NEWS.txt).

As a general meta comment, the = process by which operationally important defaults change in Cassandra seems= ad-hoc and sub-optimal.

For to record, my v= iew was that this change, which makes repair even slower than it previously= was, was probably overly optimistic.

It's also weird in that it changes default be= havior which has been unchanged since the start of Cassandra time and is th= erefore probably automated against. Why was it so critically important to s= witch to snapshot repair that it needed to be shotgunned as a new default i= n 2.0.2?

=3DRob
=C2=A0






<= /div>--
Paulo Motta





--
Ken Hancock=C2=A0| System Ar= chitect, Advanced Advertising=C2=A0
SeaChange International= =C2=A0
50 Nagog Park
Acton, Massachusetts 01720
ken.hancock@schange.com=C2=A0|=C2=A0www.sc= hange.com=C2=A0| NASDAQ:SEAC=C2=A0
Office: +1 (978) 889-3329=C2=A0|=C2=A03D"Google=C2=A0ken.hancock@scha= nge.com=C2=A0|=C2=A03D"Skype:"hancockks=C2=A0|=C2=A03D"Yahoohancockk= s		 3D"LinkedIn"

3D"SeaChange
This e-mail and any attachments may contain information which is SeaChange= International confidential. The information enclosed is intended only for = the addressees herein and may not be copied or forwarded without permission= from SeaChange International.

--001a11c146e484437304fd29a6ce--