From user-return-4590-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Apr 19 23:49:01 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 26138 invoked from network); 19 Apr 2010 23:49:01 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Apr 2010 23:49:01 -0000 Received: (qmail 63798 invoked by uid 500); 19 Apr 2010 23:49:00 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 63769 invoked by uid 500); 19 Apr 2010 23:49:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 63760 invoked by uid 99); 19 Apr 2010 23:49:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 23:49:00 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [131.215.239.119] (HELO mail.alumni.caltech.edu) (131.215.239.119) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 23:48:52 +0000 Received: from localhost (dsl081-082-089.lax1.dsl.speakeasy.net [64.81.82.89]) by mail.alumni.caltech.edu (Postfix) with ESMTPSA id E1EE73F071F; Mon, 19 Apr 2010 16:48:23 -0700 (PDT) X-DKIM: Sendmail DKIM Filter v2.8.2 mail.alumni.caltech.edu E1EE73F071F DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alumni.caltech.edu; s=enforce; t=1271720904; bh=ZDzhrZ/m0hNs4dEVPIWYBO+GTiyb0yGQBWmmZ5upMso=; h=Date:From:To:Subject:Message-ID:References:Mime-Version: Content-Type:In-Reply-To; b=abin/STohgVHMH6VXg0RLD1455KFN0rZPx2Zp7uqNGAEmhOCc2wq8tHOl93YEWjQy epcRx4BTHElBGLcz8SxoLIDrEfEGRFb38iq/X7lMtPftZq6rtwEfNiF6yEW03KwpT6 FO6GPuodqPDBWeA/N4QULtZS04QnLURXxcutAMNs= Date: Mon, 19 Apr 2010 16:48:23 -0700 From: Anthony Molinaro To: user@cassandra.apache.org Subject: Re: Clarification on Ring operations in Cassandra 0.5.1 Message-ID: <20100419234823.GC30486@alumni.caltech.edu> Mail-Followup-To: user@cassandra.apache.org References: <20100415231044.GA8977@alumni.caltech.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-MailScanner-Information-Alumni: X-Alumni-MailScanner-ID: E1EE73F071F.ABD1F X-MailScanner-Alumni: No Virii found X-Spam-Status-Alumni: not spam, SpamAssassin (not cached, score=-3.268, required 5, ALL_TRUSTED -1.80, BAYES_00 -2.60, DNS_FROM_OPENWHOIS 1.13, FH_DATE_PAST_20XX 0.00) X-MailScanner-From: anthonym@alumni.caltech.edu X-Virus-Checked: Checked by ClamAV on apache.org On Mon, Apr 19, 2010 at 03:28:26PM -0500, Jonathan Ellis wrote: > > Can I then 'nodeprobe move ', and > > achieve the same as step 2 above? > > You can't have two nodes with the same token in the ring at once. So, > you can removetoken the old node first, then bootstrap the new one > (just specify InitialToken in the config to avoid having it guess > one), or you can make it a 3 step process (bootstrap, remove, move) to > avoid transferring so much data around. So I'm still a little fuzzy for your 3 step case on why less data moves, but let me run through the two scenarios and see where we get. Please correct me if I'm wrong on some point. Let say I have 3 nodes with random partitioner and rack unaware strategy. Which means I have something like Node Size Token KeyRange (self + next in ring) ---- ---- ----- ------------------------------ A 5 G 33 1 -> 66 B 6 G 66 34 -> 0 C 2 G 0 67 -> 33 Now lets say Node B is giving us some problems, so we want to replace it with another node D. We've outlined 2 processes. In the first process you recommend 1. removetoken on node B 2. wait for data to move 3. add InitialToken of 66 and AutoBootstrap = true to node D storage-conf.xml then start it 4. wait for data to move So when you do the removetoken, this will cause the following transfers at stage 2 Node A sends 34->66 to Node C Node C sends 67->0 to Node A at stage 4 Node A sends 34->66 to Node D Node C sends 67->0 to Node D In the second process I assume you pick a token really close to another token? 1. add InitialToken of 34 and AutoBootstrap to true to node D storage-conf.xml then start it 2. wait for data to move 3. removetoken on node B 4. wait for data to move 5. movetoken on node D to 66 6. wait for data to move This results in the following moves at stage 2 Node A/B sends 33->34 to Node D (primary token range) Node B sends 34->66 to Node D (replica range) at stage 4 Node C sends 66->0 to Node D (replica range) at stage 6 No data movement as D already had 33->0 So seems like you move all the data twice for process 1 and only a small portion twice for process 2 (which is what you said, so hopefully I've outlined correctly what is happening). Does all that sound right? Once I've run bootstrap with the InitialToken value set in the config is it then ignored in subsequent restarts, and if so can I just remove it after that first time? Thanks, -Anthony -- ------------------------------------------------------------------------ Anthony Molinaro