Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 92843 invoked from network); 17 Sep 2010 19:24:42 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 17 Sep 2010 19:24:42 -0000 Received: (qmail 92517 invoked by uid 500); 17 Sep 2010 19:24:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 92472 invoked by uid 500); 17 Sep 2010 19:24:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 92464 invoked by uid 99); 17 Sep 2010 19:24:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 19:24:39 +0000 X-ASF-Spam-Status: No, hits=3.3 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,TRACKER_ID,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gurpreet.singh@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 19:24:32 +0000 Received: by iwn3 with SMTP id 3so2609151iwn.31 for ; Fri, 17 Sep 2010 12:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=AvrfqXc5VREdqjK04sT35FsJIZ5AHe1C+PYgeNmP4tk=; b=WTSBICRxltSLNh4j/y4YHuo7DWE4Y09ieGva/8d00+4r+A/sBpTqlohtJ140wVEYSw HEPsSsP0bIL12zX30Sevk3UFpw6hR/eFUF7bFK5s3jFimpEBZJbr22TyLYqZgs5ZhQOk At/8o1SvlCnXONaOBl+Ak0Zha3jDIEba7kNFc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=QqPyQTM2adS54nD1VlHbePO5tTiValZUyPqmY0QDUNgjwjHVgc3WMBLhVe8FXJG8sA CYc2KEnu2SO6lUGNisYUl7SgkEDB2OLl+hZVSV5q57hpVxWT0wXki/EpjgAQDCiJV/zr bgyDgHVHl/kfU8YHs/dMpG4goMF7Q1h7vKnIY= MIME-Version: 1.0 Received: by 10.231.16.75 with SMTP id n11mr5462810iba.49.1284751450640; Fri, 17 Sep 2010 12:24:10 -0700 (PDT) Received: by 10.231.199.198 with HTTP; Fri, 17 Sep 2010 12:24:10 -0700 (PDT) In-Reply-To: References: Date: Fri, 17 Sep 2010 12:24:10 -0700 Message-ID: Subject: Re: questions on cassandra (repair and multi-datacenter) From: Gurpreet Singh To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=002215046dcfd7144004907981b9 X-Virus-Checked: Checked by ClamAV on apache.org --002215046dcfd7144004907981b9 Content-Type: text/plain; charset=ISO-8859-1 Hi Benjamin, I reverted back to the old RF of 2, by restarting all nodes with RF 2, and then running cleanup. It came down to 2. This time, i now changed the RF to 3 for all machines and restarted all the nodes. I started running repair one by one on all machines, tracking through jconsole that compaction (readonly) was happenning, and moved over to the next node only when there was no compaction going on, and nothing in the AES stage. The process is finished on all machines, and took a long time i have to say. However my ring shows almost 8 times load. Here it is. Original data size was about 100 gigs, with RF 2 it was about 200 gigs. But here its almost 800 gigs. What could i be doing wrong? Address Status Load Range Ring 128045052799293308972222897669231312075 ip1 Up 100.09 GB 418948754358022090024968091731975038 |<--| ip2 Up 90.83 GB 11057191649494821574112431245768166381 | ^ ip3 Up 105.9 GB 21705674247570520134925875025273670789 v | ip4 Up 122.7 GB 42980788726886850121690234508345696103 | ^ ip5 Up 106.24 GB 85510669386552904928042350150568641153 v | ip6 Up 194.27 GB 106767287274351479790232508363491106683 | ^ ip7 Up 87.3 GB 128045052799293308972222897669231312075 |-->| /G On Thu, Sep 16, 2010 at 11:56 PM, Gurpreet Singh wrote: > Thanks Benjamin. I realised that, i have reverted using cleanup, got it > back to old state and testing the scenario exactly the way you put it. > > > On Thu, Sep 16, 2010 at 10:56 PM, Benjamin Black wrote: > >> On Thu, Sep 16, 2010 at 3:19 PM, Gurpreet Singh >> wrote: >> > 1. I was looking to increase the RF to 3. This process entails changing >> the >> > config and calling repair on the keyspace one at a time, right? >> > So, I started with one node at a time, changed the config file on the >> first >> > node for the keyspace, restarted the node. And then called a nodetool >> repair >> > on the node. >> >> You need to change the RF on _all_ nodes in the cluster _before_ >> running repair on _any_ of them. If nodes disagree on which nodes >> should have replicas for keys, repair will not work correctly. >> Different RF for the same keyspace creates that disagreement. >> >> >> b >> > > --002215046dcfd7144004907981b9 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Benjamin,
I reverted back to the old RF of 2, by restarting all node= s with RF 2, and then running cleanup. It came down to 2.
This ti= me, i now changed the RF to 3 for all machines and restarted all the nodes.=

I started running repair one by one on all machines, tr= acking through jconsole that compaction (readonly) was happenning, and move= d over to the next node only when there was no compaction going on, and not= hing in the AES stage.

The process is finished on all machines, and took a lon= g time i have to say. However my ring shows almost 8 times load. Here it is= .
Original data size was about 100 gigs, with RF 2 it was ab= out 200 gigs. But here its almost 800 gigs. What could i be doing wrong?

Address =A0 =A0 =A0 Status =A0 =A0 Load =A0 =A0 = =A0 =A0 =A0Range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0Ring
=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 12804505279929330897222289766923131= 2075 =A0 =A0
ip1 Up =A0 =A0 =A0 =A0 100.09 GB =A0 =A0 41894875435= 8022090024968091731975038 =A0 =A0 =A0 |<--|
ip2 Up =A0 =A0 =A0 =A0 90.83 GB =A0 =A0 =A0110571916494948215741124312= 45768166381 =A0 =A0 | =A0 ^
ip3 Up =A0 =A0 =A0 =A0 105.9 GB =A0 = =A0 =A021705674247570520134925875025273670789 =A0 =A0 v =A0 |
ip4= Up =A0 =A0 =A0 =A0 122.7 GB =A0 =A0 =A042980788726886850121690234508345696= 103 =A0 =A0 | =A0 ^
ip5 Up =A0 =A0 =A0 =A0 106.24 GB =A0 =A0 85510669386552904928042350150= 568641153 =A0 =A0 v =A0 |
ip6 Up =A0 =A0 =A0 =A0 194.27 GB =A0 = =A0 106767287274351479790232508363491106683 =A0 =A0| =A0 ^
ip7 Up= =A0 =A0 =A0 =A0 87.3 GB =A0 =A0 =A0 12804505279929330897222289766923131207= 5 =A0 =A0|-->|

/G
=A0
O= n Thu, Sep 16, 2010 at 11:56 PM, Gurpreet Singh <gurpreet.singh@gmail.com> wrote:
Thanks Benjamin. I realised that, i have re= verted using cleanup, got it back to old state and testing the scenario exa= ctly the way you put it.


On Thu, Sep= 16, 2010 at 10:56 PM, Benjamin Black <b@b3k.us> wrote:
On Thu, Sep 16, 2010 at 3:19 PM, Gurpre= et Singh
<gurpreet.= singh@gmail.com> wrote:
> 1. =A0I was looking to increase the RF to 3. This process entails chan= ging the
> config and calling repair on the keyspace one at a time, right?
> So, I started with one node at a time, changed the config file on the = first
> node for the keyspace, restarted the node. And then called a nodetool = repair
> on the node.

You need to change the RF on _all_ nodes in the cluster _before_
running repair on _any_ of them. =A0If nodes disagree on which nodes
should have replicas for keys, repair will not work correctly.
Different RF for the same keyspace creates that disagreement.


b


--002215046dcfd7144004907981b9--