Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DA840D3DF for ; Wed, 20 Jun 2012 04:33:27 +0000 (UTC) Received: (qmail 85940 invoked by uid 500); 20 Jun 2012 04:33:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 85614 invoked by uid 500); 20 Jun 2012 04:33:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 85584 invoked by uid 99); 20 Jun 2012 04:33:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jun 2012 04:33:23 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nick@datastax.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-ob0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jun 2012 04:33:18 +0000 Received: by obbwc20 with SMTP id wc20so11260144obb.31 for ; Tue, 19 Jun 2012 21:32:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding:x-gm-message-state; bh=I+H49/qXz+1+8/9XdrH5rv/GBOtxGP50DICuAxOQSzM=; b=caHfVmNNN6m9EytXpJN7zEMCLMA8UJa/VyO25ctIFWekkAGm7Urex7MFI23bA18WOk cVozUhwgStfqYnqg9uqAOd1Y6mp4zWhjt5q9ObhKpWELpwpSQhaGRbccOvNHlvlv6qUW Hmc0E0bgtXdUMEET1RMCo/foN5H9BpA7+M2aUUOz4FawQmWru9EkFGTWc6cM3HefkKMM YEWEYEHK1Wm7eDCyadf0sLn8fmjuU4a1SeDEVRe/aWeYjediEoKrgeYNHY6TH04Xnehc /k3kDfdFB7K3pfULRi1qPhOeGcDHZy6PHDhM218nYWs9GTijZUhQydfoeKJhOW2D5n7Q tAGg== Received: by 10.60.9.134 with SMTP id z6mr22052890oea.46.1340166777724; Tue, 19 Jun 2012 21:32:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.17.100 with HTTP; Tue, 19 Jun 2012 21:32:37 -0700 (PDT) In-Reply-To: References: From: Nick Bailey Date: Tue, 19 Jun 2012 23:32:37 -0500 Message-ID: Subject: Re: Unbalanced ring in Cassandra 0.8.4 To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQlA0hQ7d50zwd2I2xrsU5mn1khFgbK0QlN8R+anDyQikXCoqJHVwGIkPgLKEwqzfJ0xIGFd X-Virus-Checked: Checked by ClamAV on apache.org No. Cleanup will scan each sstable to remove data that is no longer owned by that specific node. It won't compact the sstables together however. On Tue, Jun 19, 2012 at 11:11 PM, Raj N wrote: > But wont that also run a major compaction which is not recommended anymor= e. > > -Raj > > > On Sun, Jun 17, 2012 at 11:58 PM, aaron morton > wrote: >> >> Assuming you have been running repair, it' can't hurt. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 17/06/2012, at 4:06 AM, Raj N wrote: >> >> Nick, do you think I should still run cleanup on the first node. >> >> -Rajesh >> >> On Fri, Jun 15, 2012 at 3:47 PM, Raj N wrote: >>> >>> I did run nodetool move. But that was when I was setting up the cluster >>> which means I didn't have any data at that time. >>> >>> -Raj >>> >>> >>> On Fri, Jun 15, 2012 at 1:29 PM, Nick Bailey wrote: >>>> >>>> Did you start all your nodes at the correct tokens or did you balance >>>> by moving them? Moving nodes around won't delete unneeded data after >>>> the move is done. >>>> >>>> Try running 'nodetool cleanup' on all of your nodes. >>>> >>>> On Fri, Jun 15, 2012 at 12:24 PM, Raj N wrot= e: >>>> > Actually I am not worried about the percentage. Its the data I am >>>> > concerned >>>> > about. Look at the first node. It has 102.07GB data. And the other >>>> > nodes >>>> > have around 60 GB(one has 69, but lets ignore that one). I am not >>>> > understanding why the first node has almost double the data. >>>> > >>>> > Thanks >>>> > -Raj >>>> > >>>> > >>>> > On Fri, Jun 15, 2012 at 11:06 AM, Nick Bailey >>>> > wrote: >>>> >> >>>> >> This is just a known problem with the nodetool output and multiple >>>> >> DCs. Your configuration is correct. The problem with nodetool is >>>> >> fixed >>>> >> in 1.1.1 >>>> >> >>>> >> https://issues.apache.org/jira/browse/CASSANDRA-3412 >>>> >> >>>> >> On Fri, Jun 15, 2012 at 9:59 AM, Raj N >>>> >> wrote: >>>> >> > Hi experts, >>>> >> > =A0 =A0 I have a 6 node cluster across 2 DCs(DC1:3, DC2:3). I hav= e >>>> >> > assigned >>>> >> > tokens using the first strategy(adding 1) mentioned here - >>>> >> > >>>> >> > http://wiki.apache.org/cassandra/Operations?#Token_selection >>>> >> > >>>> >> > But when I run nodetool ring on my cluster, this is the result I >>>> >> > get - >>>> >> > >>>> >> > Address =A0 =A0 =A0 =A0 DC =A0Rack =A0Status State =A0 Load =A0 = =A0 =A0 =A0Owns =A0 =A0Token >>>> >> > >>>> >> > =A0113427455640312814857969558651062452225 >>>> >> > 172.17.72.91 =A0 =A0DC1 RAC13 Up =A0 =A0 Normal =A0102.07 GB =A0 = 33.33% =A00 >>>> >> > 45.10.80.144 =A0 =A0DC2 RAC5 =A0Up =A0 =A0 Normal =A059.1 GB =A0 = =A0 0.00% =A0 1 >>>> >> > 172.17.72.93 =A0 =A0DC1 RAC18 Up =A0 =A0 Normal =A059.57 GB =A0 = =A033.33% >>>> >> > =A056713727820156407428984779325531226112 >>>> >> > 45.10.80.146 =A0 =A0DC2 RAC7 =A0Up =A0 =A0 Normal =A059.64 GB =A0= =A00.00% >>>> >> > 56713727820156407428984779325531226113 >>>> >> > 172.17.72.95 =A0 =A0DC1 RAC19 Up =A0 =A0 Normal =A069.58 GB =A0 = =A033.33% >>>> >> > =A0113427455640312814857969558651062452224 >>>> >> > 45.10.80.148 =A0 =A0DC2 RAC9 =A0Up =A0 =A0 Normal =A059.31 GB =A0= =A00.00% >>>> >> > 113427455640312814857969558651062452225 >>>> >> > >>>> >> > >>>> >> > As you can see the first node has considerably more load than the >>>> >> > others(almost double)=A0which is surprising since all these are >>>> >> > replicas >>>> >> > of >>>> >> > each other. I am running Cassandra 0.8.4. Is there an explanation >>>> >> > for >>>> >> > this >>>> >> > behaviour? >>>> >> > Could=A0https://issues.apache.org/jira/browse/CASSANDRA-2433=A0be >>>> >> > the >>>> >> > cause for this? >>>> >> > >>>> >> > Thanks >>>> >> > -Raj >>>> > >>>> > >>> >>> >> >> >