Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9DCFB7175 for ; Sun, 21 Aug 2011 15:34:16 +0000 (UTC) Received: (qmail 22136 invoked by uid 500); 21 Aug 2011 15:34:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 22106 invoked by uid 500); 21 Aug 2011 15:34:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22098 invoked by uid 99); 21 Aug 2011 15:34:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 15:34:13 +0000 X-ASF-Spam-Status: No, hits=0.3 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.174] (HELO mail-iy0-f174.google.com) (209.85.210.174) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 15:34:07 +0000 Received: by iyf40 with SMTP id 40so9564539iyf.33 for ; Sun, 21 Aug 2011 08:33:45 -0700 (PDT) Received: by 10.231.60.139 with SMTP id p11mr3484857ibh.73.1313940825565; Sun, 21 Aug 2011 08:33:45 -0700 (PDT) Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mx.google.com with ESMTPS id a9sm2852323ibi.9.2011.08.21.08.33.45 (version=SSLv3 cipher=OTHER); Sun, 21 Aug 2011 08:33:45 -0700 (PDT) Received: by iyf40 with SMTP id 40so9564525iyf.33 for ; Sun, 21 Aug 2011 08:33:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.43.43.134 with SMTP id uc6mr1730604icb.7.1313940825043; Sun, 21 Aug 2011 08:33:45 -0700 (PDT) Received: by 10.231.79.133 with HTTP; Sun, 21 Aug 2011 08:33:45 -0700 (PDT) In-Reply-To: References: Date: Sun, 21 Aug 2011 17:33:45 +0200 Message-ID: Subject: Re: Cluster key distribution wrong after upgrading to 0.8.4 From: Thibaut Britz To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi, I will wait until this is fixed beforeI upgrade, just to be sure. Shall I open a new ticket for this issue? Thanks, Thibaut On Sun, Aug 21, 2011 at 11:57 AM, aaron morton wr= ote: > This looks like an artifact of the way ownership is calculated for the OO= P. > See=A0https://github.com/apache/cassandra/blob/cassandra-0.8.4/src/java/o= rg/apache/cassandra/dht/OrderPreservingPartitioner.java#L177=A0it > was changed in this ticket > https://issues.apache.org/jira/browse/CASSANDRA-2800 > The change applied in CASSANDRA-2800 was not applied to the > AbstractByteOrderPartitioner. Looks like it should have been. I'll chase > that up. > > When each node calculates the ownership for the token ranges (for OOP and > BOP) it's based on the number of keys the node has in that range. As ther= e > is no way for the OOP to understand the range of values the keys may take= . > If you look at the 192 node it's showing ownership most with 192, 191 and > 190 - so i'm assuming RF3 and 192 also has data from the ranges owned by = 191 > and 190. > IMHO you can ignore this. > You can use load the the number of keys estimate from cfstats to get an i= dea > of whats happening. > Hope that helps. > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > On 19/08/2011, at 9:42 PM, Thibaut Britz wrote: > > Hi, > > we were using apache-cassandra-2011-06-28_08-04-46.jar so far in > production and wanted to upgrade to 0.8.4. > > Our cluster was well balanced and we only saved keys with a lower case > md5 prefix. (Orderpreserving partitioner). > Each node owned 20% of the tokens, which was also displayed on each > node in nodetool -h localhost ring. > > After upgrading, our well balanced cluster shows completely wrong > percentage on who owns which keys: > > *.*.*.190: > Address =A0=A0=A0=A0=A0=A0=A0=A0DC =A0=A0=A0=A0=A0=A0=A0=A0=A0Rack =A0=A0= =A0=A0=A0=A0=A0Status State =A0=A0Load > Owns =A0=A0=A0Token > > =A0=A0=A0=A0=A0=A0=A0ffffffffffffffff > *.*.*.190 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A087.95 GB > 34.57% =A02a > *.*.*.191 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A084.3 GB > 0.02% =A0=A055 > *.*.*.192 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.46 GB > 0.02% =A0=A080 > *.*.*.194 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A068.16 GB > 0.02% =A0=A0aa > *.*.*.196 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.9 GB > 65.36% =A0ffffffffffffffff > > *.*.*.191: > Address =A0=A0=A0=A0=A0=A0=A0=A0DC =A0=A0=A0=A0=A0=A0=A0=A0=A0Rack =A0=A0= =A0=A0=A0=A0=A0Status State =A0=A0Load > Owns =A0=A0=A0Token > > =A0=A0=A0=A0=A0=A0=A0ffffffffffffffff > *.*.*.190 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A087.95 GB > 36.46% =A02a > *.*.*.191 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A084.3 GB > 26.02% =A055 > *.*.*.192 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.46 GB > 0.02% =A0=A080 > *.*.*.194 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A068.16 GB > 0.02% =A0=A0aa > *.*.*.196 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.9 GB > 37.48% =A0ffffffffffffffff > > *.*.*.192: > Address =A0=A0=A0=A0=A0=A0=A0=A0DC =A0=A0=A0=A0=A0=A0=A0=A0=A0Rack =A0=A0= =A0=A0=A0=A0=A0Status State =A0=A0Load > Owns =A0=A0=A0Token > > =A0=A0=A0=A0=A0=A0=A0ffffffffffffffff > *.*.*.190 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A087.95 GB > 38.16% =A02a > *.*.*.191 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A084.3 GB > 27.61% =A055 > *.*.*.192 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.46 GB > 34.17% =A080 > *.*.*.194 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A068.16 GB > 0.02% =A0=A0aa > *.*.*.196 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.9 GB > 0.02% =A0=A0ffffffffffffffff > > *.*.*.194: > Address =A0=A0=A0=A0=A0=A0=A0=A0DC =A0=A0=A0=A0=A0=A0=A0=A0=A0Rack =A0=A0= =A0=A0=A0=A0=A0Status State =A0=A0Load > Owns =A0=A0=A0Token > > =A0=A0=A0=A0=A0=A0=A0ffffffffffffffff > *.*.*.190 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A087.95 GB > 0.03% =A0=A02a > *.*.*.191 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A084.3 GB > 31.43% =A055 > *.*.*.192 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.46 GB > 39.69% =A080 > *.*.*.194 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A068.16 GB > 28.82% =A0aa > *.*.*.196 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.9 GB > 0.03% =A0=A0ffffffffffffffff > > *.*.*.196: > Address =A0=A0=A0=A0=A0=A0=A0=A0DC =A0=A0=A0=A0=A0=A0=A0=A0=A0Rack =A0=A0= =A0=A0=A0=A0=A0Status State =A0=A0Load > Owns =A0=A0=A0Token > > =A0=A0=A0=A0=A0=A0=A0ffffffffffffffff > *.*.*.190 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A087.95 GB > 0.02% =A0=A02a > *.*.*.191 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A084.3 GB > 0.02% =A0=A055 > *.*.*.192 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.46 GB > 0.02% =A0=A080 > *.*.*.194 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A068.16 GB > 27.52% =A0aa > *.*.*.196 =A0=A0datacenter1 rack1 =A0=A0=A0=A0=A0=A0Up =A0=A0=A0=A0Normal= =A079.9 GB > 72.42% =A0ffffffffffffffff > > > Interestingly, each server shows something completely different. > > Removing the locationInfo files didn't help. > -Dcassandra.load_ring_state=3Dfalse didn't help as well. > > Our cassandra.yaml is at http://pastebin.com/pCVCt3RM > > Any idea on what might cause this? Is it save to suspect that > operating under this distribution will cause severe data loss? Or can > I safely ignore this? > > Thanks, > Thibaut > >