Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=UJYhN8ndNdab5sHS5no6Xmot06gToP9qBRwejpQkaso
	srkrOHRZ4ty0HN5nCXT6uaz5hS9BGiGLjeqXl0r1By8W/czUkYMTG8SQIHYmg0fY
	kG0v+7oz+YM7dVtjHO+3JdbP2myfov6MUvhGnvrD3uK4iCRSS4t+vRlg+nCTFC0I
	=
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Apple Message framework v1244.3)
Subject: Re: Cluster key distribution wrong after upgrading to 0.8.4
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CAE_AicRARg92B_HNi1Fep3KwabcgrTxEGwi+34eu8sjmoB2ApA@mail.gmail.com>
Date: Mon, 22 Aug 2011 10:01:37 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <232C9CEA-E04C-44DC-A7D6-64EFB20D53F6@thelastpickle.com>
References: 
 <CAE_AicQJGLmTzB7phnY5+ZLhVCcarFVoLFi02CpYiyuZNghtEw@mail.gmail.com>
 <F7B6BCF9-FBF5-4A7D-B75C-CB686578733D@thelastpickle.com>
 <CAE_AicRARg92B_HNi1Fep3KwabcgrTxEGwi+34eu8sjmoB2ApA@mail.gmail.com>
To: user@cassandra.apache.org

I'm not sure what the fix is.=20

When using an order preserving partitioner it's up to you to ensure the =
ring is correctly balanced.=20

Say you have the following setup=85

node : token
1 : a
2 : h
3 : p

If keys are always 1 character we can say each node own's roughly 33% of =
the ring. Because we know there are only 26 possible keys.=20

With the RP we know how many keys there are, the output of the md5 =
calculation is a 128 bit integer. So we can say what fraction of the =
total each range is.=20

If in the example above keys are of any length, how many values exist =
between a and h ?=20

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 22/08/2011, at 3:33 AM, Thibaut Britz wrote:

> Hi,
>=20
> I will wait until this is fixed beforeI upgrade, just to be sure.
>=20
> Shall I open a new ticket for this issue?
>=20
> Thanks,
> Thibaut
>=20
> On Sun, Aug 21, 2011 at 11:57 AM, aaron morton =
<aaron@thelastpickle.com> wrote:
>> This looks like an artifact of the way ownership is calculated for =
the OOP.
>> See =
https://github.com/apache/cassandra/blob/cassandra-0.8.4/src/java/org/apac=
he/cassandra/dht/OrderPreservingPartitioner.java#L177 it
>> was changed in this ticket
>> https://issues.apache.org/jira/browse/CASSANDRA-2800
>> The change applied in CASSANDRA-2800 was not applied to the
>> AbstractByteOrderPartitioner. Looks like it should have been. I'll =
chase
>> that up.
>>=20
>> When each node calculates the ownership for the token ranges (for OOP =
and
>> BOP) it's based on the number of keys the node has in that range. As =
there
>> is no way for the OOP to understand the range of values the keys may =
take.
>> If you look at the 192 node it's showing ownership most with 192, 191 =
and
>> 190 - so i'm assuming RF3 and 192 also has data from the ranges owned =
by 191
>> and 190.
>> IMHO you can ignore this.
>> You can use load the the number of keys estimate from cfstats to get =
an idea
>> of whats happening.
>> Hope that helps.
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> On 19/08/2011, at 9:42 PM, Thibaut Britz wrote:
>>=20
>> Hi,
>>=20
>> we were using apache-cassandra-2011-06-28_08-04-46.jar so far in
>> production and wanted to upgrade to 0.8.4.
>>=20
>> Our cluster was well balanced and we only saved keys with a lower =
case
>> md5 prefix. (Orderpreserving partitioner).
>> Each node owned 20% of the tokens, which was also displayed on each
>> node in nodetool -h localhost ring.
>>=20
>> After upgrading, our well balanced cluster shows completely wrong
>> percentage on who owns which keys:
>>=20
>> *.*.*.190:
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>=20
>>        ffffffffffffffff
>> *.*.*.190   datacenter1 rack1       Up     Normal  87.95 GB
>> 34.57%  2a
>> *.*.*.191   datacenter1 rack1       Up     Normal  84.3 GB
>> 0.02%   55
>> *.*.*.192   datacenter1 rack1       Up     Normal  79.46 GB
>> 0.02%   80
>> *.*.*.194   datacenter1 rack1       Up     Normal  68.16 GB
>> 0.02%   aa
>> *.*.*.196   datacenter1 rack1       Up     Normal  79.9 GB
>> 65.36%  ffffffffffffffff
>>=20
>> *.*.*.191:
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>=20
>>        ffffffffffffffff
>> *.*.*.190   datacenter1 rack1       Up     Normal  87.95 GB
>> 36.46%  2a
>> *.*.*.191   datacenter1 rack1       Up     Normal  84.3 GB
>> 26.02%  55
>> *.*.*.192   datacenter1 rack1       Up     Normal  79.46 GB
>> 0.02%   80
>> *.*.*.194   datacenter1 rack1       Up     Normal  68.16 GB
>> 0.02%   aa
>> *.*.*.196   datacenter1 rack1       Up     Normal  79.9 GB
>> 37.48%  ffffffffffffffff
>>=20
>> *.*.*.192:
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>=20
>>        ffffffffffffffff
>> *.*.*.190   datacenter1 rack1       Up     Normal  87.95 GB
>> 38.16%  2a
>> *.*.*.191   datacenter1 rack1       Up     Normal  84.3 GB
>> 27.61%  55
>> *.*.*.192   datacenter1 rack1       Up     Normal  79.46 GB
>> 34.17%  80
>> *.*.*.194   datacenter1 rack1       Up     Normal  68.16 GB
>> 0.02%   aa
>> *.*.*.196   datacenter1 rack1       Up     Normal  79.9 GB
>> 0.02%   ffffffffffffffff
>>=20
>> *.*.*.194:
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>=20
>>        ffffffffffffffff
>> *.*.*.190   datacenter1 rack1       Up     Normal  87.95 GB
>> 0.03%   2a
>> *.*.*.191   datacenter1 rack1       Up     Normal  84.3 GB
>> 31.43%  55
>> *.*.*.192   datacenter1 rack1       Up     Normal  79.46 GB
>> 39.69%  80
>> *.*.*.194   datacenter1 rack1       Up     Normal  68.16 GB
>> 28.82%  aa
>> *.*.*.196   datacenter1 rack1       Up     Normal  79.9 GB
>> 0.03%   ffffffffffffffff
>>=20
>> *.*.*.196:
>> Address         DC          Rack        Status State   Load
>> Owns    Token
>>=20
>>        ffffffffffffffff
>> *.*.*.190   datacenter1 rack1       Up     Normal  87.95 GB
>> 0.02%   2a
>> *.*.*.191   datacenter1 rack1       Up     Normal  84.3 GB
>> 0.02%   55
>> *.*.*.192   datacenter1 rack1       Up     Normal  79.46 GB
>> 0.02%   80
>> *.*.*.194   datacenter1 rack1       Up     Normal  68.16 GB
>> 27.52%  aa
>> *.*.*.196   datacenter1 rack1       Up     Normal  79.9 GB
>> 72.42%  ffffffffffffffff
>>=20
>>=20
>> Interestingly, each server shows something completely different.
>>=20
>> Removing the locationInfo files didn't help.
>> -Dcassandra.load_ring_state=3Dfalse didn't help as well.
>>=20
>> Our cassandra.yaml is at http://pastebin.com/pCVCt3RM
>>=20
>> Any idea on what might cause this? Is it save to suspect that
>> operating under this distribution will cause severe data loss? Or can
>> I safely ignore this?
>>=20
>> Thanks,
>> Thibaut
>>=20
>>=20