Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 00DB37559 for ; Wed, 17 Aug 2011 12:02:12 +0000 (UTC) Received: (qmail 24160 invoked by uid 500); 17 Aug 2011 12:02:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23795 invoked by uid 500); 17 Aug 2011 12:02:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23782 invoked by uid 99); 17 Aug 2011 12:02:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Aug 2011 12:02:01 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Aug 2011 12:01:55 +0000 Received: by vws12 with SMTP id 12so699901vws.31 for ; Wed, 17 Aug 2011 05:01:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.110.8 with SMTP id hw8mr892739vdb.66.1313582493574; Wed, 17 Aug 2011 05:01:33 -0700 (PDT) Received: by 10.52.157.130 with HTTP; Wed, 17 Aug 2011 05:01:33 -0700 (PDT) In-Reply-To: References: Date: Wed, 17 Aug 2011 07:01:33 -0500 Message-ID: Subject: Re: Partitioning, tokens, and sequential keys From: David McNelis To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec548a4fbe9a71e04aab24180 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec548a4fbe9a71e04aab24180 Content-Type: text/plain; charset=ISO-8859-1 Well, I think what happened was that we had three tokens generated, 0, 567x, and 1134x... but the way that we read the comments in the yaml file, we just set the second two nodes with the initial token and left the token for the seed node blank. Then we started the seed node, started the other two, and then the seed node took on the 613x token. One question on nodetool ring, the "owns" refers to how many of the possible keys each node owns, not the actual node size correct? So you could technically have a load of 15gb, 60gb, and 15gb on a three node cluster, but if you have the tokens set correctly each would own 33.33%. Thanks. On Tue, Aug 16, 2011 at 3:33 PM, Jonathan Ellis wrote: > Yes, that looks about right. > > Totally baffled how the wiki script could spit out those tokens for a > 3-node cluster. > > On Tue, Aug 16, 2011 at 2:04 PM, David McNelis > wrote: > > Currently we have the initial_token for the seed node blank, and then the > > three tokens we ended up with are: > > 56713727820156410577229101238628035242 > > 61396109050359754194262152792166260437 > > 113427455640312821154458202477256070485 > > I would assume that we'd want to take the node that > > is 61396109050359754194262152792166260437 and move it to 0, yes? > > In theory that should largely balance out our data... or am I missing > > something there? > > On Tue, Aug 16, 2011 at 1:54 PM, Jonathan Ellis > wrote: > >> > >> what tokens did you end up using? > >> > >> are you sure it's actually due to different amounts of rows? have you > >> run cleanup and compact to make sure it's not unused data / obsolete > >> replicas taking up the space? > >> > >> On Tue, Aug 16, 2011 at 1:41 PM, David McNelis > >> wrote: > >> > We are currently running a three node cluster where we assigned the > >> > initial > >> > tokens using the Python script that is in the Wiki, and we're > currently > >> > using the Random Partitioner, RF=1, Cassandra 0.8 from the Riptano RPM > >> > ....however we're seeing one node taken on over 60% of the data as we > >> > load > >> > data. > >> > Our keys are sequential, and can range from 0 to 2^64, though in > >> > practice > >> > we're between 1 and 2,000,000,000, with the current max around > 50,000. > >> > In > >> > order to balance out the load would we be best served changing our > >> > tokens > >> > to make the top and bottom 1/3rd of the node go to the previous and > next > >> > nodes respectively, then running nodetool move? > >> > Even if we do that, it would seem that we'd likely continue to run > into > >> > this > >> > sort of issue as we add additionally data... would we be better > >> > served > >> > with a different Partitioner strategy? Or will we need to very > actively > >> > manage our tokens to avoid getting into an unbalanced situation? > >> > > >> > -- > >> > David McNelis > >> > Lead Software Engineer > >> > Agentis Energy > >> > www.agentisenergy.com > >> > o: 630.359.6395 > >> > c: 219.384.5143 > >> > A Smart Grid technology company focused on helping consumers of energy > >> > control an often under-managed resource. > >> > > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra support > >> http://www.datastax.com > > > > > > > > -- > > David McNelis > > Lead Software Engineer > > Agentis Energy > > www.agentisenergy.com > > o: 630.359.6395 > > c: 219.384.5143 > > A Smart Grid technology company focused on helping consumers of energy > > control an often under-managed resource. > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > -- *David McNelis* Lead Software Engineer Agentis Energy www.agentisenergy.com o: 630.359.6395 c: 219.384.5143 *A Smart Grid technology company focused on helping consumers of energy control an often under-managed resource.* --bcaec548a4fbe9a71e04aab24180 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Well, I think what =A0happened was that we had three tokens generated, 0, 5= 67x, and 1134x... but the way that we read the comments in the yaml file, w= e just set the second two nodes with the initial token and left the token f= or the seed node =A0blank. =A0Then we started the seed node, started the ot= her =A0two, and then the seed node took on the 613x token.

One question on nodetool ring, the "owns" refers t= o how many of the possible keys each node owns, not the actual node size co= rrect? =A0So you could technically have a load of 15gb, 60gb, and 15gb on a= three node cluster, but if you have the tokens set correctly each would ow= n 33.33%.

Thanks.

On Tue, Aug 1= 6, 2011 at 3:33 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
Yes, that looks about right.

Totally baffled how the wiki script could spit out those tokens for a
3-node cluster.

On Tue, Aug 16, 2011 at 2:04 PM, David McNelis
<dmcnelis@agentisenergy.com> wrote:
> Currently we have the initial_token for the seed node blank, and then = the
> three tokens we ended =A0up with are:
> 56713727820156410577229101238628035242
> 61396109050359754194262152792166260437
> 113427455640312821154458202477256070485
> I would assume that we'd want to take the node that
> is=A061396109050359754194262152792166260437 and move it to 0, yes?
> In theory that should largely balance out our data... or am I missing<= br> > something there?
> On Tue, Aug 16, 2011 at 1:54 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> what tokens did you end up using?
>>
>> are you sure it's actually due to different amounts of rows? = =A0have you
>> run cleanup and compact to make sure it's not unused data / ob= solete
>> replicas taking up the space?
>>
>> On Tue, Aug 16, 2011 at 1:41 PM, David McNelis
>> <dmcnelis@agentis= energy.com> wrote:
>> > We are currently running a three node cluster where we assign= ed the
>> > initial
>> > tokens using the Python script that is in the Wiki, and we= 9;re currently
>> > using the Random Partitioner, RF=3D1, Cassandra 0.8 from the = Riptano RPM
>> > ....however we're seeing one node taken on over 60% of th= e data as we
>> > load
>> > data.
>> > Our keys are sequential, and can range from 0 to 2^64, though= in
>> > practice
>> > we're between 1 and 2,000,000,000, with the current =A0ma= x around 50,000.
>> > =A0 In
>> > order to balance out the =A0load would we be best served chan= ging our
>> > tokens
>> > to make the top and bottom 1/3rd of the node go to the previo= us and next
>> > nodes respectively, then running nodetool move?
>> > Even if we do that, it would seem that we'd likely contin= ue to run into
>> > this
>> > sort of issue as =A0we =A0add =A0additionally=A0data... would= we be better
>> > served
>> > with a different Partitioner strategy? =A0Or will we need to = very actively
>> > manage our tokens to avoid getting into an unbalanced situati= on?
>> >
>> > --
>> > David McNelis
>> > Lead Software Engineer
>> > Agentis Energy
>> > ww= w.agentisenergy.com
>> > o: 630.35= 9.6395
>> > c: 219.38= 4.5143
>> > A Smart Grid technology company focused on helping consumers = of energy
>> > control an often under-managed resource.
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra supp= ort
>> http://www.d= atastax.com
>
>
>
> --
> David McNelis
> Lead Software Engineer
> Agentis Energy
> www.agentis= energy.com
> o: 630.359.6395
> c: 219.384.5143
> A Smart Grid technology company focused on helping consumers of energy=
> control an often under-managed resource.
>
>



--



--
= David McNelis
Lead Software E= ngineer
Agentis Energy<= /font>
o: 630.359.6395
c: 219.384.5143<= /span>

A Smart Grid technology company focused on helping consumers of = energy control an often under-managed resource.


--bcaec548a4fbe9a71e04aab24180--