Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 47643 invoked from network); 15 Oct 2010 23:56:52 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 15 Oct 2010 23:56:52 -0000 Received: (qmail 23395 invoked by uid 500); 15 Oct 2010 23:56:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23365 invoked by uid 500); 15 Oct 2010 23:56:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23357 invoked by uid 99); 15 Oct 2010 23:56:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 23:56:51 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 23:56:43 +0000 Received: by fxm17 with SMTP id 17so975922fxm.31 for ; Fri, 15 Oct 2010 16:56:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.40.1 with SMTP id s1mr849372muj.4.1287186982605; Fri, 15 Oct 2010 16:56:22 -0700 (PDT) Received: by 10.103.175.7 with HTTP; Fri, 15 Oct 2010 16:56:22 -0700 (PDT) X-Originating-IP: [64.132.24.248] In-Reply-To: References: Date: Fri, 15 Oct 2010 18:56:22 -0500 Message-ID: Subject: Re: Recommended sort mechanism and partitioner From: Tyler Hobbs To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e65aeb0adb92ad0492b092d3 X-Virus-Checked: Checked by ClamAV on apache.org --0016e65aeb0adb92ad0492b092d3 Content-Type: text/plain; charset=ISO-8859-1 i) Yes ii) Well, so you don't actually want to use version 1 UUIDs for keys here. Although they mostly increase in byte order over time, it's only for the first 8 bytes. Instead, you can use something like: 'timestamp-foo' Where 'foo' might be a randomly generated string or something unique per client. You could also use 'YYYYMMDDSSmmm' instead of the timestamp if that makes queries easier for you. - Tyler On Fri, Oct 15, 2010 at 6:22 PM, Wicked J wrote: > Tyler, > Thanks for answering my question. Can you please clarify on point (c)? > > i] Are you saying that if I move to second row (identified by a rowKey in > Cassandra) after I hit 10 million col. values for 1st row, only then the > second row will be written to a new node in the cluster? meaning all the 10 > million column values within the first row (rowKey) until then have been > written to one and the same node regardless of the # of nodes in the > cluster. > > ii] Assume I change my data model to the one in below (CF1) with a > "OrderPreservingPartitioner" then would I be able to read data in the order > inserted? Because my understanding is TimeUUID values cannot be inserted for > row Keys based on the Thrift API in v0.6.4 i.e. from the insert method in > Cassandra.Client or am I missing something? > > CF1: > > Key: '1' > name: colname, value: 'First Inserted', timestamp: 1287165326492 > Key: '2' > name: colname, value: 'Second Inserted', timestamp: 1287165326523 > > Thanks! > > > On Fri, Oct 15, 2010 at 12:18 PM, Tyler Hobbs wrote: > >> a) 10 mil sounds fine. Just watch out for compaction. Huge rows can kill >> you there, >> from my understanding. >> >> b) Use RandomPartitioner unless you absolutely have to use something else. >> >> c) If you're inserting all along one row and only moving to another row >> when you >> hit 10 mil, you're only going to be writing to one node at a time. In >> this sense, >> you might want to consider using the TimeUUID as a row key instead. >> There's >> not really a problem with having tons of rows in a column family. >> >> If you want to be able to get a slice of time with this scheme, you can >> either use >> an order preserving partitioner or have a second column family with an >> index >> row (or rows) sorted by TimeUUID. (This sounds like what you're >> suggesting.) >> >> - Tyler >> >> >> I wrote some thoughts about this on my blog. I think it's still mostly >>> correct: >>> >>> * http://www.ayogo.com/techblog/2010/04/sorting-in-cassandra/ >>> >>> On Fri, Oct 15, 2010 at 11:14 AM, Wicked J >>> wrote: >>> > Hi, >>> > I'm using TimeUUID/Sort by column name mechanism. The column value can >>> > contain text data (in future they may contain image data as well) >>> leading to >>> > the possibility of a row out-growing the RAM capacity. Given this >>> background >>> > my questions are: >>> > >>> > a] How many columns are recommended against one row? Based on my app. >>> needs, >>> > I can imagine having 10 million would be a good starting point for the >>> > max_limit (based on text data). Also note that my app. will use search >>> in >>> > ranges of 100 or 200 columns when there are large number of >>> records(columnar >>> > data) without a caching solution in the front. >>> > b] What partitioner is recommended? so that the load in the cluster >>> nodes is >>> > not largely uneven. >>> > c] Would you recommend changing the TimeUUID/Columnar sort mechanism >>> (with a >>> > change in the data model) to sort using row key mechanism? If so then >>> what >>> > partitioner is recommended? with load not being largely uneven. >>> > >>> > Thanks >>> > >>> >> >> > --0016e65aeb0adb92ad0492b092d3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable i) Yes

ii) Well, so you don't actually want to use version 1 UUI= Ds for keys here. Although
they mostly increase in byte order over time,= it's only for the first 8 bytes.=A0 Instead,
you can use something = like:

'timestamp-foo'

Where 'foo' might be a randomly = generated string or something unique per client.

You could also use = 'YYYYMMDDSSmmm' instead of the timestamp if that makes
queries e= asier for you.

- Tyler

On Fri, Oct 15, 2010 at 6:22 = PM, Wicked J <wickedj2010@gmail.com> wrote:
Tyler,
Thanks for answering my question. Can you please clarify on point= (c)?

i] Are you saying that if I move to second row (identified by = a rowKey in Cassandra) after I hit 10 million=A0 col. values for 1st row, o= nly then the second row will be written to a new node in the cluster?=A0 me= aning all the 10 million column values within the first row (rowKey) until = then have been written to one and the same node regardless of the # of node= s in the cluster.

ii] Assume I change my data model to the one in below (CF1) with a &quo= t;OrderPreservingPartitioner" then would I be able to read data in the= order inserted? Because my understanding is TimeUUID values cannot be inse= rted for row Keys based on the Thrift API in v0.6.4 i.e. from the insert me= thod in Cassandra.Client or am I missing something?

CF1:

Key: '1'
=A0 name: colname, value: 'First In= serted', timestamp: 1287165326492
Key: '2'
=A0 name: coln= ame, value: 'Second Inserted', timestamp: 1287165326523

Than= ks!


On Fri, Oct 15, 2010 at 12:18 PM, Tyler Hobb= s <tyler@riptano.com> wrote:
a) 10 mil sounds fine.=A0 Just watch out for compaction. Huge rows can kill= you there,
from my understanding.

b) Use RandomPartitioner unles= s you absolutely have to use something else.

c) If you're insert= ing all along one row and only moving to another row when you
hit 10 mil, you're only going to be writing to one node at a time.=A0 I= n this sense,
you might want to consider using the TimeUUID as a row key= instead.=A0 There's
not really a problem with having tons of rows i= n a column family.

If you want to be able to get a slice of time with this scheme, you can= either use
an order preserving partitioner or have a second column fami= ly with an index
row (or rows) sorted by TimeUUID. (This sounds like wha= t you're suggesting.)

- Tyler


<= blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border= -left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">I wrote some thoug= hts about this on my blog. I think it's still mostly correct:

=A0* http://www.ayogo.com/techblog/2010/04/sorting-in-cassan= dra/

On Fri, Oct 15, 2010 at 11:14 AM, Wicked J <wickedj2010@gmail.com> wrote:
> Hi,
> I'm using TimeUUID/Sort by column name mechanism. The column value= can
> contain text data (in future they may contain image data as well) lead= ing to
> the possibility of a row out-growing the RAM capacity. Given this back= ground
> my questions are:
>
> a] How many columns are recommended against one row? Based on my app. = needs,
> I can imagine having 10 million would be a good starting point for the=
> max_limit (based on text data). Also note that my app. will use search= in
> ranges of 100 or 200 columns when there are large number of records(co= lumnar
> data) without a caching solution in the front.
> b] What partitioner is recommended? so that the load in the cluster no= des is
> not largely uneven.
> c] Would you recommend changing the TimeUUID/Columnar sort mechanism (= with a
> change in the data model) to sort using row key mechanism? If so then = what
> partitioner is recommended?=A0 with load not being largely uneven.
>
> Thanks
>



--0016e65aeb0adb92ad0492b092d3--