Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 29F5511BC8 for ; Sat, 7 Jun 2014 17:42:09 +0000 (UTC) Received: (qmail 94609 invoked by uid 500); 7 Jun 2014 17:42:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94570 invoked by uid 500); 7 Jun 2014 17:42:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94561 invoked by uid 99); 7 Jun 2014 17:42:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Jun 2014 17:42:05 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.192.46] (HELO mail-qg0-f46.google.com) (209.85.192.46) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Jun 2014 17:42:03 +0000 Received: by mail-qg0-f46.google.com with SMTP id q108so6788841qgd.5 for ; Sat, 07 Jun 2014 10:41:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:from:in-reply-to:mime-version:date :message-id:subject:to:content-type; bh=Ge8wChgsqS/8zcSuXek+eUy9NmXF9JyE6xYKBtQCgQQ=; b=aIf+iWPvgFjcHs/h9o5wRxKswTZZ5HnxwhLQfky1Fk/Mwmyt8CeRdTPtoJQft+Gjj3 WCOmQoM+OGBNLcOvZQKbgzvMr5I2NcKCUMC12ImS4Y/siBjDlZVKSHwvJd+SxktjtZ/V rn0VVWWh1VXhtrs3/8/cb9qiuO9p+5vqyoF+JBMHVkmsDoPQKFjqQyNejJVxysYsc+7j JJo7IXfDd/Ew0nlL8Tp5anwc5Q1ptEb0SBoKEwELQTatgsdWOWF0RYlzh521+8ZshO2E L5UstwRThxXheJDV+wnCmqTXy68dXaUNpiBVTxHLHW2b7r/FHQ5gi0rq7jv44cm/2gba 0w6w== X-Gm-Message-State: ALoCoQkPodGG/lDFV7HTZP3W4VZII16jCsWWJftQ7aeDBvyWDgJG4perQx/dCpY32ovd8iWlY7Sp X-Received: by 10.140.100.198 with SMTP id s64mr18397802qge.106.1402162898851; Sat, 07 Jun 2014 10:41:38 -0700 (PDT) References: <81A38010-3844-47F7-8DF6-5F069D12AD7C@gmail.com> From: Colin Clark In-Reply-To: Mime-Version: 1.0 (1.0) Date: Sat, 7 Jun 2014 12:41:38 -0500 Message-ID: <-6327655872995295763@unknownmsgid> Subject: Re: Data model for streaming a large table in real time. To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a11c16ba880f64304fb427d50 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c16ba880f64304fb427d50 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable It's an anti-pattern and there are better ways to do this. I have implemented the paging algorithm you've described using wide rows and bucketing. This approach is a more efficient utilization of Cassandra's built in wholesome goodness. Also, I wouldn't let any number of clients (huge) connect directly the cluster to do this-put some type of app server in between to handle the comm's and fan out. You'll get better utilization of resources and less overhead in addition to flexibility of which data center you're utilizing to serve requests. -- Colin 320-221-9531 On Jun 7, 2014, at 12:28 PM, Kevin Burton wrote: I just checked the source and in 2.1.0 it's not deprecated. So it *might* be *being* deprecated but I haven't seen anything stating that. On Sat, Jun 7, 2014 at 8:03 AM, Colin wrote: > I believe Byteorderedpartitioner is being deprecated and for good reason. > I would look at what you could achieve by using wide rows and > murmur3partitioner. > > > > -- > Colin > 320-221-9531 > > > On Jun 6, 2014, at 5:27 PM, Kevin Burton wrote: > > We have the requirement to have clients read from our tables while they'r= e > being written. > > Basically, any write that we make to cassandra needs to be sent out over > the Internet to our customers. > > We also need them to resume so if they go offline, they can just pick up > where they left off. > > They need to do this in parallel, so if we have 20 cassandra nodes, they > can have 20 readers each efficiently (and without coordination) reading > from our tables. > > Here's how we're planning on doing it. > > We're going to use the ByteOrderedPartitioner . > > I'm writing with a primary key of the timestamp, however, in practice, > this would yield hotspots. > > (I'm also aware that time isn't a very good pk in a distribute system as = I > can easily have a collision so we're going to use a scheme similar to a > uuid to make it unique per writer). > > One node would take all the load, followed by the next node, etc. > > So my plan to stop this is to prefix a slice ID to the timestamp. This > way each piece of content has a unique ID, but the prefix will place it o= n > a node. > > The slide ID is just a byte=E2=80=A6 so this means there are 255 buckets = in which > I can place data. > > This means I can have clients each start with a slice, and a timestamp, > and page through the data with tokens. > > This way I can have a client reading with 255 threads from 255 regions in > the cluster, in parallel, without any hot spots. > > Thoughts on this strategy? > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > Skype: *burtonator* > blog: http://burtonator.wordpress.com > =E2=80=A6 or check out my Google+ profile > > > War is peace. Freedom is slavery. Ignorance is strength. Corporations are > people. > > --=20 Founder/CEO Spinn3r.com Location: *San Francisco, CA* Skype: *burtonator* blog: http://burtonator.wordpress.com =E2=80=A6 or check out my Google+ profile War is peace. Freedom is slavery. Ignorance is strength. Corporations are people. --001a11c16ba880f64304fb427d50 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
It's an anti-pattern and there= are better ways to do this.

I have implemented th= e paging algorithm you've described using wide rows and bucketing. =C2= =A0This approach is a more efficient utilization of Cassandra's built i= n wholesome goodness.

Also, I wouldn't let any number of clients (huge) c= onnect directly the cluster to do this-put some type of app server in betwe= en to handle the comm's and fan out. =C2=A0You'll get better utiliz= ation of resources and less overhead in addition to flexibility of which da= ta center you're utilizing to serve requests.=C2=A0



--
Colin
320-221-9531
<= br>

On Jun 7, 2014, at 12:28 PM, Kevin Burton <burton@spinn3r.com> wrote:

I just checked the so= urce and in 2.1.0 it's not deprecated. =C2=A0

So it = *might* be *being* deprecated but I haven't seen anything stating that.=


On Sat, Jun 7, 2014 at 8:03 AM, Colin <co= lpclark@gmail.com> wrote:
I believe Byteorderedpartitioner is being deprecated= and for good reason. =C2=A0I would look at what you could achieve by using= wide rows and murmur3partitioner.



--
= Colin


= On Jun 6, 2014, at 5:27 PM, Kevin Burton <burton@spinn3r.com> wrote:

We have the requi= rement to have clients read from our tables while they're being written= .

Basically, any write that we make to cassandra needs t= o be sent out over the Internet to our customers.

We also need them to resume so if they go offline, they can j= ust pick up where they left off.

They need to do t= his in parallel, so if we have 20 cassandra nodes, they can have 20 readers= each efficiently (and without coordination) reading from our tables.

Here's how we're planning on doing it.

We're going to use the ByteOrderedPartitioner .
<= div>
I'm writing with a primary key of the timestamp, how= ever, in practice, this would yield hotspots.

(I'm also aware that time isn't a very good pk = in a distribute system as I can easily have a collision so we're going = to use a scheme similar to a uuid to make it unique per writer).

One node would take all the load, followed by the next node,= etc.

So my plan to stop this is to prefix a slice= ID to the timestamp. =C2=A0This way each piece of content has a unique ID,= but the prefix will place it on a node.

The slide ID is just a byte=E2=80=A6 so this means ther= e are 255 buckets in which I can place data. =C2=A0

This means I can have clients each start with a slice, and a timestamp, a= nd page through the data with tokens.

This way I can have a client reading with 255 threads f= rom 255 regions in the cluster, in parallel, without any hot spots.

Thoughts on this strategy? =C2=A0

--

Founder/CEO=C2=A0Spinn3r.com
Location:=C2=A0San F= rancisco, CA
Skype:=C2=A0burtonator
=E2=80=A6 or check out my Google+ profile
War is peace. Fre= edom is slavery. Ignorance is strength. Corporations are people.




--

Founder/CEO=C2=A0Spinn3r.com
Location:=C2=A0San Francisco, CA
Skype:=C2=A0burton= ator
=E2=80=A6 or check out my Google+ profile
War is peace. Fre= edom is slavery. Ignorance is strength. Corporations are people.

--001a11c16ba880f64304fb427d50--