Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 83C709249 for ; Thu, 16 Feb 2012 23:41:47 +0000 (UTC) Received: (qmail 73200 invoked by uid 500); 16 Feb 2012 23:41:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 73162 invoked by uid 500); 16 Feb 2012 23:41:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73120 invoked by uid 99); 16 Feb 2012 23:41:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Feb 2012 23:41:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of database.craftsman@gmail.com designates 209.85.160.172 as permitted sender) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Feb 2012 23:41:39 +0000 Received: by ghbg16 with SMTP id g16so1826058ghb.31 for ; Thu, 16 Feb 2012 15:41:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=zQVHCor3XfTP1ZQVE9SPisjijQIQ+Ntk62zrecnYFk0=; b=NUziGv7zeJmSLWVdwqXP2UGv3C3MtzmSpwa1KGfgKOiJks9xHTrndXbpOYjRdb5cB8 curRzTC7aR11D+mbTuyl4n/ahukfd1d/0BPaFv6PrNq6Urf5EqDjgFnTPlBwKPPewkx8 By/DroodBKbWG3gUttXSNdtr+UXKyPezYxkiQ= MIME-Version: 1.0 Received: by 10.236.46.193 with SMTP id r41mr6457101yhb.123.1329435678583; Thu, 16 Feb 2012 15:41:18 -0800 (PST) Received: by 10.146.50.14 with HTTP; Thu, 16 Feb 2012 15:41:18 -0800 (PST) In-Reply-To: <0C9711BA-7306-416C-A092-F701152E7D9A@thelastpickle.com> References: <0C9711BA-7306-416C-A092-F701152E7D9A@thelastpickle.com> Date: Thu, 16 Feb 2012 15:41:18 -0800 Message-ID: Subject: Re: Wide row column slicing - row size shard limit From: Data Craftsman To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf303bfad65fad5904b91d5dea --20cf303bfad65fad5904b91d5dea Content-Type: text/plain; charset=ISO-8859-1 Hi Aaron Morton and R. Verlangen, Thanks for the quick answer. It's good to know Thrift's limit on the amount of data it will accept / send. I know the hard limit is 2 billion columns per row. My question is at what size it will slowdown read/write performance and maintenance. The blog I reference said the row size should be less than 10MB. It'll be better if Cassandra can transparently shard/split the wide row and then distribute them to many nodes, to help the load balancing. Are there any other ways to model historical data (or time-series-data) besides wide row column slicing in Cassandra? Thanks, Charlie | Data Solution Architect Developer http://mujiang.blogspot.com On Thu, Feb 16, 2012 at 12:38 AM, aaron morton wrote: > > Based on this blog of Basic Time Series with Cassandra data modeling, > > http://rubyscale.com/blog/2011/03/06/basic-time-series-with-cassandra/ > I've not read that one but it sounds right. Mat Dennis knows his stuff > http://www.slideshare.net/mattdennis/cassandra-nyc-2011-data-modeling > > > There is a limit on how big the row size can be before slowing down the > update and query performance, that is 10MB or less. > There is no hard limit. Wide rows wont upset writes too much. Some read > queries can avoid problems but most will not. > > Wide rows are a pain when it comes to maintenance. They take longer to > compact and repair. > > > Is this still true in Cassandra latest version? or in what release > Cassandra will remove this limit? > There is a limit of 2 billion columns per row. There is a not a limit of > 10MB per row. I've seen some rows in the 100's of MB and they are always a > pain. > > > Manually sharding the wide row will increase the application complexity, > it would be better if Cassandra can handle it transparently. > it's not that hard :) > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 16/02/2012, at 7:40 AM, Data Craftsman wrote: > > > Hello experts, > > > > Based on this blog of Basic Time Series with Cassandra data modeling, > > http://rubyscale.com/blog/2011/03/06/basic-time-series-with-cassandra/ > > > > "This (wide row column slicing) works well enough for a while, but over > time, this row will get very large. If you are storing sensor data that > updates hundreds of times per second, that row will quickly become gigantic > and unusable. The answer to that is to shard the data up in some way" > > > > There is a limit on how big the row size can be before slowing down the > update and query performance, that is 10MB or less. > > > > Is this still true in Cassandra latest version? or in what release > Cassandra will remove this limit? > > > > Manually sharding the wide row will increase the application complexity, > it would be better if Cassandra can handle it transparently. > > > > Thanks, > > Charlie | DBA & Developer > > > > > > p.s. Quora link, > > > http://www.quora.com/Cassandra-database/What-are-good-ways-to-design-data-model-in-Cassandra-for-historical-data > > > > > > > > --20cf303bfad65fad5904b91d5dea Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Aaron Morton and=A0=A0=A0R. Verlangen,

Th= anks for the quick answer. It's good to know=A0Thrift's= limit on the amount of data it will accept / send.

I know the hard limit is=A02 billion columns per row. My= question is at what size it will slowdown read/write performance and=A0mai= ntenance. =A0The blog I reference said the row size should be less than 10M= B.

It'll be better if Cassandra can transparently shar= d/split the wide row and then=A0distribute=A0them to many nodes, to help th= e=A0load balancing.

Are there a= ny other ways to model historical data (or=A0time-series-data)=A0besides wi= de row column slicing in Cassandra?

Thanks,
Charlie=A0| Data Solution Architect Developer



On Thu, Feb 16, 2012 at 12:38 AM, a= aron morton <aaron@thelastpickle.com> wrote:
> Based on this blog of= Basic Time Series with Cassandra data modeling,
> http://rubyscale.com/blog/2011/03/06/basic-t= ime-series-with-cassandra/
I've not read that one but it sounds right. Mat Dennis knows his = stuff =A0http://www.slideshare.net/mattdennis/cass= andra-nyc-2011-data-modeling

> There is a limit on how big the row size can be before slowing down th= e update and query performance, that is 10MB or less.
There is no hard limit. Wide rows wont upset writes too much. Some re= ad queries can avoid problems but most will not.

Wide rows are a pain when it comes to maintenance. =A0They take longer to c= ompact and repair.

> Is this still true in Cassandra latest version? or in what release Cas= sandra will remove this limit?
There is a limit of 2 billion columns per row. There is a not a limit= of 10MB per row. I've seen some rows in the 100's of MB and they a= re always a pain.

> Manually sharding the wide row will increase the application complexit= y, it would be better if Cassandra can handle it transparently.
it's not that hard :)

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thela= stpickle.com

On 16/02/2012, at 7:40 AM, Data Craftsman wrote:

> Hello experts,
>
> Based on this blog of Basic Time Series with Cassandra data modeling,<= br> > http://rubyscale.com/blog/2011/03/06/basic-t= ime-series-with-cassandra/
>
> "This (wide row column slicing) works well enough for a while, bu= t over time, this row will get very large. If you are storing sensor data t= hat updates hundreds of times per second, that row will quickly become giga= ntic and unusable. The answer to that is to shard the data up in some way&q= uot;
>
> There is a limit on how big the row size can be before slowing down th= e update and query performance, that is 10MB or less.
>
> Is this still true in Cassandra latest version? or in what release Cas= sandra will remove this limit?
>
> Manually sharding the wide row will increase the application complexit= y, it would be better if Cassandra can handle it transparently.
>
> Thanks,
> Charlie | DBA & Developer
>
>
> p.s. Quora link,
> ht= tp://www.quora.com/Cassandra-database/What-are-good-ways-to-design-data-mod= el-in-Cassandra-for-historical-data
>
>
>


--20cf303bfad65fad5904b91d5dea--