Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6246E9BC6 for ; Thu, 27 Sep 2012 14:46:08 +0000 (UTC) Received: (qmail 59741 invoked by uid 500); 27 Sep 2012 14:46:05 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 59707 invoked by uid 500); 27 Sep 2012 14:46:05 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 59699 invoked by uid 99); 27 Sep 2012 14:46:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Sep 2012 14:46:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mvallebr@gmail.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bk0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Sep 2012 14:45:59 +0000 Received: by bkcjc3 with SMTP id jc3so1972636bkc.31 for ; Thu, 27 Sep 2012 07:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=azGNsPIMCotX0Fwn5jSC89PlTYS8R1Wg+otVJk9mHso=; b=J//NnnxsKHpNXRhSzE+TT2sKmeCNqlgxI/YeL5F4GsE+sHfwnZXUEoVwayFHUa6nG7 DI2zrESlB4mp7/0yg7pE7U7vPXuV2ZzRCHwlUhfwU1fA+QjJA198rKYWss+vCdZxx8Z4 KS4OsrsR9vIbBrZcaUQAWKUcOuVKqh/Bg62bGqQ+GOXN45uY+1WxaJa1/I1b4ncYzFC1 noI+WkHnlvI/oXFf6rrHatvfXAftcZtjJF6WIKV2g9MY10aets5322L6Fmm0sKpYuCkA SS4CTptnlYoj4cvPF6caGd8fYyjw0jwtJaaihwUnUgtt0N/29R5wvi4+0hXErzBX121o q/Lg== MIME-Version: 1.0 Received: by 10.152.112.233 with SMTP id it9mr3419272lab.40.1348757139117; Thu, 27 Sep 2012 07:45:39 -0700 (PDT) Received: by 10.112.21.132 with HTTP; Thu, 27 Sep 2012 07:45:39 -0700 (PDT) In-Reply-To: References: Date: Thu, 27 Sep 2012 11:45:39 -0300 Message-ID: Subject: Re: 1000's of column families From: Marcelo Elias Del Valle To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d040838d32a546804caaffe26 --f46d040838d32a546804caaffe26 Content-Type: text/plain; charset=ISO-8859-1 Dean, I was used, in the relational world, to use hibernate and O/R mapping. There were times when I used 3 classes (2 inheriting from 1 another) and mapped all of the to 1 table. The common part was in the super class and each sub class had it's own columns. The table, however, use to have all the columns and this design was hard because of that, as creating more subclasses would need changes in the table. However, if you use playOrm and if playOrm has/had a feature to allow inheritance mapping to a CF, it would solve your problem, wouldn't it? Of course it is probably much harder than it might problably appear... :D Best regards, Marcelo Valle. 2012/9/27 Hiller, Dean > We have 1000's of different building devices and we stream data from these > devices. The format and data from each one varies so one device has > temperature at timeX with some other variables, another device has CO2 > percentage and other variables. Every device is unique and streams it's > own data. We dynamically discover devices and register them. Basically, > one CF or table per thing really makes sense in this environment. While we > could try to find out which devices "are" similar, this would really be a > pain and some devices add some new variable into the equation. NOT only > that but researchers can register new datasets and upload them as well and > each dataset they have they do NOT want to share with other researches > necessarily so we have security groups and each CF belongs to security > groups. We dynamically create CF's on the fly as people register new > datasets. > > On top of that, when the data sets get too large, we probably want to > partition a single CF into time partitions. We could create one CF and put > all the data and have a partition per device, but then a time partition > will contain "multiple" devices of data meaning we need to shrink our time > partition size where if we have CF per device, the time partition can be > larger as it is only for that one device. > > THEN, on top of that, we have a meta CF for these devices so some people > want to query for streams that match criteria AND which returns a CF name > and they query that CF name so we almost need a query with variables like > select cfName from Meta where x = y and then select * from cfName where > xxxxx. Which we can do today. > > Dean > > From: Marcelo Elias Del Valle mvallebr@gmail.com>> > Reply-To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Date: Thursday, September 27, 2012 8:01 AM > To: "user@cassandra.apache.org" < > user@cassandra.apache.org> > Subject: Re: 1000's of column families > > Out of curiosity, is it really necessary to have that amount of CFs? > I am probably still used to relational databases, where you would use a > new table just in case you need to store different kinds of data. As > Cassandra stores anything in each CF, it might probably make sense to have > a lot of CFs to store your data... > But why wouldn't you use a single CF with partitions in these case? > Wouldn't it be the same thing? I am asking because I might learn a new > modeling technique with the answer. > > []s > > 2012/9/26 Hiller, Dean > > We are streaming data with 1 stream per 1 CF and we have 1000's of CF. > When using the tools they are all geared to analyzing ONE column family at > a time :(. If I remember correctly, Cassandra supports as many CF's as you > want, correct? Even though I am going to have tons of funs with > limitations on the tools, correct? > > (I may end up wrapping the node tool with my own aggregate calls if needed > to sum up multiple column families and such). > > Thanks, > Dean > > > > -- > Marcelo Elias Del Valle > http://mvalle.com - @mvallebr > -- Marcelo Elias Del Valle http://mvalle.com - @mvallebr --f46d040838d32a546804caaffe26 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Dean,=A0

=A0 =A0 =A0I was used, in the relational world,= to use hibernate and O/R mapping. There were times when I used 3 classes (= 2 inheriting from 1 another) and mapped all of the to 1 table. The common p= art was in the super class and each sub class had it's own columns. The= table, however, use to have all the columns and this design was hard becau= se of that, as creating more subclasses would need changes in the table.
=A0 =A0 =A0However, if you use playOrm and if playOrm has/had a featur= e to allow inheritance mapping to a CF, it would solve your problem, wouldn= 't it? Of course it is probably much harder than it might problably app= ear... :D

Best regards,
Marcelo Valle.

2012/9/27 Hiller, Dean <Dean.Hiller@nrel.gov><= /span>
We have 1000's of different building dev= ices and we stream data from these devices. =A0The format and data from eac= h one varies so one device has temperature at timeX with some other variabl= es, another device has CO2 percentage and other variables. =A0Every device = is unique and streams it's own data. =A0We dynamically discover devices= and register them. =A0Basically, one CF or table per thing really makes se= nse in this environment. =A0While we could try to find out which devices &q= uot;are" similar, this would really be a pain and some devices add som= e new variable into the equation. =A0NOT only that but researchers can regi= ster new datasets and upload them as well and each dataset they have they d= o NOT want to share with other researches necessarily so we have security g= roups and each CF belongs to security groups. =A0We dynamically create CF&#= 39;s on the fly as people register new datasets.

On top of that, when the data sets get too large, we probably want to parti= tion a single CF into time partitions. =A0We could create one CF and put al= l the data and have a partition per device, but then a time partition will = contain "multiple" devices of data meaning we need to shrink our = time partition size where if we have CF per device, the time partition can = be larger as it is only for that one device.

THEN, on top of that, we have a meta CF for these devices so some people wa= nt to query for streams that match criteria AND which returns a CF name and= they query that CF name so we almost need a query with variables like sele= ct cfName from Meta where x =3D y and then select * from cfName where xxxxx= . Which we can do today.

Dean

From: Marcelo Elias Del Valle <mva= llebr@gmail.com<mailto:mvalleb= r@gmail.com>>
Reply-To: "user@cassandra= .apache.org<mailto:user= @cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Thursday, September 27, 2012 8:01 AM
To: "user@cassandra.apach= e.org<mailto:user@cassa= ndra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: 1000's of column families

Out of curiosity, is it really necessary to have that amount of CFs?
I am probably still used to relational databases, where you would use a new= table just in case you need to store different kinds of data. As Cassandra= stores anything in each CF, it might probably make sense to have a lot of = CFs to store your data...
But why wouldn't you use a single CF with partitions in these case? Wou= ldn't it be the same thing? I am asking because I might learn a new mod= eling technique with the answer.

[]s

2012/9/26 Hiller, Dean <De= an.Hiller@nrel.gov<mailto:De= an.Hiller@nrel.gov>>
We are streaming data with 1 stream= per 1 CF and we have 1000's of CF. =A0When using the tools they are al= l geared to analyzing ONE column family at a time :(. =A0If I remember corr= ectly, Cassandra supports as many CF's as you want, correct? =A0Even th= ough I am going to have tons of funs with limitations on the tools, correct= ?

(I may end up wrapping the node tool with my own aggregate calls if needed = to sum up multiple column families and such).

Thanks,
Dean



--
Marcelo Elias Del Valle
http://mvalle.com - @mv= allebr



--
= Marcelo Elias Del Valle
= http://mvalle.com=A0- @mvallebr
--f46d040838d32a546804caaffe26--