Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of synfinatic@gmail.com designates
 209.85.214.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CC89B8E9.11F8D%Dean.Hiller@nrel.gov>
References: 
 <CABKQidt+BEK_T-CYD7bUXF3GTgoLtQDZ1sgTRnKJUppK3CKEBA@mail.gmail.com>
 <CC89B8E9.11F8D%Dean.Hiller@nrel.gov>
From: Aaron Turner <synfinatic@gmail.com>
Date: Thu, 27 Sep 2012 18:10:53 +0100
Message-ID: 
 <CANAZdzWkDCjuLXGaxAndiKs7t_4v6Tu1Sv=vOVcYuNdT8sDB6g@mail.gmail.com>
Subject: Re: 1000's of column families
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Sep 27, 2012 at 3:11 PM, Hiller, Dean <Dean.Hiller@nrel.gov> wrote:
> We have 1000's of different building devices and we stream data from thes=
e devices.  The format and data from each one varies so one device has temp=
erature at timeX with some other variables, another device has CO2 percenta=
ge and other variables.  Every device is unique and streams it's own data. =
 We dynamically discover devices and register them.  Basically, one CF or t=
able per thing really makes sense in this environment.  While we could try =
to find out which devices "are" similar, this would really be a pain and so=
me devices add some new variable into the equation.  NOT only that but rese=
archers can register new datasets and upload them as well and each dataset =
they have they do NOT want to share with other researches necessarily so we=
 have security groups and each CF belongs to security groups.  We dynamical=
ly create CF's on the fly as people register new datasets.
>
> On top of that, when the data sets get too large, we probably want to par=
tition a single CF into time partitions.  We could create one CF and put al=
l the data and have a partition per device, but then a time partition will =
contain "multiple" devices of data meaning we need to shrink our time parti=
tion size where if we have CF per device, the time partition can be larger =
as it is only for that one device.
>
> THEN, on top of that, we have a meta CF for these devices so some people =
want to query for streams that match criteria AND which returns a CF name a=
nd they query that CF name so we almost need a query with variables like se=
lect cfName from Meta where x =3D y and then select * from cfName where xxx=
xx. Which we can do today.

How strict are your security requirements?  If it wasn't for that,
you'd be much better off storing data on a per-statistic basis then
per-device.  Hell, you could store everything in a single CF by using
a composite row key:

<devicename>|<stat type>|<instance>

But yeah, there isn't a hard limit for the number of CF's, but there
is overhead associated with each one and so I wouldn't consider your
design as scalable.  Generally speaking, hundreds are ok, but
thousands is pushing it.


--=20
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Win=
dows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin
"carpe diem quam minimum credula postero"