Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of arodrime@gmail.com designates
 209.85.215.172 as permitted sender)
MIME-Version: 1.0
From: Alain RODRIGUEZ <arodrime@gmail.com>
Date: Fri, 4 Nov 2011 18:07:19 +0100
Message-ID: 
 <CA+VSrLp3F_TfY_8YJ3jCf4msr17YWoceJRBg9Fx-VaP8KmPFwQ@mail.gmail.com>
Subject: Modeling big data to allow filtering with a lot of distinct
 combinations of dimesions, in real time and with no latency
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=0015174c1a302797ca04b0ebbed3

--0015174c1a302797ca04b0ebbed3
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I started this thread in the phpCassa google group, but I thinks its place
is here.

There is my first post :

"I was wondering about a specific point of Cassandra Modeling.

If I need to know the number of connexion to my website using each browser,
every hour, I can do:

Row key: $browser, column key: date('YmdH', $timestamp), value: counter.

I can increment this counter for any visit, this should work. The point is
that I want to be able to render the results of a lot of statistics used as
filters.

I mean, I will have information such as browser, browser version, screen
resolution, OS, OS version, localization... And I want to allow users to
get data (number of views) filtering it as much as they want.

For example, if I want to know how many people visited my website with
safari, windos, and from New York, every hour, I can store:

Row key : $browser:$os:$localization, column key : date('YmdH',
$timestamp), value : counter.

This can't be the best solution because according to the combinational
mathematics I will have to store n! counters to be able to store data with
all filters. If I got 10 filters I will increment 3 628 800 counters.

That's not the good solution, for sure. How am I supposed to model this to
be able to read data with any filter I want ?

Thanks,

Alain"


And there is the first answer given (thanks to Tyler Hobbs) :

"Technically, the number of potential different counters would be the
cardinality of each field multiplied together.  (Since one of the fields
holds a time, this number would continue to grow.) However, in practice
you'll have far fewer than this number of counters, because not every
possible combination of these will happen.

>That's not the good solution, for sure. How am I supposed to model

> this to be able to read data with any filter I want ?

It's a reasonable solution if you want to be able to drill down and filter
by any attribute.  If you want to be able to filter based on all of these
attributes, you have to store that information about every request in one
way or another."


I know it's a non-trivial problem, but I'm sure that some people already
faced this problem before I do.

I'll allow user to filter however they want, chosing dimensions with
checkboxes. They will be able to combine dimensions and ask for any
combination.

So, with this solution, I will have to store every event n times, with n =
number of possible combinations.

I saw this yesterday : http://t.co/EXL6yAO8 (thanks to Dave Gardner). This
company seems to something equivalent of the idea exposed in my first
post....

Any experience to share with this kind of problem ?

thank you,

Alain

--0015174c1a302797ca04b0ebbed3
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div><span class=3D"Apple-style-span" style=3D"background-color: rgb(255, 2=
55, 255); "><div>Hi all,</div><div><br></div><div>I started this thread in =
the phpCassa google group, but I thinks its place is here.</div><div><br></=
div>

<div>There is my first post :</div><div><br></div><div>&quot;I was wonderin=
g about a specific point of Cassandra Modeling.=A0</div><div><br></div><div=
>If I need to know the number of connexion to my website using each browser=
, every hour, I can do:=A0</div>

<div><br></div><div>Row key: $browser, column key: date(&#39;YmdH&#39;, $ti=
mestamp), value: counter.=A0</div><div><br></div><div>I can increment this =
counter for any visit, this should work. The point is that I want to be abl=
e to render the results of a lot of statistics used as filters.=A0</div>

<div><br></div><div>I mean, I will have information such as browser, browse=
r version, screen resolution, OS, OS version, localization... And I want to=
 allow users to get data (number of views) filtering it as much as they wan=
t.</div>

<div><br></div><div>For example, if I want to know how many people visited =
my website with safari, windos, and from New York, every hour, I can store:=
=A0</div><div><br></div><div>Row key : $browser:$os:$localization, column k=
ey : date(&#39;YmdH&#39;, $timestamp), value : counter.=A0</div>

<div><br></div><div>This can&#39;t be the best solution because according t=
o the combinational mathematics I will have to store n! counters to be able=
 to store data with all filters. If I got 10 filters I will increment 3 628=
 800 counters.=A0</div>

<div><br></div><div>That&#39;s not the good solution, for sure. How am I su=
pposed to model this to be able to read data with any filter I want ? =A0</=
div><div><br></div><div>Thanks,=A0</div><div><br></div><div>Alain&quot;</di=
v>

<div><br></div><div><br></div><div><br></div><div>And there is the first an=
swer given (thanks to Tyler Hobbs) :</div><div><br></div><div>&quot;Technic=
ally, the number of potential different counters would be the cardinality o=
f each field multiplied together. =A0(Since one of the fields holds a time,=
 this number would continue to grow.) However, in practice you&#39;ll have =
far fewer than this number of counters, because not every possible combinat=
ion of these will happen.=A0</div>

<div><br></div><div>&gt;That&#39;s not the good solution, for sure. How am =
I supposed to model=A0</div><div><br></div><div>&gt; this to be able to rea=
d data with any filter I want ?=A0</div><div><br></div><div>It&#39;s a reas=
onable solution if you want to be able to drill down and filter by any attr=
ibute. =A0If you want to be able to filter based on all of these attributes=
, you have to store that information about every request in one way or anot=
her.&quot;</div>

<div><br></div><div><br></div><div><br></div><div>I know it&#39;s a non-tri=
vial problem, but I&#39;m sure that some people already faced this problem =
before I do.</div><div><br></div><div>I&#39;ll allow user to filter however=
 they want, chosing dimensions with checkboxes. They will be able to combin=
e dimensions and ask for any combination.</div>

<div><br></div><div>So, with this solution, I will have to store every even=
t n times, with n =3D number of possible combinations.=A0</div><div><br></d=
iv><div>I saw this yesterday : <a href=3D"http://t.co/EXL6yAO8">http://t.co=
/EXL6yAO8</a> (thanks to Dave Gardner). This company seems to something equ=
ivalent of the idea exposed in my first post....</div>

<div><br></div><div>Any experience to share with this kind of problem ?</di=
v><div><br></div><div>thank you,</div><div><br></div><div>Alain</div><div s=
tyle=3D"font-family: arial, sans-serif; font-size: 12px; "><br></div></span=
></div>


--0015174c1a302797ca04b0ebbed3--