Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <5654366A.1060204@edgelab.ch>
References: <5654366A.1060204@edgelab.ch>
Date: Tue, 24 Nov 2015 07:49:02 -0500
Message-ID: 
 <CAOxAL619H0HcBth-tzEy3vE7TzJ3R=dgsFZiOBLDv92V6jkwqQ@mail.gmail.com>
Subject: Re: Many keyspaces pattern
From: Jack Krupansky <jack.krupansky@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a1143637823d21d052548c4da

--001a1143637823d21d052548c4da
Content-Type: text/plain; charset=UTF-8

How often is sometimes - closer to 20% of the batches or 2%?

How are you querying batches, both current and older ones?

As always, your queries should drive your data models.

If deleting a batch is very infrequent, maybe best to not do it and simply
have logic in the app to ignore deleted batches - if your queries would
reference them at all.

What reasons would you have to delete a batch? Depending on the nature of
the reason there may be an alternative.

Make sure your cluster is adequately provisioned so that these expensive
operations can occur in parallel to reduce their time and resources per
node.

Do all batches eventually get aged and deleted or are you expecting that
most batches will live for many years to come? Have you planned for how you
will grow the cluster over time?

Maybe bite the bullet and use a background process to delete a batch if
deletion is competing too heavily with query access - if they really need
to be deleted at all.

Number of keyspaces - and/or tables - should be limited to "low hundreds",
and even then you are limited by RAM and CPU of each node. If a keyspace
has 14 tables, then 250/14 = 20 would be a recommended upper limit for
number of key spaces. Even if your total number of tables was under 300 or
even 200, you would need to do a proof of concept implementation to verify
that your specific data works well on your specific hardware.


-- Jack Krupansky

On Tue, Nov 24, 2015 at 5:05 AM, Jonathan Ballet <jballet@edgelab.ch> wrote:

> Hi,
>
> we are running an application which produces every night a batch with
> several hundreds of Gigabytes of data. Once a batch has been computed, it
> is never modified (nor updates nor deletes), we just keep producing new
> batches every day.
>
> Now, we are *sometimes* interested to remove a complete specific batch
> altogether. At the moment, we are accumulating all these data into only one
> keyspace which has a batch ID column in all our tables which is also part
> of the primary key. A sample table looks similar to this:
>
>   CREATE TABLE computation_results (
>       batch_id int,
>       id1 int,
>       id2 int,
>       value double,
>       PRIMARY KEY ((batch_id, id1), id2)
>   ) WITH CLUSTERING ORDER BY (id2 ASC);
>
> But we found out it is very difficult to remove a specific batch as we
> need to know all the IDs to delete the entries and it's both time and
> resource consuming (ie. it takes a long time and I'm not sure it's going to
> scale at all.)
>
> So, we are currently looking into having each of our batches in a keyspace
> of their own so removing a batch is merely equivalent to delete a keyspace.
> Potentially, it means we will end up having several hundreds of keyspaces
> in one cluster, although most of the time only the very last one will be
> used (we might still want to access the older ones, but that would be a
> very seldom use-case.) At the moment, the keyspace has about 14 tables and
> is probably not going to evolve much.
>
>
> Are there any counter-indications of using lot of keyspaces (300+) into
> one Cassandra cluster?
> Are there any good practices that we should follow?
> After reading the "Anti-patterns in Cassandra > Too many keyspaces or
> tables", does it mean we should plan ahead to already split our keyspace
> among several clusters?
>
> Finally, would there be any other way to achieve what we want to do?
>
> Thanks for your help!
>
>  Jonathan
>

--001a1143637823d21d052548c4da
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">How often is sometimes - closer to 20% of the batches or 2=
%?<div><br></div><div>How are you querying batches, both current and older =
ones?</div><div><br></div><div>As always, your queries should drive your da=
ta models.</div><div><br></div><div>If deleting a batch is very infrequent,=
 maybe best to not do it and simply have logic in the app to ignore deleted=
 batches - if your queries would reference them at all.</div><div><br></div=
><div>What reasons would you have to delete a batch? Depending on the natur=
e of the reason there may be an alternative.</div><div><br></div><div>Make =
sure your cluster is adequately provisioned so that these expensive operati=
ons can occur in parallel to reduce their time and resources per node.</div=
><div><br></div><div>Do all batches eventually get aged and deleted or are =
you expecting that most batches will live for many years to come? Have you =
planned for how you will grow the cluster over time?</div><div><br></div><d=
iv>Maybe bite the bullet and use a background process to delete a batch if =
deletion is competing too heavily with query access - if they really need t=
o be deleted at all.</div><div><br></div><div>Number of keyspaces - and/or =
tables - should be limited to &quot;low hundreds&quot;, and even then you a=
re limited by RAM and CPU of each node. If a keyspace has 14 tables, then 2=
50/14 =3D 20 would be a recommended upper limit for number of key spaces. E=
ven if your total number of tables was under 300 or even 200, you would nee=
d to do a proof of concept implementation to verify that your specific data=
 works well on your specific hardware.</div><div><br></div></div><div class=
=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_signature"><div=
 dir=3D"ltr">-- Jack Krupansky</div></div></div>
<br><div class=3D"gmail_quote">On Tue, Nov 24, 2015 at 5:05 AM, Jonathan Ba=
llet <span dir=3D"ltr">&lt;<a href=3D"mailto:jballet@edgelab.ch" target=3D"=
_blank">jballet@edgelab.ch</a>&gt;</span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex">Hi,<br>
<br>
we are running an application which produces every night a batch with sever=
al hundreds of Gigabytes of data. Once a batch has been computed, it is nev=
er modified (nor updates nor deletes), we just keep producing new batches e=
very day.<br>
<br>
Now, we are *sometimes* interested to remove a complete specific batch alto=
gether. At the moment, we are accumulating all these data into only one key=
space which has a batch ID column in all our tables which is also part of t=
he primary key. A sample table looks similar to this:<br>
<br>
=C2=A0 CREATE TABLE computation_results (<br>
=C2=A0 =C2=A0 =C2=A0 batch_id int,<br>
=C2=A0 =C2=A0 =C2=A0 id1 int,<br>
=C2=A0 =C2=A0 =C2=A0 id2 int,<br>
=C2=A0 =C2=A0 =C2=A0 value double,<br>
=C2=A0 =C2=A0 =C2=A0 PRIMARY KEY ((batch_id, id1), id2)<br>
=C2=A0 ) WITH CLUSTERING ORDER BY (id2 ASC);<br>
<br>
But we found out it is very difficult to remove a specific batch as we need=
 to know all the IDs to delete the entries and it&#39;s both time and resou=
rce consuming (ie. it takes a long time and I&#39;m not sure it&#39;s going=
 to scale at all.)<br>
<br>
So, we are currently looking into having each of our batches in a keyspace =
of their own so removing a batch is merely equivalent to delete a keyspace.=
 Potentially, it means we will end up having several hundreds of keyspaces =
in one cluster, although most of the time only the very last one will be us=
ed (we might still want to access the older ones, but that would be a very =
seldom use-case.) At the moment, the keyspace has about 14 tables and is pr=
obably not going to evolve much.<br>
<br>
<br>
Are there any counter-indications of using lot of keyspaces (300+) into one=
 Cassandra cluster?<br>
Are there any good practices that we should follow?<br>
After reading the &quot;Anti-patterns in Cassandra &gt; Too many keyspaces =
or tables&quot;, does it mean we should plan ahead to already split our key=
space among several clusters?<br>
<br>
Finally, would there be any other way to achieve what we want to do?<br>
<br>
Thanks for your help!<span class=3D"HOEnZb"><font color=3D"#888888"><br>
<br>
=C2=A0Jonathan<br>
</font></span></blockquote></div><br></div>

--001a1143637823d21d052548c4da--