Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of doanduyhai@gmail.com designates
 209.85.214.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACz-ikuYF85EOr0fMZ2QC7QKWc3JN0sQC3U_Zd=ZGBUtCx47DQ@mail.gmail.com>
References: 
 <CACz-ikuYF85EOr0fMZ2QC7QKWc3JN0sQC3U_Zd=ZGBUtCx47DQ@mail.gmail.com>
Date: Fri, 20 Feb 2015 22:43:07 +0100
Message-ID: 
 <CABNXB2Duhv0W1daAieoR9Z-a5pnEuvtR6-GLLq2KAYiQrvQuRQ@mail.gmail.com>
Subject: Re: Running Cassandra + Spark on AWS - architecture questions
From: DuyHai Doan <doanduyhai@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c2522c2bfd57050f8bf037

--001a11c2522c2bfd57050f8bf037
Content-Type: text/plain; charset=UTF-8

"Cassandra would take care of keeping the data synced between the two sets
of five nodes.  Is that correct?"

Correct

"But doing so means that we need 2x as many nodes as we need for the
real-time cluster alone"

Not necessarily. With multi DC you can configure the replication factor
value per DC, meaning that you can have RF = 3 for the real time DC and
RF=1 or RF=2 for the analytics DC. Thus the number of nodes can be
different for each DC

In addition, you can also tune the hardware. If the realtime DC is mostly
write only for incoming data and read-only from aggregated table, it is
less IO intensive than the analytics DC with lot of read & write to compute
aggregations.


On Fri, Feb 20, 2015 at 10:17 PM, Clint Kelly <clint.kelly@gmail.com> wrote:

> Hi all,
>
> I read the DSE 4.6 documentation and I'm still not 100% sure what a mixed
> workload Cassandra + Spark installation would look like, especially on
> AWS.  What I gather is that you use OpsCenter to set up the following:
>
>
>    - One "virtual data center" for real-time processing (e.g., ingestion
>    of time-series data, replying to requests for an interactive application)
>    - Another "virtual data center" for batch analytics (Spark, possibly
>    for machine learning)
>
>
> If I understand this correctly, if I estimate that I need a five-node
> cluster to handle all of my data, under the system described above, I would
> have five nodes serving real-time traffic and all of the data replicated in
> another five nodes that I use for batch processing.  Cassandra would take
> care of keeping the data synced between the two sets of five nodes.  Is
> that correct?
>
> I assume the motivation for such a dual-virtual-data-center architecture
> is to prevent the Spark jobs (which are going to do lots of scans from
> Cassandra, and maybe run computation on the same machines hosting
> Cassandra) from disrupting the real-time performance.  But doing so means
> that we need 2x as many nodes as we need for the real-time cluster alone.
>
> *Could someone confirm that my interpretation above of what I read about
> in the DSE documentation is correct?*
>
> If my application needs to run analytics on Spark only a few hours a day,
> might we be better off spending our money to get a bigger Cassandra cluster
> and then just spin up Spark jobs on EMR for a few hours at night?  (I know
> this is a hard question to answer, since it all depends on the
> application---just curious if anyone else here has had to make similar
> tradeoffs.)  e.g., maybe instead of having a five-node real-time cluster,
> we would have an eight-node real-time cluster, and use our remaining budget
> on EMR jobs.
>
> I am curious if anyone has any thoughts / experience about this.
>
> Best regards,
> Clint
>

--001a11c2522c2bfd57050f8bf037
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">&quot;<span style=3D"font-size:12.8000001907349px">Cassand=
ra would take care of keeping the data synced between the two sets of five =
nodes.=C2=A0 Is that correct?&quot;</span><div><span style=3D"font-size:12.=
8000001907349px"><br></span></div><div><span style=3D"font-size:12.80000019=
07349px">Correct</span></div><div><span style=3D"font-size:12.8000001907349=
px"><br></span></div><div><span style=3D"font-size:12.8000001907349px">&quo=
t;</span><span style=3D"font-size:12.8000001907349px">But doing so means th=
at we need 2x as many nodes as we need for the real-time cluster alone&quot=
;</span></div><div><span style=3D"font-size:12.8000001907349px"><br></span>=
</div><div><span style=3D"font-size:12.8000001907349px">Not necessarily. Wi=
th multi DC you can configure the replication factor value per DC, meaning =
that you can have RF =3D 3 for the real time DC and RF=3D1 or RF=3D2 for th=
e analytics DC. Thus the number of nodes can be different for each DC</span=
></div><div><span style=3D"font-size:12.8000001907349px"><br></span></div><=
div><span style=3D"font-size:12.8000001907349px">In addition, you can also =
tune the hardware. If the realtime DC is mostly write only for incoming dat=
a and read-only from aggregated table, it is less IO intensive than the ana=
lytics DC with lot of read &amp; write to compute aggregations.</span></div=
><div><span style=3D"font-size:12.8000001907349px"><br></span></div><div><s=
pan style=3D"font-size:12.8000001907349px"><br></span></div></div><div clas=
s=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Feb 20, 2015 at 10=
:17 PM, Clint Kelly <span dir=3D"ltr">&lt;<a href=3D"mailto:clint.kelly@gma=
il.com" target=3D"_blank">clint.kelly@gmail.com</a>&gt;</span> wrote:<br><b=
lockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px =
#ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi all,<div><br></div><div>I =
read the DSE 4.6 documentation and I&#39;m still not 100% sure what a mixed=
 workload Cassandra + Spark installation would look like, especially on AWS=
.=C2=A0 What I gather is that you use OpsCenter to set up the following:</d=
iv><div><br></div><div><ul><li>One &quot;virtual data center&quot; for real=
-time processing (e.g., ingestion of time-series data, replying to requests=
 for an interactive application)</li><li>Another &quot;virtual data center&=
quot; for batch analytics (Spark, possibly for machine learning)</li></ul><=
div><br></div><div>If I understand this correctly, if I estimate that I nee=
d a five-node cluster to handle all of my data, under the system described =
above, I would have five nodes serving real-time traffic and all of the dat=
a replicated in another five nodes that I use for batch processing.=C2=A0 C=
assandra would take care of keeping the data synced between the two sets of=
 five nodes.=C2=A0 Is that correct?</div></div><div><br></div><div>I assume=
 the motivation for such a dual-virtual-data-center architecture is to prev=
ent the Spark jobs (which are going to do lots of scans from Cassandra, and=
 maybe run computation on the same machines hosting Cassandra) from disrupt=
ing the real-time performance.=C2=A0 But doing so means that we need 2x as =
many nodes as we need for the real-time cluster alone.</div><div><br></div>=
<div><b><i>Could someone confirm that my interpretation above of what I rea=
d about in the DSE documentation is correct?</i></b></div><div><br></div><d=
iv>If my application needs to run analytics on Spark only a few hours a day=
, might we be better off spending our money to get a bigger Cassandra clust=
er and then just spin up Spark jobs on EMR for a few hours at night? =C2=A0=
(I know this is a hard question to answer, since it all depends on the appl=
ication---just curious if anyone else here has had to make similar tradeoff=
s.) =C2=A0e.g., maybe instead of having a five-node real-time cluster, we w=
ould have an eight-node real-time cluster, and use our remaining budget on =
EMR jobs.</div><div><br></div><div>I am curious if anyone has any thoughts =
/ experience about this.</div><div><br></div><div>Best regards,</div><div>C=
lint</div></div>
</blockquote></div><br></div>

--001a11c2522c2bfd57050f8bf037--