Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <56964077.7080203@pubgrade.com>
References: <56964077.7080203@pubgrade.com>
From: Alex Popescu <alexp@datastax.com>
Date: Wed, 13 Jan 2016 05:50:29 -0800
Message-ID: 
 <CAJ8Xi+Rd-nmB=SExZ=jmt_fkbUEK2inFcKArkuTVU8DnPGsUuA@mail.gmail.com>
Subject: Re: Spark Cassandra Java Connector: records missing despite
 consistency=ALL
To: user <user@cassandra.apache.org>
Cc: user@spark.apache.org
Content-Type: multipart/alternative; boundary=001a1140b7702b5060052937750d

--001a1140b7702b5060052937750d
Content-Type: text/plain; charset=UTF-8

Dennis,

You'll have better chances to get an answer on the
spark-cassandra-connector mailing list
https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-connector-user
or on IRC #spark-cassandra-connector

On Wed, Jan 13, 2016 at 4:17 AM, Dennis Birkholz <birkholz@pubgrade.com>
wrote:

> Hi together,
>
> we Cassandra to log event data and process it every 15 minutes with Spark.
> We are using the Cassandra Java Connector for Spark.
>
> Randomly our Spark runs produce too few output records because no data is
> returned from Cassandra for a several minutes window of input data. When
> querying the data (with cqlsh), after multiple tries, the data eventually
> becomes available.
>
> To solve the problem, we tried to use consistency=ALL when reading the
> data in Spark. We use the
> CassandraJavaUtil.javafunctions().cassandraTable() method and have set
> "spark.cassandra.input.consistency.level"="ALL" on the config when creating
> the Spark context. The problem persists but according to
> http://stackoverflow.com/a/25043599 using a consistency level of ONE on
> the write side (which we use) and ALL on the READ side should be sufficient
> for data consistency.
>
> I would really appreciate if someone could give me a hint how to fix this
> problem, thanks!
>
> Greets,
> Dennis
>
> P.s.:
> some information about our setup:
> Cassandra 2.1.12 in a two Node configuration with replication factor=2
> Spark 1.5.1
> Cassandra Java Driver 2.2.0-rc3
> Spark Cassandra Java Connector 2.10-1.5.0-M2
>


-- 
Bests,

Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax

--001a1140b7702b5060052937750d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Dennis,<div><br></div><div>You&#39;ll have better chances =
to get an answer on the spark-cassandra-connector mailing list=C2=A0<a href=
=3D"https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-conn=
ector-user">https://groups.google.com/a/lists.datastax.com/forum/#!forum/sp=
ark-connector-user</a> or on IRC=C2=A0#spark-cassandra-connector</div></div=
><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Jan 13, =
2016 at 4:17 AM, Dennis Birkholz <span dir=3D"ltr">&lt;<a href=3D"mailto:bi=
rkholz@pubgrade.com" target=3D"_blank">birkholz@pubgrade.com</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex">Hi together,<br>
<br>
we Cassandra to log event data and process it every 15 minutes with Spark. =
We are using the Cassandra Java Connector for Spark.<br>
<br>
Randomly our Spark runs produce too few output records because no data is r=
eturned from Cassandra for a several minutes window of input data. When que=
rying the data (with cqlsh), after multiple tries, the data eventually beco=
mes available.<br>
<br>
To solve the problem, we tried to use consistency=3DALL when reading the da=
ta in Spark. We use the CassandraJavaUtil.javafunctions().cassandraTable() =
method and have set &quot;spark.cassandra.input.consistency.level&quot;=3D&=
quot;ALL&quot; on the config when creating the Spark context. The problem p=
ersists but according to <a href=3D"http://stackoverflow.com/a/25043599" re=
l=3D"noreferrer" target=3D"_blank">http://stackoverflow.com/a/25043599</a> =
using a consistency level of ONE on the write side (which we use) and ALL o=
n the READ side should be sufficient for data consistency.<br>
<br>
I would really appreciate if someone could give me a hint how to fix this p=
roblem, thanks!<br>
<br>
Greets,<br>
Dennis<br>
<br>
P.s.:<br>
some information about our setup:<br>
Cassandra 2.1.12 in a two Node configuration with replication factor=3D2<br=
>
Spark 1.5.1<br>
Cassandra Java Driver 2.2.0-rc3<br>
Spark Cassandra Java Connector 2.10-1.5.0-M2<br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=
=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div>Bests,</div>=
<div><br>Alex Popescu |=C2=A0<span style=3D"line-height:1.6em">@al3xandru</=
span></div><div dir=3D"ltr">Sen. Product Manager @ DataStax<br><br></div></=
div></div></div></div></div></div></div></div></div></div></div>
</div>

--001a1140b7702b5060052937750d--