cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Birkholz <birkh...@pubgrade.com>
Subject Re: Spark Cassandra Java Connector: records missing despite consistency=ALL
Date Thu, 21 Jan 2016 17:16:05 GMT
Hi Anthony,

no, the logging is not done via Spark (but PHP). But that does not 
really matter, as the records are eventually there. So it is the 
READ_CONSISTENCY=ALL that is not working.

Btw. it seems that using withReadConf() and setting the consistency 
level there is working but I need to wait a few more days before I am 
sure of that.

Kind regards,
Dennis

Am 19.01.2016 um 19:39 schrieb Femi Anthony:
> So is the logging to Cassandra being done via Spark ?
>
> On Wed, Jan 13, 2016 at 7:17 AM, Dennis Birkholz <birkholz@pubgrade.com
> <mailto:birkholz@pubgrade.com>> wrote:
>
>     Hi together,
>
>     we Cassandra to log event data and process it every 15 minutes with
>     Spark. We are using the Cassandra Java Connector for Spark.
>
>     Randomly our Spark runs produce too few output records because no
>     data is returned from Cassandra for a several minutes window of
>     input data. When querying the data (with cqlsh), after multiple
>     tries, the data eventually becomes available.
>
>     To solve the problem, we tried to use consistency=ALL when reading
>     the data in Spark. We use the
>     CassandraJavaUtil.javafunctions().cassandraTable() method and have
>     set "spark.cassandra.input.consistency.level"="ALL" on the config
>     when creating the Spark context. The problem persists but according
>     to http://stackoverflow.com/a/25043599 using a consistency level of
>     ONE on the write side (which we use) and ALL on the READ side should
>     be sufficient for data consistency.
>
>     I would really appreciate if someone could give me a hint how to fix
>     this problem, thanks!
>
>     Greets,
>     Dennis
>
>     P.s.:
>     some information about our setup:
>     Cassandra 2.1.12 in a two Node configuration with replication factor=2
>     Spark 1.5.1
>     Cassandra Java Driver 2.2.0-rc3
>     Spark Cassandra Java Connector 2.10-1.5.0-M2
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>     <mailto:user-unsubscribe@spark.apache.org>
>     For additional commands, e-mail: user-help@spark.apache.org
>     <mailto:user-help@spark.apache.org>
>
>
>
>
> --
> http://www.femibyte.com/twiki5/bin/view/Tech/
> http://www.nextmatrix.com
> "Great spirits have always encountered violent opposition from mediocre
> minds." - Albert Einstein.


Mime
View raw message