cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Tremblay <jean.tremb...@zen-innovations.com>
Subject Re: Written data is lost and no exception thrown back to the client
Date Wed, 26 Aug 2015 05:58:18 GMT
I have the same problem.

When I bulk load my data, I have a problem with Cassandra Datastax driver.

<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.4</version> <!-- Driver 2.1.6, 2.1.7.1 gives problems. Some data
is lost. -->
</dependency>

With version 2.1.6 and also with version 2.1.7.1 I have lost records with no error message
what so ever.
With version 2.1.4 I have no missing records.

I use CL.ONE to write my records. I use RF 3.






On 21 Aug 2015, at 13:06 , Robert Wille <rwille@fold3.com<mailto:rwille@fold3.com>>
wrote:

But it shouldn’t matter. I have missing data, and no errors, which shouldn’t be possible
except with CL=ANY.

FWIW, I’m working on some sample code so I can post a Jira.

Robert

On Aug 21, 2015, at 5:04 AM, Robert Wille <rwille@fold3.com<mailto:rwille@fold3.com>>
wrote:

RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should be the same
as ONE. If’s QUORUM instead of ONE because production has RF=3, and I was running this against
my test cluster with RF=1.

On Aug 20, 2015, at 7:28 PM, Jason <jkushmaul@rocketfuelinc.com<mailto:jkushmaul@rocketfuelinc.com>>
wrote:

What consistency level were the writes?
________________________________
From: Robert Wille<mailto:rwille@fold3.com>
Sent: ‎8/‎20/‎2015 18:25
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Written data is lost and no exception thrown back to the client

I wrote a data migration application which I was testing, and I pushed it too hard and the
FlushWriter thread pool blocked, and I ended up with dropped mutation messages. I compared
the source data against what is in my cluster, and as expected I have missing records. The
strange thing is that my application didn’t error out. I’ve been doing some forensics,
and there’s a lot about this that makes no sense and makes me feel very uneasy.

I use a lot of asynchronous queries, and I thought it was possible that I had bad error handling,
so I checked for errors in other, independent ways.

I have a retry policy that on the first failure logs the error and then requests a retry.
On the second failure it logs the error and then rethrows. A few retryable errors appeared
in my logs, but no fatal errors. In theory, I should have a fatal error in my logs for any
error that gets reported back to the client.

I wrap my Session object, and all queries go through this wrapper. This wrapper logs all query
errors. Synchronous queries are wrapped in a try/catch which logs and rethrows. Asynchronous
queries use a FutureCallback to log any onFailure invocations.

My logs indicate that no errors whatsoever were reported back to me. I do not understand how
I can get dropped mutation messages and not know about it. I am running 2.0.16 with datastax
Java driver 2.0.8. Three node cluster with RF=1. If someone could help me understand how this
can occur, I would greatly appreciate it. A database that errors out is one thing. A database
that errors out and makes you think everything was fine is quite another.

Thanks

Robert




Mime
View raw message