cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Estevez <sebastian.este...@datastax.com>
Subject Re: Written data is lost and no exception thrown back to the client
Date Fri, 21 Aug 2015 17:01:10 GMT
Please let us know the Jira number.

All the best,


[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>

<http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Aug 21, 2015 at 7:06 AM, Robert Wille <rwille@fold3.com> wrote:

> But it shouldn’t matter. I have missing data, and no errors, which
> shouldn’t be possible except with CL=ANY.
>
> FWIW, I’m working on some sample code so I can post a Jira.
>
> Robert
>
>
> On Aug 21, 2015, at 5:04 AM, Robert Wille <rwille@fold3.com> wrote:
>
> RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it
> should be the same as ONE. If’s QUORUM instead of ONE because production
> has RF=3, and I was running this against my test cluster with RF=1.
>
> On Aug 20, 2015, at 7:28 PM, Jason <jkushmaul@rocketfuelinc.com> wrote:
>
> What consistency level were the writes?
> ------------------------------
> From: Robert Wille <rwille@fold3.com>
> Sent: ‎8/‎20/‎2015 18:25
> To: user@cassandra.apache.org
> Subject: Written data is lost and no exception thrown back to the client
>
> I wrote a data migration application which I was testing, and I pushed it
> too hard and the FlushWriter thread pool blocked, and I ended up with
> dropped mutation messages. I compared the source data against what is in my
> cluster, and as expected I have missing records. The strange thing is that
> my application didn’t error out. I’ve been doing some forensics, and
> there’s a lot about this that makes no sense and makes me feel very uneasy.
>
> I use a lot of asynchronous queries, and I thought it was possible that I
> had bad error handling, so I checked for errors in other, independent ways.
>
> I have a retry policy that on the first failure logs the error and then
> requests a retry. On the second failure it logs the error and then
> rethrows. A few retryable errors appeared in my logs, but no fatal errors.
> In theory, I should have a fatal error in my logs for any error that gets
> reported back to the client.
>
> I wrap my Session object, and all queries go through this wrapper. This
> wrapper logs all query errors. Synchronous queries are wrapped in a
> try/catch which logs and rethrows. Asynchronous queries use a
> FutureCallback to log any onFailure invocations.
>
> My logs indicate that no errors whatsoever were reported back to me. I do
> not understand how I can get dropped mutation messages and not know about
> it. I am running 2.0.16 with datastax Java driver 2.0.8. Three node cluster
> with RF=1. If someone could help me understand how this can occur, I would
> greatly appreciate it. A database that errors out is one thing. A database
> that errors out and makes you think everything was fine is quite another.
>
> Thanks
>
> Robert
>
>
>
>

Mime
View raw message