incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "olek.stasiak@gmail.com" <olek.stas...@gmail.com>
Subject Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3
Date Mon, 03 Feb 2014 21:02:51 GMT
Yes, I haven't run sstableloader. The data loss apperared somwhere on the line:
1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3
normal operations -> now
Today I've noticed that oldest files with broken values appear during
repair (we do repair once a week on each node). Maybe it's the repair
operation, which caused data loss? I've no idea. Currently our cluster
is runing 2.0.3 version.
We can do some tests on data to give you all info to track the bug.
But our most crucial question is: can we recover loss, or should we
start to think how to re-gather them?
best regards
Aleksander
ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
were a rule: never deploy RDBMS before release 2 ;)

2014-02-03 Robert Coli <rcoli@eventbrite.com>:
> On Mon, Feb 3, 2014 at 12:51 AM, olek.stasiak@gmail.com
> <olek.stasiak@gmail.com> wrote:
>>
>> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
>> 1.2.10). Probably after upgradesstable  (but it's only a guess,
>> because we noticed problem few weeks later), some rows became
>> tombstoned.
>
>
> To be clear, you didn't run SSTableloader at all? If so, this is the
> hypothetical case where normal streaming operations (replacing a node? what
> streaming did you do?) results in data loss...
>
> Also, CASSANDRA-6527 is a good reminder regarding the following :
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
> =Rob

Mime
View raw message