On Mon, Feb 3, 2014 at 1:02 PM, olek.stasiak@gmail.com <olek.stasiak@gmail.com> wrote:
Today I've noticed that oldest files with broken values appear during
repair (we do repair once a week on each node). Maybe it's the repair
operation, which caused data loss?

Yes, unless you added or removed or replaced nodes, it would have to be the repair operation, which streams SSTables. Did you run the repair during the upgradesstables?
 
I've no idea. Currently our cluster
is runing 2.0.3 version.

2.0.3 has serious bugs, upgrade to 2.0.4 ASAP.
 
But our most crucial question is: can we recover loss, or should we
start to think how to re-gather them?

If I were you, I would do the latter. You can to some extent recover them via manual processes dumping with sstable2json and so forth, but it will be quite painful.

http://thelastpickle.com/2011/12/15/Anatomy-of-a-Cassandra-Partition/

Contains an explanation of how one could deal with it.

=Rob


 
best regards
Aleksander
ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
were a rule: never deploy RDBMS before release 2 ;)

2014-02-03 Robert Coli <rcoli@eventbrite.com>:
> On Mon, Feb 3, 2014 at 12:51 AM, olek.stasiak@gmail.com
> <olek.stasiak@gmail.com> wrote:
>>
>> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
>> 1.2.10). Probably after upgradesstable  (but it's only a guess,
>> because we noticed problem few weeks later), some rows became
>> tombstoned.
>
>
> To be clear, you didn't run SSTableloader at all? If so, this is the
> hypothetical case where normal streaming operations (replacing a node? what
> streaming did you do?) results in data loss...
>
> Also, CASSANDRA-6527 is a good reminder regarding the following :
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
> =Rob