Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of olek.stasiak@gmail.com
 designates 209.85.212.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd29zmjutGCrB-8JPmyjmrVo9VeckS1wUiRRmv59cy4Vpg@mail.gmail.com>
References: 
 <CA+VqcYMbgdwbqDpAnR7mWqBtN16X+X0oRTEoA11z-33PjsCxaQ@mail.gmail.com>
	<CAEDUwd1rT_CRCibqhMG9_gWZH98gzcurw3aBTc+VJv1y=UPhSw@mail.gmail.com>
	<CACUS8n1jAdx1sDi=dkO58G9KLw86LK4zAVvrSGXQhXJ+HvT5sQ@mail.gmail.com>
	<CAEDUwd29zmjutGCrB-8JPmyjmrVo9VeckS1wUiRRmv59cy4Vpg@mail.gmail.com>
Date: Mon, 3 Feb 2014 22:02:51 +0100
Message-ID: 
 <CACUS8n3CXXhRUJ4VewdoaNnRHdTLt3RFxXpKf_TLnxfZK135dg@mail.gmail.com>
Subject: Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3
From: "olek.stasiak@gmail.com" <olek.stasiak@gmail.com>
To: Emaillist for cass users <user@cassandra.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Yes, I haven't run sstableloader. The data loss apperared somwhere on the line:
1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3
normal operations -> now
Today I've noticed that oldest files with broken values appear during
repair (we do repair once a week on each node). Maybe it's the repair
operation, which caused data loss? I've no idea. Currently our cluster
is runing 2.0.3 version.
We can do some tests on data to give you all info to track the bug.
But our most crucial question is: can we recover loss, or should we
start to think how to re-gather them?
best regards
Aleksander
ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
were a rule: never deploy RDBMS before release 2 ;)

2014-02-03 Robert Coli <rcoli@eventbrite.com>:
> On Mon, Feb 3, 2014 at 12:51 AM, olek.stasiak@gmail.com
> <olek.stasiak@gmail.com> wrote:
>>
>> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
>> 1.2.10). Probably after upgradesstable  (but it's only a guess,
>> because we noticed problem few weeks later), some rows became
>> tombstoned.
>
>
> To be clear, you didn't run SSTableloader at all? If so, this is the
> hypothetical case where normal streaming operations (replacing a node? what
> streaming did you do?) results in data loss...
>
> Also, CASSANDRA-6527 is a good reminder regarding the following :
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
> =Rob