Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of olek.stasiak@gmail.com
 designates 209.85.220.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd1rT_CRCibqhMG9_gWZH98gzcurw3aBTc+VJv1y=UPhSw@mail.gmail.com>
References: 
 <CA+VqcYMbgdwbqDpAnR7mWqBtN16X+X0oRTEoA11z-33PjsCxaQ@mail.gmail.com>
	<CAEDUwd1rT_CRCibqhMG9_gWZH98gzcurw3aBTc+VJv1y=UPhSw@mail.gmail.com>
Date: Mon, 3 Feb 2014 09:51:20 +0100
Message-ID: 
 <CACUS8n1jAdx1sDi=dkO58G9KLw86LK4zAVvrSGXQhXJ+HvT5sQ@mail.gmail.com>
Subject: Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3
From: "olek.stasiak@gmail.com" <olek.stasiak@gmail.com>
To: Emaillist for cass users <user@cassandra.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Hi All,
We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
1.2.10). Probably after upgradesstable  (but it's only a guess,
because we noticed problem few weeks later), some rows became
tombstoned. They just disappear from results of queries. After
inverstigation I've noticed, that they are reachable via sstable2json.
Example output for "non-existent" row:

{"key": "6e6e37716c6d665f6f61695f6463","metadata": {"deletionInfo":
{"markedForDeleteAt":2201170739199,"localDeletionTime":0}},"columns":
[["DATA","3c6f61695f64633a64(...)",1357677928108]]}
]

If I understand correctly row is marked as deleted with timestamp in
the far future, but it's still on the disk. Also localDeletionTime is
set to 0, which may means, that it's kind of internal bug, not effect
of client error. So my question is: is it true, that upgradesstable
may do soemthing like that? How to find reasons for such strange
cassandra behaviour? Is there any option of recovering such strange
marked nodes?
This problem touches about 500K rows of all 14M in our database, so
the percentage is quite big.
best regards
Aleksander

2013-12-12 Robert Coli <rcoli@eventbrite.com>:
> On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang <mathijs@apptornado.com>
> wrote:
>>
>> When I use sstable2json on the sstable on the destination cluster, it has
>> "metadata": {"deletionInfo":
>> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas
>> it doesn't have that in the source sstable.
>> (Yes, this is a timestamp far into the future. All our hosts are
>> properly synced through ntp).
>
>
> This seems like a bug in sstableloader, I would report it on JIRA.
>
>>
>> Naturally, copying the data again doesn't work to fix it, as the
>> tombstone is far in the future. Apart from not having this happen at
>> all, how can it be fixed?
>
>
> Briefly, you'll want to purge that tombstone and then reload the data with a
> reasonable timestamp.
>
> Dealing with rows with data (and tombstones) in the far future is described
> in detail here :
>
> http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html
>
> =Rob
>