cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynald Bourtembourg <reynald.bourtembo...@esrf.fr>
Subject Re: [Marketing Mail] Re: [Marketing Mail] Cassandra 2.1: Snapshot data changing while transferring
Date Wed, 01 Jun 2016 12:20:12 GMT
Hi Paul,

If I understand correctly, you are making a tar file with all the 
folders named "snapshots" (i.e. the folder under which all the snapshots 
are created. So you have one /snapshots /folder per table).
If this is the case, when you are executing "nodetool repair", Cassandra 
will create a snapshot at the beginning of the repair, creating a new 
directory under each /snapshots/ directories. If this happens while you 
are creating your tar file, you will get the error you saw.

If you are not yet doing it, I advise you to use the -t option of the 
"nodetool snapshot" command to specify a specific name to your snapshot.
Then you should copy only the directories named 
snapshots/<your_specific_snapshot_name> in your tar file.

Can you confirm that you are creating your tar file from the snapshots 
directories directly (resulting in taking all the currently generated 
snapshots)?

Kind regards

Reynald

On 01/06/2016 13:27, Paul Dunkler wrote:
>> I guess this might come from the incremental repairs...
>> The repair time is stored in the sstable (RepairedAt timestamp metadata).
>
> By the way: We are not using incremental repairs at all. So can't be 
> the case here.
>
> It really seems like there is somewhat that can still change data in 
> snapshot directories. I feel like it's something to do with flushing / 
> compaction. But no clue, what... :(
>
>>
>> Cheers,
>> Reynald
>>
>> On 31/05/2016 11:03, Paul Dunkler wrote:
>>> Hi there,
>>>
>>> i am sometimes running in very strange errors while backing up 
>>> snapshots from a cassandra cluster.
>>>
>>> Cassandra version:
>>> 2.1.11
>>>
>>> What i basically do:
>>> 1. nodetool snapshot
>>> 2. tar all snapshot folders into one file
>>> 3. transfer them to another server
>>>
>>> What happens is that tar just sometimes give the error message "file 
>>> changed as we read it" while its adding a .db-file from the folder 
>>> of the previously created snapshot.
>>> If i understand everything correct, this SHOULD never happen. 
>>> Snapshots should be totally immutable, right?
>>>
>>> Am i maybe hitting a bug or is there some rare case with running 
>>> repair operations or what-so-ever which can change snapshotted data?
>>> I already searched through cassandra jira but couldn't find a bug 
>>> which looks related to this behaviour.
>>>
>>> Would love to get some help on this.
>>>
>>> —
>>> Paul Dunkler
>>
>
> —
> Paul Dunkler


Mime
View raw message