cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Dunkler <p...@uplex.de>
Subject Re: [Marketing Mail] [Marketing Mail] Cassandra 2.1: Snapshot data changing while transferring
Date Tue, 07 Jun 2016 09:28:23 GMT
Hi there,

just for your information: I found out whats actually happening.
It's not the file content which is changing. It's the number of hard links to the files which
is changing because during the tar is running, some incremental backups are created.

In that case, tar gives a warning that some data changed while it was reading. It exits with
code 1 - and this needs to be ignored in the script.


> Hi Reynald,
> 
>> If I understand correctly, you are making a tar file with all the folders named "snapshots"
(i.e. the folder under which all the snapshots are created. So you have one snapshots folder
per table).
> 
> No, Thats not the case. We are doing a nightly snapshot of the whole database (named
with a date).
> 
>> If this is the case, when you are executing "nodetool repair", Cassandra will create
a snapshot at the beginning of the repair, creating a new directory under each snapshots directories.
If this happens while you are creating your tar file, you will get the error you saw.
> 
> Sure, i am aware of that. No, we are only taring the snapshot we just created some second
before.
> 
>> If you are not yet doing it, I advise you to use the -t option of the "nodetool snapshot"
command to specify a specific name to your snapshot.
>> Then you should copy only the directories named snapshots/<your_specific_snapshot_name>
in your tar file.
> 
> We're doing exactly that.
> 
>> Can you confirm that you are creating your tar file from the snapshots directories
directly (resulting in taking all the currently generated snapshots)?
> 
> As i wrote above, we just do a snapshot and ONLY tar this exact snapshot.
> 
> Short (high level) overview about our backup script:
> 
> 1. check if repair is running - if yes, exit
> 2. Dump the current db-schema
> 3. nodetool snapshot -t $(date)
> 4. Wait 60 seconds
> 5. Create a tar of all snapshot folders with the date we just created
> 6. Copy that away to a remote server
> 
>> 
>> Kind regards
>> 
>> Reynald
>> 
>> On 01/06/2016 13:27, Paul Dunkler wrote:
>>>> I guess this might come from the incremental repairs...
>>>> The repair time is stored in the sstable (RepairedAt timestamp metadata).
>>> 
>>> By the way: We are not using incremental repairs at all. So can't be the case
here.
>>> 
>>> It really seems like there is somewhat that can still change data in snapshot
directories. I feel like it's something to do with flushing / compaction. But no clue, what...
:(
>>> 
>>>> 
>>>> Cheers,
>>>> Reynald
>>>> 
>>>> On 31/05/2016 11:03, Paul Dunkler wrote:
>>>>> Hi there,
>>>>> 
>>>>> i am sometimes running in very strange errors while backing up snapshots
from a cassandra cluster.
>>>>> 
>>>>> Cassandra version:
>>>>> 2.1.11
>>>>> 
>>>>> What i basically do:
>>>>> 1. nodetool snapshot
>>>>> 2. tar all snapshot folders into one file
>>>>> 3. transfer them to another server
>>>>> 
>>>>> What happens is that tar just sometimes give the error message "file
changed as we read it" while its adding a .db-file from the folder of the previously created
snapshot.
>>>>> If i understand everything correct, this SHOULD never happen. Snapshots
should be totally immutable, right?
>>>>> 
>>>>> Am i maybe hitting a bug or is there some rare case with running repair
operations or what-so-ever which can change snapshotted data?
>>>>> I already searched through cassandra jira but couldn't find a bug which
looks related to this behaviour.
>>>>> 
>>>>> Would love to get some help on this.
>>>>> 
>>>>> —
>>>>> Paul Dunkler
>>>> 
>>> 
>>> —
>>> Paul Dunkler
>> 
> 
> —
> Paul Dunkler
> 
> ** * * UPLEX - Nils Goroll Systemoptimierung
> 
> Scheffelstraße 32
> 22301 Hamburg
> 
> tel +49 40 288 057 31
> mob +49 151 252 228 42
> fax +49 40 429 497 53
> 
> xmpp://pauldunkler@jabber.ccc <xmpp://pauldunkler@jabber.ccc>.de
> 
> http://uplex.de/ <http://uplex.de/>
—
Paul Dunkler

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

tel +49 40 288 057 31
mob +49 151 252 228 42
fax +49 40 429 497 53

xmpp://pauldunkler@jabber.ccc <xmpp://pauldunkler@jabber.ccc>.de

http://uplex.de/ <http://uplex.de/>

Mime
View raw message