Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of adrian.cockcroft@gmail.com
 designates 209.85.214.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ph4aBADiOkkZKurvRAbUNgqfP2RmJoJ8m4DF8L60LSo6F40i9uPo0siW0y0CHzwfzQ
         r2vbWoUvSjcCHsW+ubAwsehdBNl34hNdhgdUuWj3NR5ozLqJivUL8iLbPBktveWHCRoa
         EnOh4fPZxfjWcgS8IdhRQrws8SZHIXaisrKqU=
MIME-Version: 1.0
In-Reply-To: <C28D2BCD-92D3-4D55-A63E-B3FC98D76E65@gmail.com>
References: <BANLkTimvjQ5=59aNMn5h=K5q=jJ3LZFRbQ@mail.gmail.com>
	<C28D2BCD-92D3-4D55-A63E-B3FC98D76E65@gmail.com>
Date: Thu, 28 Apr 2011 14:15:25 -0700
Message-ID: <BANLkTimkJHpq5pOrfNZBJ9K=WiixroF+1w@mail.gmail.com>
Subject: Re: best way to backup
From: Adrian Cockcroft <adrian.cockcroft@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Netflix has also gone down this path, we run a regular full backup to
S3 of a compressed tar, and we have scripts that restore everything
into the right place on a different cluster (it needs the same node
count). We also pick up the SSTables as they are created, and drop
them in S3.

Whatever you do, make sure you have a regular process to restore the
data and verify that it contains what you think it should...

Adrian

On Thu, Apr 28, 2011 at 1:35 PM, Jeremy Hanna
<jeremy.hanna1234@gmail.com> wrote:
> one thing we're looking at doing is watching the cassandra data directory=
 and backing up the sstables to s3 when they are created. =A0Some guys at s=
implegeo started tablesnap that does this:
> https://github.com/simplegeo/tablesnap
>
> What it does is for every sstable that is pushed to s3, it also copies a =
json file with the current files in the directory, so you can know what to =
restore in that event (as far as I understand).
>
> On Apr 28, 2011, at 2:53 PM, William Oberman wrote:
>
>> Even with N-nodes for redundancy, I still want to have backups. =A0I'm a=
n amazon person, so naturally I'm thinking S3. =A0Reading over the docs, an=
d messing with nodeutil, it looks like each new snapshot contains the previ=
ous snapshot as a subset (and I've read how cassandra uses hard links to av=
oid excessive disk use). =A0When does that pattern break down?
>>
>> I'm basically debating if I can do a "rsync" like backup, or if I should=
 do a compressed tar backup. =A0And I obviously want multiple points in tim=
e. =A0S3 does allow file versioning, if a file or file name is changed/resu=
sed over time (only matters in the rsync case). =A0My only concerns with co=
mpressed tars is I'll have to have free space to create the archive and I g=
et no "delta" space savings on the backup (the former is solved by not allo=
wing the disk space to get so low and/or adding more nodes to bring down th=
e space, the latter is solved by S3 being really cheap anyways).
>>
>> --
>> Will Oberman
>> Civic Science, Inc.
>> 3030 Penn Avenue., First Floor
>> Pittsburgh, PA 15201
>> (M) 412-480-7835
>> (E) oberman@civicscience.com
>
>