Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6DF3F10156 for ; Fri, 31 May 2013 19:30:14 +0000 (UTC) Received: (qmail 31003 invoked by uid 500); 31 May 2013 19:30:14 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 30935 invoked by uid 500); 31 May 2013 19:30:14 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 30926 invoked by uid 99); 31 May 2013 19:30:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 May 2013 19:30:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [209.85.212.43] (HELO mail-vb0-f43.google.com) (209.85.212.43) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 May 2013 19:30:08 +0000 Received: by mail-vb0-f43.google.com with SMTP id e12so1316568vbg.2 for ; Fri, 31 May 2013 12:29:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=prOoHM42tXR66xk6exuX0xpFCH0t1MdfaMtTLSohdTA=; b=YVhvc7a78s5uIUqppF8t4Q9fKX7+fxppf5j+r8fo42nMH7NfNrwBoOybotC/6QZMZw AvWPAajJMyGlbglBRvlixWKe5cktXnM3fIkXdljWFnW+dWpcXUVawn4SY+6hSbILCySY zhA6+eFbsS7FqIsidnYQkOhBdJMIWnnwO0Zoamg5MlxK++5xwUspe7uygtt0F8MASv/n AeAuK9nO58UuYXHdeXYjwS46ShT0BT/H4l4gyAfNqfUKjNXhMlUcJ4ivIO3Blo0JJ6p6 b/kqQ30PzVCjCkT879Mj8tO9cx3amtzLF2NYZ8u+4ojicCWPqVe5Npjku5LmqiP/R9Nl U5UA== MIME-Version: 1.0 X-Received: by 10.58.172.67 with SMTP id ba3mr11404552vec.58.1370028566395; Fri, 31 May 2013 12:29:26 -0700 (PDT) Received: by 10.220.83.137 with HTTP; Fri, 31 May 2013 12:29:26 -0700 (PDT) In-Reply-To: References: Date: Fri, 31 May 2013 15:29:26 -0400 Message-ID: Subject: Re: Backup Strategies From: Keith Turner To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=047d7b677f760853c104de08a23e X-Gm-Message-State: ALoCoQkbBridMW/RuhpylvmFb1ZffbWDpYWkJu8O13sg+wLmXw9KH5jpLAsIpEx/+yV0aqqA157t X-Virus-Checked: Checked by ClamAV on apache.org --047d7b677f760853c104de08a23e Content-Type: text/plain; charset=ISO-8859-1 On Fri, May 31, 2013 at 2:39 PM, Billie Rinaldi wrote: > I'm not sure copying data out of HDFS is what you would want to do, though > I suppose it depends on how much data you're storing there. If you want a > backup on a different system, but you have too much data to store outside > of a distributed file system, you could consider using distcp to copy from > one HDFS instance to another. > > You can't clone the !METADATA table. In 1.5.0, you can export and import > tables, which is designed to help you copy a table to a different cluster > (see docs/examples/README.export). Cloning your tables could help, but in > the case of !METADATA corruption you're still in the position of manually > creating a new table with the same configuration (and split points if you > know them) and bulk importing the old data files. I don't know if table > export could be used to back up the metadata and configuration of a cloned > table to help you recover its state later on the same system if the > original table has gotten corrupted. It's possible. > Export table will save the tables state (whats in !METADATA in zookeeper) to a zipfile. So even if you do not actually copy the exported table, it can be used to save table metadata. I made comment on ACCUMULO-942 about using export table to obtain a consistent snapshot of HDFS and Accumulo metadata using export table. That system metadata could be backed up. > > > Billie > > > On Fri, May 31, 2013 at 11:05 AM, Mike Hugo wrote: > >> I'm curious to know how people are backing up data in Accumulo. >> >> We are planning on copying data out of HDFS on a some regular basis to be >> able to do full restore. >> >> We've also ended up getting into a state of having a corrupt !METADATA >> table a few times. I'm wondering if doing a clone on a few tables on a >> periodic basis (like every hour, for a few hours) might be one way to help >> us recover from that situation. >> >> E.g if we did a clone on all tables, including the !METADATA table >> hourly, and we didn't necessarily care about losing data in the last hour >> time frame, could we simply restore from one of those clones if we get into >> a corrupted state? >> >> Is there another mechanism for snapshotting / backing up data in Accumulo? >> >> Thanks for your thoughts! >> >> Mike >> > > --047d7b677f760853c104de08a23e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Fri, May 31, 2013 at 2:39 PM, Billie Rinaldi &= lt;billie.rin= aldi@gmail.com> wrote:
I'm not sure copying da= ta out of HDFS is what you would want to do, though I suppose it depends on= how much data you're storing there.=A0 If you want a backup on a diffe= rent system, but you have too much data to store outside of a distributed f= ile system, you could consider using distcp to copy from one HDFS instance = to another.

You can't clone the !METADATA table.=A0 In 1.5.0, you can expo= rt and import tables, which is designed to help you copy a table to a diffe= rent cluster (see docs/examples/README.export).=A0 Cloning your tables coul= d help, but in the case of !METADATA corruption you're still in the pos= ition of manually creating a new table with the same configuration (and spl= it points if you know them) and bulk importing the old data files.=A0 I don= 't know if table export could be used to back up the metadata and confi= guration of a cloned table to help you recover its state later on the same = system if the original table has gotten corrupted.=A0 It's possible.

Export table will save the tab= les state (whats in !METADATA in zookeeper) to a zipfile. =A0So even if you= do not actually copy the exported table, it can be used to save table meta= data. =A0 I made comment on ACCUMULO-942 about using export table to obtain= a consistent snapshot of HDFS and Accumulo metadata using export table. = =A0That system metadata could be backed up. =A0=A0

=A0


Billie


On Fri, May= 31, 2013 at 11:05 AM, Mike Hugo <mike@piragua.com> wrote:
I'm curious to know how= people are backing up data in Accumulo.

We are planning= on copying data out of HDFS on a some regular basis to be able to do full = restore.

We've also ended up getting into a state of having a cor= rupt !METADATA table a few times. =A0I'm wondering if doing a clone on = a few tables on a periodic basis (like every hour, for a few hours) might b= e one way to help us recover from that situation. =A0

E.g if we did a clone on all tables, including the !MET= ADATA table hourly, and we didn't necessarily care about losing data in= the last hour time frame, could we simply restore from one of those clones= if we get into a corrupted state?

Is there another mechanism for snapshotting / backing u= p data in Accumulo?

Thanks for your thoughts!

Mike


--047d7b677f760853c104de08a23e--