Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
MIME-Version: 1.0
References: <CAL5zq9bxqvMVU-nFpe1mzVo7u4rT4X3BG862URzab5OQGXKZFA@mail.gmail.com>
 <CAFAMeYLYmPV1QSYWdCA-ncoCzLaQc1mOTSU8=p6A0TOaxEzVGA@mail.gmail.com>
In-Reply-To: <CAFAMeYLYmPV1QSYWdCA-ncoCzLaQc1mOTSU8=p6A0TOaxEzVGA@mail.gmail.com>
From: Christopher <ctubbsii@apache.org>
Date: Fri, 14 Jul 2017 18:07:58 +0000
Message-ID: <CAL5zq9bSMvo5hd0obcou=H_e6wvKxTL8V1xVpFnVfPjE0xKywg@mail.gmail.com>
Subject: Re: [DISCUSS] Periodic table exports
To: Accumulo Dev List <dev@accumulo.apache.org>
Content-Type: multipart/alternative; boundary="001a11438746811e7b05544aee76"
archived-at: Fri, 14 Jul 2017 18:08:12 -0000

--001a11438746811e7b05544aee76
Content-Type: text/plain; charset="UTF-8"

The problem is HDFS corrupt blocks which affect the metadata tables. I
don't know that this window is all that narrow. I've seen corrupt blocks
far more often than HDFS outages. Some due to HDFS bugs, some due to
hardware failures and too few replicas, etc. We know how to recover corrupt
blocks in user tables (accepting data loss) by essentially replacing a
corrupt file with an empty one. But, we don't really have a good way to
recover when the corrupt blocks occur in metadata tables. That's what this
would address.

On Fri, Jul 14, 2017 at 1:47 PM Mike Drob <mdrob@apache.org> wrote:

> What's the risk that we are trying to address?
>
> Storing data locally won't help in case of a namenode failure. If you have
> a failure that's severe enough to actually kill blocks but not severe
> enough that your HDFS is still up, that's a pretty narrow window.
>
> How do you test that your backups are good? That you haven't lost any data
> there? Or is it a set and forget (and pray?)
>
> This seems like something that is not worth while to automate because
> everybody is going to have such different needs. Write a blog post, then
> push people onto existing backup/disaster recovery solutions, including off
> site storage, etc. If they're not already convinced that they need this,
> then their data likely isn't that valuable to begin with. If this same
> problem happens multiple times to the same user... I don't think a periodic
> export table will help them.
>
> Mike
>
> On Fri, Jul 14, 2017 at 12:29 PM, Christopher <ctubbsii@apache.org> wrote:
>
> > I saw a user running a very old version of Accumulo run into a pretty
> > severe failure, where they lost an HDFS block containing part of their
> root
> > tablet. This, of course, will cause a ton of problems. Without the root
> > tablet, you can't recover the metadata table, and without that, you can't
> > recover your user tables.
> >
> > Now, you can recover the RFiles, of course... but without knowing the
> split
> > points, you can run into all sorts of problems trying to restore an
> > Accumulo instance from just these RFiles.
> >
> > We have an export table feature which creates a snapshot of the split
> > points for a table, allowing a user to relatively easily recover from a
> > serious failure, provided the RFiles are available. However, that
> requires
> > a user to manually run it on occasion, which of course does not happen by
> > default.
> >
> > I'm interested to know what people think about possibly doing something
> > like this internally on a regular basis. Maybe hourly by default,
> performed
> > by the Master for all user tables, and saved to a file in /accumulo on
> > HDFS?
> >
> > The closest think I can think of to this, which has saved me more than
> > once, is the way Chrome and Firefox backup open tabs and bookmarks
> > regularly, to restore from a crash.
> >
> > Users could already be doing this on their own, so it's not really
> > necessary to bake it in... but as we all probably know... people are
> really
> > bad at customizing away from defaults.
> >
> > What are some of the issues and trade-offs of incorporating this as a
> > default feature? What are some of the issues we'd have to address with
> it?
> > What would its configuration look like? Should it be on by default?
> >
> > Perhaps a simple blog describing a custom user service running alongside
> > Accumulo which periodically runs "export table" would suffice? (this is
> > what I'm leaning towards, but the idea of making it default is
> compelling,
> > given the number of times I've seen users struggle to plan for or respond
> > to catastrophic failures, especially at the storage layer).
> >
>

--001a11438746811e7b05544aee76--