Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of tychang@gmail.com designates
 209.85.213.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHpPDZ5HOqcX0jgLAjzuo9Mft+NVbQMyOhBAv4kPU1RpMoq0JQ@mail.gmail.com>
References: 
 <CAHpPDZ5HOqcX0jgLAjzuo9Mft+NVbQMyOhBAv4kPU1RpMoq0JQ@mail.gmail.com>
Date: Tue, 25 Mar 2014 14:38:01 -0700
Message-ID: 
 <CAHpPDZ7TdaxDFAU+rEUAwzoEyeqRXcqytQ6355KbpT_OqxkG6w@mail.gmail.com>
Subject: Re: no-flush based snapshot policy?
From: Tianying Chang <tychang@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf300e4cf19deab604f5752abe

--20cf300e4cf19deab604f5752abe
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I need a new snapshot policy. Basically, I cannot disable the table, but I
also don't need the snapshot to be that "consistent" where all RS
coordinated to flush the region before taking the snapshot since it slow
down production cluster when flush take too long. It is OK for me if the
snapshot missed the data in memstore because I will use WALPlayer to fill
the data gap that is not in the snapshot but has been persisted (in WAL).
So I should have no data loss.

As a quick hack way to test this in my hbase backup workflow, I just add a
config key, and skip the flushcache() in file
*regionserver/snapshot/FlushSnapshotSubprocedure.java*, something like
below.  It seems works fine for me, where all data are recovered in a new
cluster after running WALPlayer.

Does anyone see any problem like data corruption, etc?


LOG.debug("Flush Snapshotting region " + region.toString() + " started...");
if (noFlushNeeded)
{
   LOG.debug("No flush before taking snapshot");
} else
{
    region.flushcache();
}

If there is no data corruption issue with this policy, I can add an
parameter from hbase shell, so that people can dynamically decide when to
use no-flush snapshot.

Thanks
Tian-Ying

On Tue, Mar 25, 2014 at 2:08 PM, Tianying Chang <tychang@gmail.com> wrote:

> Hi,
>
> I need a new snapshot policy which sits in between the disabled and
> flushed version. So, basically:
> I cannot disable the table, but I also don't need the snapshot to be that
> "consistent" where all RS coordinated to flush the region before taking the
> snapshot.
>

--20cf300e4cf19deab604f5752abe--