Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of varun@pinterest.com designates
 209.85.210.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGpTDNcVae_dtsFwzJKQSbm6v4VuhmSRYD6uSeYpiwN87S6O1Q@mail.gmail.com>
References: 
 <CAKxWWm2MAVspehfO-9UU3yA+d+JBKvAv_P3eAsdpBUzzGB37ng@mail.gmail.com>
	<1369021773.11235.YahooMailNeo@web140605.mail.bf1.yahoo.com>
	<CAKxWWm1dD1APaHaAQmDqH=nvbErjqdMaG0ewYze2z5y=cXp6Zw@mail.gmail.com>
	<CAGpTDNcVae_dtsFwzJKQSbm6v4VuhmSRYD6uSeYpiwN87S6O1Q@mail.gmail.com>
Date: Mon, 20 May 2013 15:48:34 -0700
Message-ID: 
 <CAKxWWm0BFUzKUGxYj771ymszWBvRZv6H2WJMkzrdTNVhfSzMEQ@mail.gmail.com>
Subject: Re: Questions about HBase replication
From: Varun Sharma <varun@pinterest.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae9340ff5f37eab04dd2e210a

--14dae9340ff5f37eab04dd2e210a
Content-Type: text/plain; charset=ISO-8859-1

Thanks JD for the response... I was just wondering if issues have ever been
seen with regards to moving over a large number of WAL(s) entirely from one
region server to another since that would double the replication related
load on the one server which takes over.

Another side question: After the WAL has been replicated - is it purged
immediately or soonish from the zookeeper ?

Thanks
Varun


On Mon, May 20, 2013 at 9:57 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> On Mon, May 20, 2013 at 12:35 AM, Varun Sharma <varun@pinterest.com>
> wrote:
> > Hi Lars,
> >
> > Thanks for the response.
> >
> > Regarding #2 again, so if RS1 failed, then the following happens...
> > 1) RS2 takes over its logs...
> > 2) Master renames the log containing directory to have a -splitting in
> the
> > path
> > 3) Does RS2 already know about the "-splitting" path ?
>
> It will look at all the possible locations. See
> ReplicationSource.openReader
>
> >
> > Also on a related note, was there a reason that we have all region
> servers
> > watching all other region server's queue of logs. Otherwise, couldn't the
> > master have done the reassignment of outstanding logs to other region
> > servers more fairly upon failure ?
>
> I think I did it like that because it was easier since the region
> server has to be told to grab the queue(s) anyway.
>
> >
> > Thanks
> > Varun
> >
> >
> > On Sun, May 19, 2013 at 8:49 PM, lars hofhansl <larsh@apache.org> wrote:
> >
> >> #1 yes
> >> #2 no
> >>
> >> :)
> >>
> >> Now, there are scenarios where inconsistencies can happen. The edits are
> >> not necessarily shipped in order when there are failures.
> >> So it is possible to have some Puts at T1 and some Deletes at T2 (T1 <
> >> T2), and end up with the deletes shipped first.
> >> Now imagine a compaction happens at the slave after the Deletes are
> >> shipped to the slave, but before the Puts are shipped... The Puts will
> >> reappear.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Varun Sharma <varun@pinterest.com>
> >> To: user@hbase.apache.org
> >> Sent: Sunday, May 19, 2013 12:13 PM
> >> Subject: Questions about HBase replication
> >>
> >>
> >> Hi,
> >>
> >> I have a couple of questions about HBase replication...
> >>
> >> 1) When we ship edits to slave cluster - do we retain the timestamps in
> the
> >> edits - if we don't, I can imagine hitting some inconsistencies ?
> >>
> >> 2) When a region server fails, the master renames the directory
> containing
> >> WAL(s). Does this impact reading of those logs for replication ?
> >>
> >> Thanks
> >> Varun
> >>
>

--14dae9340ff5f37eab04dd2e210a--