hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: HBASE-2312 discussion
Date Wed, 17 Mar 2010 01:08:47 GMT
On Tue, Mar 16, 2010 at 6:04 PM, Karthik Ranganathan <
kranganathan@facebook.com> wrote:

> @Stack: With the overwrite=false, I think option #2 looks fine.
>
> @Todd: By "bound number of log files", I meant that the if the latest log
> file is log.N, the master would try to open log.N+1, log.N+2, log.N+3 etc
> until one of it succeeds and the RS cannot open more log files after that
> one. So the master "bounds" the number of times the log file is opened.
>
> I am a little nervous about the master backing off on detecting the RS's
> progress - because the RS has already lost its zk lease. Not sure that if
> the master backs off, this will allow everything to proceed smoothly. But
> probably calling sync() on zk makes sense. Will think about this some more.
>
> I too like option #3 because it's a useful pattern, but it was initially
> much easier to reason about #2. Of course #1 is the easiest either way.
> Again, let me think about this more.
>
>
What do you think about the trick of making the RS do a ZK sync before any
meta op? This forces it to take at most one action after it's been
terminated.


> Thanks
> Karthik
>
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Tuesday, March 16, 2010 3:18 PM
> To: hbase-dev@hadoop.apache.org
> Subject: Re: HBASE-2312 discussion
>
> On Tue, Mar 16, 2010 at 1:49 PM, Stack <stack@duboce.net> wrote:
>
> > Karthik:
> >
> > Thanks for looking into this.
> >
> > Reading over the issue, you think option #2 "not clean" before Todd
> > proposes changing overwrite to false.  Do you still think it so?  If
> > not, then option #2 seems straight-forward.
> >
> > While option #3 is more code, its attractive in that its a pattern we
> > might take on to solve other filesystem transitions; e.g. recovering
> > failed compactions.  Do you think option #3 harder to verify?  The
> > 'chasing logs' would be hard to do up in tests.
> >
>
> I think the "chasing logs" thing is actually avoidable pretty easily. I
> commented on HBASE-2312 with thoughts there.
>
> Regarding option 1, I'm not entirely against the new HDFS API, so if others
> think it's a good solution we may as well go with it (we're already
> requiring patched HDFS for sync, so another simple patch isn't a huge
> deal).
>
> Regarding option 2, not sure what you mean be "The number of log files the
> RS can create will be bound." -- can you explain?
>
> Stack's point that #3 is a useful pattern for lots of transitions seems
> very
> valid to me as well.
>
>
> >
> > Thanks,
> > St.Ack
> >
> > P.S. Tsuna, up https://issues.apache.org/jira/browse/HBASE-2238 there
> > is some discussion of why hdfs state changes has to be managed in the
> > filesystem only, of how state can't bridge filesystem and zookeeper.
> >
> >
> > On Tue, Mar 16, 2010 at 11:13 AM, Karthik Ranganathan
> > <kranganathan@facebook.com> wrote:
> > > Hey guys,
> > >
> > > Just wanted to close on which solution we wanted to pick for this issue
> -
> > I was thinking about working on this one. There are 3 possibilities here.
> I
> > have briefly written up the issue and the three solutions below.
> > >
> > > Issue:
> > > There is a very corner case when bad things could happen(ie data loss):
> > > 1) RS #1 is going to roll its HLog - not yet created the new one, old
> one
> > will get no more writes
> > > 2) RS #1 enters GC Pause of Death
> > > 3) Master lists HLog files of RS#1 that is has to split as RS#1 is
> dead,
> > starts splitting
> > > 4) RS #1 wakes up, created the new HLog (previous one was rolled) and
> > appends an edit - which is lost
> > >
> > > Solution 1:
> > > 1) Master detects RS#1 is dead
> > > 2) The master renames the /hbase/.logs/<regionserver name> directory
to
> > something else (say /hbase/.logs/<regionserver name>-dead)
> > > 3) Add mkdir support (as opposed to mkdirs) to HDFS - so that a file
> > create fails if the directory doesn't exist. Dhruba tells me this is very
> > doable.
> > > 4) RS#1 comes back up and is not able create the new hlog. It restarts
> > itself.
> > > NOTE: Need another HDFS API to be supported, Todd wants to avoid this.
> > This API exists in Hadoop 0.21, but is not back-ported to 0.20.
> > >
> > > Solution 2:
> > > 1) RS #1 has written log.1, log.2, log.3
> > > 2) RS #1 is just about to write log.4 and enters gc pause before doing
> so
> > > 3) Master detects RS #1 dead
> > > 4) Master sees log.1, log.2, log.3. It then opens log.3 for append and
> > also creates log.4 as a lock
> > > 5) RS #1 wakes up and isn't allowed to write to either log.3 or log.4
> > since HMaster holds both.
> > > NOTE:  This changes the log file names, changes the create mode of the
> > log files from overwrite = true to false. Master needs to create the last
> > log file and open it in append mode to prevent RS from proceeding. RS
> will
> > fail if it cannot create the next log file. The number of log files the
> RS
> > can create will be bound.
> > >
> > > Solution 3:
> > > 1) Write "intend to roll HLog to new file hlog.N+1" to hlog.N
> > > 2) Open hlog.N+1 for append
> > > 3) Write "finished rolling" to hlog.N
> > > 4) continue writing to hlog.N+1
> > > NOTE: This requires new types edits to go into the log file - "intent
> to
> > roll" and "finished roll". Master has to open the last log file for
> append
> > first. Also, master has to "chase" log files created by the region server
> > (please see the issue for details) as there is an outside chance of log
> > files rolling when the GC pause happens.
> > >
> > > In my opinion, from the perspective of code simplicity, I would rank
> the
> > solutions as 1 being simplest, then 2, then 3. Since 1 needs another HDFS
> > API, I was thinking that 2 seemed simpler to do and easier to verify
> > correctness.
> > >
> > > What are your thoughts?
> > >
> > > Thanks
> > > Karthik
> > >
> > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message