hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. C. Srivas" <mcsri...@gmail.com>
Subject Re: Apparent data loss on 90.4 rc2 after partial zookeeper network partition (on MapR)
Date Fri, 05 Aug 2011 03:13:23 GMT
On Thu, Aug 4, 2011 at 10:34 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> > Thanks for the feedback.  So you're inclined to think it would be at the
> dfs
> > layer?
>
> That's where the evidence seems to point.
>
> >
> > Is it accurate to say the most likely places where the data could have
> been
> > lost were:
> > 1. wal writes didn't actually get written to disk (no log entries to
> suggest
> > any issues)
>
> Most likely.
>
> > 2. wal corrupted (no log entries suggest any trouble reading the log)
>
> In that case the logs would scream (and I didn't see that in the logs
> I looked at).
>
> > 3. not all split logs were read by regionservers  (?? is there any way to
> > ensure this either way... should I look at the filesystem some place?)
>
> Some regions would have recovered edits files, but that seems highly
> unlikely. With DEBUG enabled we could have seen which files were split
> by the master and which ones were created for the regions, and then
> which were read by the region servers.
>
> >
> > Do you think the type of network partition I'm talking about is
> adequately
> > covered in existing tests? (Specifically running an external zk cluster?)
>
> The IO fencing was only tested with HDFS, I don't know what happens in
> that case with MapR. What I mean is that when the master splits the
> logs, it takes ownership of the HDFS writer lease (only one per file)
> so that it can safely close the log file. Then after that it checks if
> there are any new log files that were created (the region server could
> have rolled a log while the master was splitting them) and will
> restart if that situation happens until it's able to own all files and
> split them.
>

JD,   I didn't think the master explicitly dealt with writer leases.

Does HBase rely on single-writer semantics on the log file? That is, if the
master and a RS both decide to mucky-muck with a log file, you expect the FS
to lock out one of the writers?




>
> >
> > Have you heard if anyone else is been having problems with the second
> 90.4
> > rc?
>
> Nope, we run it here on our dev cluster and didn't encounter any issue
> (with the code or node failure).
>
> >
> > Thanks again for your help.  I'm following up with the MapR guys as well.
>
> Good idea!
>
> J-D
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message