hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wukang Lin <vboylin1...@gmail.com>
Subject Re: Some questions on HLog
Date Thu, 24 Oct 2013 17:30:15 GMT
Thank you, Ted. I got it, :-).
I read the comments on HDFS-744 and HBASE-5954, it seems HDFS support true
fsync and enable by default since hadoop 2.0.0 alpha, hbase has the ablity
to configure how WAL and Hfile use HDFS's fsync until 0.98, so, for our
version of HBase(0.94.6+hadoop 2.0.0), we will use fsync by default, and
data will not loss in above scenarios,  am i right?


2013/10/25 Ted Yu <yuzhihong@gmail.com>

> This was due to the fact that when HBASE-2312 was integrated, there were
> many flavors of hadoop running in production.
> So the code had to support all the flavors.
>
> Cheers
>
>
> On Thu, Oct 24, 2013 at 9:27 AM, Wukang Lin <vboylin1987@gmail.com> wrote:
>
> > Hi Ted,
> >     Thank you for your response. for #1, I have tried to understand that
> > comment in SequenceFileLogWriter, i can't figure out that instead of
> > reflection, why not use the version of SF.createWriter below directly?
> >     SequenceFile.Writer createWriter(FileSystem fs,
> >                                                      Configuration conf,
> >                                                      Path name,
> >                                                      Class keyClass,
> >                                                      Class valClass,
> >                                                      int bufferSize,
> >                                                      short replication,
> >                                                      long blockSize,
> >                                                      boolean
> createParent,
> >                                                      CompressionType
> > compressionType,
> >                                                      CompressionCodec
> > codec,
> >                                                      Metadata metadata)
> > throws java.io.IOException;
> >
> >     Thank you again.
> >
> >
> >
> > 2013/10/24 Ted Yu <yuzhihong@gmail.com>
> >
> > > For #2, see HBASE-5954
> > >
> > > For #1, see the following comment in SequenceFileLogWriter:
> > >
> > > +      // reflection for a version of SequenceFile.createWriter that
> > > doesn't
> > > +      // automatically create the parent directory (see HBASE-2312)
> > > +      this.writer = (SequenceFile.Writer) SequenceFile.class
> > >
> > >
> > >
> > > On Thu, Oct 24, 2013 at 8:49 AM, Wukang Lin <vboylin1987@gmail.com>
> > wrote:
> > >
> > > > Hi all,
> > > >     Recently, i read the source of HBase's HLog, and i got some
> > questions
> > > > that puzzled me a lot. here there are:
> > > >     1 why use reflection to init a SequenceFile.Writer
> > > > in SequenceFileLogWriter? i read HBASE-2312 but still can't catch the
> > > > point.
> > > >      2 It seems that hlog use SequenceFile.Writer's append method to
> > sync
> > > > the WAL logs to DataNode, not FSDataOutputStream.hflush(), for each
> > > > mutation(or batch mutations), so may it lose data when HDFS crash
> while
> > > WAL
> > > > logs were 'sync' to DataNode but not flush to disk? or are there
> > > something
> > > > i misunderstanded?
> > > >
> > > >       Thank you.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message