hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jingguo yao <yaojing...@gmail.com>
Subject Re: Does Hadoop 1.0.4 provide a durable sync for HBase-0.94.6?
Date Wed, 29 May 2013 01:54:27 GMT
Stack & Enis:

Your responses clear my thoughts. Thanks.


On Wed, May 29, 2013 at 6:19 AM, Enis Söztutar <enis.soz@gmail.com> wrote:

> Hi,
>
> HDFS has two interfaces for durability: hflush and hsync:
>
> Hflush() : Flush the data packet down the datanode pipeline. Wait for
> ack’s.
> Hsync() : Flush the data packet down the pipeline. Have datanodes execute
> FSYNC equivalent. Wait for ack’s.
>
> There is some work on adding a Durability API in HBase: see HBASE-7801 and
> HBASE-8375.
>
> However, as Stack mentioned, without HBASE-5954 is fixed, HBase right now
> cannot make use of the hsync() API. I want to rebase the patch in
> HBASE-5954, but it might take some more time.
>
> The good news is that although not perfect, hflush, which is current
> default makes sure that the update is send to 3 replicas, so unless there
> is a data center power failure or similar, the data will make into the
> disks pretty quickly.
>
> Hope this helps.
> Enis
>
>
>
>
> On Tue, May 28, 2013 at 9:53 AM, Stack <stack@duboce.net> wrote:
>
> > On Tue, May 28, 2013 at 7:09 AM, jingguo yao <yaojingguo@gmail.com>
> wrote:
> >
> > > Section 2.1.3 says that Hadoop 1.0.4 works with HBase-0.94.x [1]. And
> > > Section 2.1.3.3 says that 1.0.4 has a working durable sync. But when I
> > > check the source code of DFSClient.DFSOutputStream's sync method, I
> > > finds the following javadoc:
> > >
> > >     /**
> > >      * All data is written out to datanodes. It is not guaranteed
> > >      * that data has been flushed to persistent store on the
> > >      * datanode. Block allocations are persisted on namenode.
> > >      */
> > >
> > > So it seems that sync does not support a durable sync. It contradicts
> > > with [1].
> > >
> > > Can anybody help me on this confusion? Thanks.
> >
> >
> >
> > This issue is probably the best source for the state of sync in hbase
> (and
> > hdfs): https://issues.apache.org/jira/browse/HBASE-5954
> >
> > In short, the refguide is misleading -- let me fix -- as 1.0.4 indeed
> has a
> > sync but it is just a sync to the memory of three datanodes, not a true
> > fsync out to disk.  The above cited issue is tracking issues that our
> Lars
> > and other have contributed to HDFS to add fsync support.
> >
> > Yours,
> > St.Ack
> >
>



-- 
Jingguo

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message