hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 20:58:55 GMT
On Wed, Mar 21, 2012 at 1:31 PM, Tsz Wo Sze <szetszwo@yahoo.com> wrote:
>
> Some of the information in the email is not correct.  Let me clarify them.
>
>> Where we are today.. append was added in the 0.17-19
> releases
>> (HADOOP-1700) . . .
>
> We never have append/sync in 0.17.  Sync was added to 0.18 but not append.  Append
was added to 0.19.  By append/sync above, I mean the
> implementation by HADOOP-1700.  We also
> have HDFS-265, the new append/hflush.  Below are the details.
>
> Versions         Features
> <= 0.17:          no sync/append
> 0.18:               1700
> sync
> 0.19.0:             1700
> append
> 0.19.1, 0.20:   1700 append disabled
> 0.20-append:append branch used by facebook
> 0.20.205.0:     merged 1700 append to 0.20
>>= 0.21:          265 append/hflush
>

Thanks for fleshing out the specifics, I put "17-19" to indicate that
parts went in over a series of releases.

>> . . . To my knowledge, there has
> been no real production use. . .
>
> The reason of no production use today
> is simply that append is not yet in a stable release.  Besides, it does not mean append
is not
> useful.
>

Agree, not saying it isn't useful.   "usefulness" is necessary but not
sufficient. There are plenty of useful things we may not want to put
in HDFS.

>> . . . The design however, is much
> improved, and people think we can get
>> hsync (and append) stabilized in
> trunk (mostly testing and bug fixing).
>
> hsync is not yet implemented.  I think you may mean hflush.
>

Yup, good catch,  I meant hflush. (For those following along hsync is
implemented, just not according to the design since today it just
calls hflush).

>> . . . This probably explains why,
> over 5 years after the original implementation
>> started, we don't have a stable
> release with append.
>
> HADOOP-1700 was committed on July 25,
> 2008.  I don’t know how it could be “over
> 5 years”.  It is well known that append
> from 0.20.x releases is not stable and hence probably not used.  It is not the case
that we don’t have a
> stable release because append is not stable.
>
>> Append introduces non-trivial
> design and code complexity, which is not
>> worth the cost if we don't have
> real users. . . .
>
> I don’t agree.  The non-trivial design and code complexity
> come from hflush but not append.  Once we
> have hflush, append is straightforward.  Roughly speaking, the append work is about
10% of the entire
> append/hflush work.

Do you think having the invariant that blocks are not mutated would
significantly simply the design?

Thanks,
Eli

>
> Moreover, there are real users/use
> cases as mentioned by Dave and Milind.
>
> The jira that you have created to split
> the flag into hflush supported and append supported is a good idea. Folks who
> do not need append, but need hflush, can still disable append.
>
> Regards,
> Nicholas
>
>
>
> ________________________________
>  From: Eli Collins <eli@cloudera.com>
> To: hdfs-dev@hadoop.apache.org
> Sent: Tuesday, March 20, 2012 5:37 PM
> Subject: [DISCUSS] Remove append?
>
> Hey gang,
>
> I'd like to get people's thoughts on the following proposal. I think
> we should consider removing append from HDFS.
>
> Where we are today.. append was added in the 0.17-19 releases
> (HADOOP-1700) and subsequently disabled (HADOOP-5224) due to quality
> issues. It and sync were re-designed, re-implemented, and shipped in
> 21.0 (HDFS-265). To my knowledge, there has been no real production
> use. Anecdotally people who worked on branch-20-append have told me
> they think the new trunk code is substantially less well-tested than
> the branch-20-append code (at least for sync, append was never well
> tested). It has certainly gotten way less pounding from HBase users.
> The design however, is much improved, and people think we can get
> hsync (and append) stabilized in trunk (mostly testing and bug
> fixing).
>
> Rationale follows..
>
> Append does not seem to be an important requirement, hflush was. There
> has not been much demand for append, from users or downstream
> projects. Because Hadoop 1.x does not have a working append
> implementation (see HDFS-3120, the branch-20-append work was focused
> on sync not getting append working) which is not enabled by default
> and downstream projects will want to support Hadoop 1.x releases for
> years, most will not introduce dependencies on append anyway. This is
> not to say demand does not exist, just that if it does, it's been much
> smaller than security, sync, HA, backwards compatbile RPC, etc. This
> probably explains why, over 5 years after the original implementation
> started, we don't have a stable release with append.
>
> Append introduces non-trivial design and code complexity, which is not
> worth the cost if we don't have real users. Removing append means we
> have the property that HDFS blocks, when finalized, are immutable.
> This significantly simplifies the design and code, which significantly
> simplifies the implementation of other features like snapshots,
> HDFS-level caching, dedupe, etc.
>
> The vast majority of the HDFS-265 effort is still leveraged w/o
> append. The new data durability and read consistency behavior was the
> key part.
>
> GFS, which HDFS' design is based on, has append (and atomic record
> append) so obviously a workable design does not preclude append.
> However we also should not ape the GFS feature set simply because it
> exists. I've had conversations with people who worked on GFS that
> regret adding record append (see also
> http://queue.acm.org/detail.cfm?id=1594206). In short, unless append
> is a real priority for our users I think we should focus our energy
> elsewhere.
>
> Thanks,
> Eli

Mime
View raw message