hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Shine <Dave.Sh...@channelintelligence.com>
Subject RE: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 12:36:32 GMT
I am not a contributor to this project, so I don't know how much weight my opinion carries.
 But I have been hoping to see append become stable soon.  We are constantly dealing with
the "small file problem", and I have written M/R jobs to periodically roll up lots of small
files into a few small ones.  Having append would prevent me from needing to use up cluster
resources performing these tasks.

Therefore, all things being equal I +1 making append work.  However, if the level of complexity
is as bad as Eli implies below, then I can understand that perhaps it is not worth the effort.
If it will cause too much technical debt, then removing it makes sense.  But don't just remove
it because you don't believe there is a need for it.

Thanks,
Dave Shine


-----Original Message-----
From: Eli Collins [mailto:eli@cloudera.com]
Sent: Tuesday, March 20, 2012 8:38 PM
To: hdfs-dev@hadoop.apache.org
Subject: [DISCUSS] Remove append?

Hey gang,

I'd like to get people's thoughts on the following proposal. I think we should consider removing
append from HDFS.

Where we are today.. append was added in the 0.17-19 releases
(HADOOP-1700) and subsequently disabled (HADOOP-5224) due to quality issues. It and sync were
re-designed, re-implemented, and shipped in
21.0 (HDFS-265). To my knowledge, there has been no real production use. Anecdotally people
who worked on branch-20-append have told me they think the new trunk code is substantially
less well-tested than the branch-20-append code (at least for sync, append was never well
tested). It has certainly gotten way less pounding from HBase users.
The design however, is much improved, and people think we can get hsync (and append) stabilized
in trunk (mostly testing and bug fixing).

Rationale follows..

Append does not seem to be an important requirement, hflush was. There has not been much demand
for append, from users or downstream projects. Because Hadoop 1.x does not have a working
append implementation (see HDFS-3120, the branch-20-append work was focused on sync not getting
append working) which is not enabled by default and downstream projects will want to support
Hadoop 1.x releases for years, most will not introduce dependencies on append anyway. This
is not to say demand does not exist, just that if it does, it's been much smaller than security,
sync, HA, backwards compatbile RPC, etc. This probably explains why, over 5 years after the
original implementation started, we don't have a stable release with append.

Append introduces non-trivial design and code complexity, which is not worth the cost if we
don't have real users. Removing append means we have the property that HDFS blocks, when finalized,
are immutable.
This significantly simplifies the design and code, which significantly simplifies the implementation
of other features like snapshots, HDFS-level caching, dedupe, etc.

The vast majority of the HDFS-265 effort is still leveraged w/o append. The new data durability
and read consistency behavior was the key part.

GFS, which HDFS' design is based on, has append (and atomic record
append) so obviously a workable design does not preclude append.
However we also should not ape the GFS feature set simply because it exists. I've had conversations
with people who worked on GFS that regret adding record append (see also http://queue.acm.org/detail.cfm?id=1594206).
In short, unless append is a real priority for our users I think we should focus our energy
elsewhere.

Thanks,
Eli

The information contained in this email message is considered confidential and proprietary
to the sender and is intended solely for review and use by the named recipient. Any unauthorized
review, use or distribution is strictly prohibited. If you have received this message in error,
please advise the sender by reply email and delete the message.

Mime
View raw message