hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: [DISCUSS] Remove append?
Date Thu, 22 Mar 2012 17:47:14 GMT
On Thu, Mar 22, 2012 at 10:15 AM, Daryn Sharp <daryn@yahoo-inc.com> wrote:
> On Mar 20, 2012, at 7:37 PM, Eli Collins wrote:
>> Hey gang,
>> I'd like to get people's thoughts on the following proposal. I think
>> we should consider removing append from HDFS.
>> Where we are today.. append was added in the 0.17-19 releases
>> (HADOOP-1700) and subsequently disabled (HADOOP-5224) due to quality
>> issues. It and sync were re-designed, re-implemented, and shipped in
>> 21.0 (HDFS-265). To my knowledge, there has been no real production
>> use. Anecdotally people who worked on branch-20-append have told me
>> they think the new trunk code is substantially less well-tested than
>> the branch-20-append code (at least for sync, append was never well
>> tested). It has certainly gotten way less pounding from HBase users.
>> The design however, is much improved, and people think we can get
>> hsync (and append) stabilized in trunk (mostly testing and bug
>> fixing).
> Up front:  I think append is a needed feature.

Can you elaborate.. eg are there particular use cases at Yahoo!  that
have been running for years that are itching to start using append
when 0.23 is deployed?  Are you guys testing the new append
implementation extensively because you have an app that's ready to use
it when 0.23 is deployed?

So far Milind has been the only one to chime in saying "we really need
append, here's why". Which is great.

> Politely speaking, I think the premise of the question is a bit dubious due to circular
nature.  Ie. It's not used in production so is it worth it?

No, I'm saying we've absorbed a lot of complexity for it, but I don't
see downstream projects using it any time soon.  Similarly there
hasn't been a big push to get it working, eg there was a push for
security and hbase support on 20, but not append (the append rewrite
was invasive but so was security). It will have been several years
from the time the rewrite was started until it gets deployed in
production, which makes me think it's less of a priority. So much so
that I wondered whether it was a big priority at all.

> The stigma/perception that append has been unstable and is not well-tested is a compelling
reason to not be in production at major installations.  The situation is going to be akin
to "You go first. No, you go first!  No way, you go first!".
> Downstream projects also aren't going to use something until it's stable, so they either
work around the limitation, or...  they chose something other hdfs.  There's also the unanswerable
question of how potential users have been silently lost.  We are unlikely to have heard the
user demand from those that chose another solution.  Generally for every complaint/request,
a large N-many people didn't even bother.
> I envision a day where hdfs is a performant posix filesystem.

I think that's unlikely. Posix compliance is a non-goal for HDFS. We
are intentionally not-compliant in many cases to achieve scale and
performance. Check out the Ceph file system paper (they manage the
tradeoff between Posix and scale/performance explicitly). The primary
motivation for Posix compliance is compatibility with existing
Unix-like software. That's not HDFS' raison d'etre (which is the
ecosystem of projects that run atop HDFS: MR, HBase, Pig, Hive, Flume,
Sqoop, etc etc).  HDFS' focus on it's core use case and simplicity is
one of the reasons it's been as successful as it has been.

That's not to say we don't need to do a lot more work to better
integrate HDFS with existing software and tools. Fuse-DFS is just a
start. We need to support a standard interface like NFS etc. These
efforts do not require HDFS become fully Posix compliant.

There's always a trade off between adding more features (increasing
the size of the addressable market) and focusing on your core uses
cases, quality, etc.  In my mind append is on the boundary. I'm happy
to be convinced that append is in HDFS' wheel house.


View raw message