hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitchcock, Andrew" <a...@amazon.com>
Subject Re: concurrent write [ is it defined and consistent ? ]
Date Wed, 02 Dec 2009 00:48:15 GMT
This interview might be interesting to folks following this discussion:


It talks about some of the design decisions behind GFS. I recommend the whole article, but
here is a snippet from the section on append:

QUINLAN At the time, [RecordAppend] must have seemed like a good idea, but in retrospect I
think the consensus is that it proved to be more painful than it was worth. It just doesn't
meet the expectations people have of a file system, so they end up getting surprised. Then
they had to figure out work-arounds.

MCKUSICK In retrospect, how would you handle this differently?

QUINLAN I think it makes more sense to have a single writer per file.

MCKUSICK All right, but what happens when you have multiple people wanting to append to a

QUINLAN You serialize the writes through a single process that can ensure the replicas are


On 12/1/09 8:02 AM, "Owen O'Malley" <omalley@apache.org> wrote:

On Dec 1, 2009, at 6:30 AM, Brian Bockelman wrote:

> The upcoming 0.21.x release will be the first to support appends
> after the file is closed (still single-writer though).  I'd
> speculate that this lays out some ground work that is necessary for
> multi-writer atomic appends, but I don't even know if atomic appends
> are even on the roadmap.

The short answer is that I don't think anyone has even thought deeply
about it. I haven't heard anyone talking about doing it anytime soon.

My personal inclination is that atomic append does very bad things to
both the design of the file system and the user interface to the file
system. Clearly they added atomic append to GFS before they had
MapReduce. It seems like most applications would be better served by
implementing in MapReduce rather than using atomic append anyways...

-- Owen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message