hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Milind.Bhandar...@emc.com>
Subject Re: [DISCUSS] Remove append?
Date Wed, 21 Mar 2012 20:24:10 GMT
I would also like to point to work being done on PLFS-HDFS:
http://institute.lanl.gov/isti/irhpit/presentations/PLFS-HDFS.pdf

This would be made much simpler by allowing appends.

Checkpointing in MPI is a very common use-case, and after Hamster,
PLFS-HDFS becomes an attractive way to do this.

(Section 2 of the 2009 HotCloud paper by PDL:
http://www.cs.cmu.edu/~svp/2009hotcloud-tablefs.pdf discusses the reasons
for seeking commonalities between HPC and DISC file systems.)

- Milind


On 3/21/12 12:48 PM, "Bhandarkar, Milind" <Milind.Bhandarkar@emc.com>
wrote:

>Eli,
>
>To clarify a little bit, I think HDFS-3120 is the right thing to do, to
>disable appends, while still enabling hsync in branch-1.
>
>But, going forward, (say 0.23+) having appends working correctly will
>definitely add value, and make HDFS more palatable for lots of other
>workloads.
>
>Of course, I have a vested interest in this, because our team is working
>on a project that requires append and truncate, and we will be testing it
>thoroughly at scale in Q2 this year. Would it be okay to wait for the
>results of this testing ?
>
>Thanks,
>
>- milind
>
>---
>Milind Bhandarkar
>Greenplum Labs, EMC
>(Disclaimer: Opinions expressed in this email are those of the author, and
>do not necessarily represent the views of any organization, past or
>present, the author might be affiliated with.)
>
>


Mime
View raw message