Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 225269C98 for ; Wed, 21 Mar 2012 20:25:59 +0000 (UTC) Received: (qmail 77666 invoked by uid 500); 21 Mar 2012 20:25:58 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 77574 invoked by uid 500); 21 Mar 2012 20:25:58 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 77566 invoked by uid 99); 21 Mar 2012 20:25:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Mar 2012 20:25:58 +0000 X-ASF-Spam-Status: No, hits=-5.0 required=5.0 tests=RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Milind.Bhandarkar@emc.com designates 128.222.32.20 as permitted sender) Received: from [128.222.32.20] (HELO mexforward.lss.emc.com) (128.222.32.20) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Mar 2012 20:25:53 +0000 Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q2LKPSWe026864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 21 Mar 2012 16:25:31 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.130]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Wed, 21 Mar 2012 16:25:17 -0400 Received: from mxhub33.corp.emc.com (mxhub33.corp.emc.com [10.254.93.81]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id q2LKPGMn031557 for ; Wed, 21 Mar 2012 16:25:16 -0400 Received: from mx21a.corp.emc.com ([169.254.1.36]) by mxhub33.corp.emc.com ([::1]) with mapi; Wed, 21 Mar 2012 16:25:16 -0400 From: To: Date: Wed, 21 Mar 2012 16:24:10 -0400 Subject: Re: [DISCUSS] Remove append? Thread-Topic: [DISCUSS] Remove append? Thread-Index: Ac0HoLnm+ug282ejTdu5SbVHrCbYYw== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 X-Virus-Checked: Checked by ClamAV on apache.org I would also like to point to work being done on PLFS-HDFS: http://institute.lanl.gov/isti/irhpit/presentations/PLFS-HDFS.pdf This would be made much simpler by allowing appends. Checkpointing in MPI is a very common use-case, and after Hamster, PLFS-HDFS becomes an attractive way to do this. (Section 2 of the 2009 HotCloud paper by PDL: http://www.cs.cmu.edu/~svp/2009hotcloud-tablefs.pdf discusses the reasons for seeking commonalities between HPC and DISC file systems.) - Milind On 3/21/12 12:48 PM, "Bhandarkar, Milind" wrote: >Eli, > >To clarify a little bit, I think HDFS-3120 is the right thing to do, to >disable appends, while still enabling hsync in branch-1. > >But, going forward, (say 0.23+) having appends working correctly will >definitely add value, and make HDFS more palatable for lots of other >workloads. > >Of course, I have a vested interest in this, because our team is working >on a project that requires append and truncate, and we will be testing it >thoroughly at scale in Q2 this year. Would it be okay to wait for the >results of this testing ? > >Thanks, > >- milind > >--- >Milind Bhandarkar >Greenplum Labs, EMC >(Disclaimer: Opinions expressed in this email are those of the author, and >do not necessarily represent the views of any organization, past or >present, the author might be affiliated with.) > >