From general-return-1641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 16:53:50 2010 Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 88858 invoked from network); 15 Jun 2010 16:53:50 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 15 Jun 2010 16:53:50 -0000 Received: (qmail 97343 invoked by uid 500); 15 Jun 2010 16:53:49 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 97285 invoked by uid 500); 15 Jun 2010 16:53:48 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 97277 invoked by uid 99); 15 Jun 2010 16:53:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:53:48 +0000 X-ASF-Spam-Status: No, hits=0.9 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_FRT_STOCK1,T_FRT_STOCK2,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.193 as permitted sender) Received: from [209.85.211.193] (HELO mail-yw0-f193.google.com) (209.85.211.193) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:53:43 +0000 Received: by ywh31 with SMTP id 31so4033648ywh.20 for ; Tue, 15 Jun 2010 09:53:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=6Gw/O5PHeqZ861mQUl+7RT3EgJUGsBmMLU+Ca6WoP98=; b=TThyIL7MbAuKFQ/JjE1oYh5E7x84ZUjYy9VGS8tCm5qNi+CpwRG//N92rZt/1YqAza d8PrfeEVoTrVo5dHbnpoySHv6NZ4N0iPB9UxS0C9UJKj3cysAPaITwZw6dV8cpgpJdvK n9mqltiMdmbJSfs6vd5kuFy34a3L60km4F4UE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=JCYZc2hdc8FzFK5QG1J4KKDHMZrwfhbC+sURRNG3nOfwF/DmX+mIWdatYRw/BSxVLB 76YPO3XMbaHFRuV05s7S270wlTcfqJjliNeLJsorqoNaHk3XNaOtDvqjhIZlq5M0+H2a wgm5iIrmaOROLWjiqQKEt5zI2QFU055Dd//Sc= MIME-Version: 1.0 Received: by 10.150.2.1 with SMTP id 1mr9254803ybb.65.1276620802610; Tue, 15 Jun 2010 09:53:22 -0700 (PDT) Received: by 10.151.145.16 with HTTP; Tue, 15 Jun 2010 09:53:22 -0700 (PDT) In-Reply-To: References: Date: Tue, 15 Jun 2010 09:53:22 -0700 Message-ID: Subject: Re: HDFS file append From: Dhruba Borthakur To: general@hadoop.apache.org Content-Type: multipart/alternative; boundary=000e0cd4879074691204891471e2 --000e0cd4879074691204891471e2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You can start downloading and using the code form the hadoop-0.20-append. thanks, dhruba On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon wrote: > On Tue, Jun 15, 2010 at 5:31 AM, Jan St=F6cker > wrote: > > > Hello, > > > > > > > > in the hdfs-default.xml, I found the warning that the property > > > > "dfs.support.append" is false by default, because of the "append > > > > code" containing bugs. > > > > For our test case (which is not very complicated), I tried it out > > > > nevertheless (in a C program using libhdfs), and I did not > > > > encounter any problems yet. The only thing I noticed is that the > > > > modification date of the file to which I append is not changed. > > > > > > > > Could someone indicate which other bugs may occur? And in > > > > which cases? Because the append could be quite useful for me, > > > > and I would like to know the risk. > > > > Mostly in failure handling cases, but not entirely. The result is usually > truncated files, sometimes truncated as if your append didn't happen, > sometimes the entire last block going missing. Look for the 0.20-append f= ix > version on JIRA for a more significantly list. > > I'd recommend waiting a couple weeks and using the hadoop-0.20-append > branch > which will contain fixes for many of these bugs. > > -Todd > > > -- > Todd Lipcon > Software Engineer, Cloudera > --=20 Connect to me at http://www.facebook.com/dhruba --000e0cd4879074691204891471e2--