Return-Path: Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: (qmail 70777 invoked from network); 22 Jun 2010 05:04:19 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 22 Jun 2010 05:04:19 -0000 Received: (qmail 90668 invoked by uid 500); 22 Jun 2010 05:04:19 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 90471 invoked by uid 500); 22 Jun 2010 05:04:17 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 90463 invoked by uid 99); 22 Jun 2010 05:04:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:04:16 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vidur@students.iiit.ac.in designates 121.242.23.201 as permitted sender) Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:04:10 +0000 MailScanner-NULL-Check: 1277787817.59699@9BvoJ5/t8JOtJXnr5FhRJA Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1]) by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id o5M53bl3023006 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 22 Jun 2010 10:33:37 +0530 Received: (from apache@localhost) by students.iiit.ac.in (8.13.8/8.14.1/Submit) id o5M53bJV023002; Tue, 22 Jun 2010 10:33:37 +0530 X-Authentication-Warning: students.iiit.ac.in: apache set sender to vidur@localhost using -f Received: from 125.16.17.151 (proxying for unknown) (SquirrelMail authenticated user vidur) by students.iiit.ac.in with HTTP; Tue, 22 Jun 2010 10:33:37 +0530 (IST) Message-ID: <26917.125.16.17.151.1277183017.squirrel@students.iiit.ac.in> In-Reply-To: References: <29430.125.16.17.151.1277181711.squirrel@students.iiit.ac.in> Date: Tue, 22 Jun 2010 10:33:37 +0530 (IST) Subject: Re: Overwriting the same block instead of creating a new one From: "Vidur Goyal" To: hdfs-dev@hadoop.apache.org User-Agent: SquirrelMail/1.4.8-4.el5 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in X-yoursite-MailScanner-Information: Please contact the IIIT Server Room for more information X-MailScanner-ID: o5M53bl3023006 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: vidur@students.iiit.ac.in X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX autolearn=no version=3.2.5, No Dear Todd, By truncating i meant removing unused *blocks* from the namespace and let them be garbage collected. There will be no truncation of the last block(even if it is not full). This way , rather then garbage collecting all the blocks of a file , we will only be garbage collecting the remaining blocks. -vidur > HDFS assumes in hundreds of places that blocks never shrink. So, there is > no > option to truncate a block. > > -Todd > > On Mon, Jun 21, 2010 at 9:41 PM, Vidur Goyal > wrote: > >> Hi All, >> >> In FSNamesystem#startFileInternal , whenever there is a overwrite flag >> set >> , why is the INode removed from the namespace and a new >> INodeFileUnderConstruction is created. Why can't we use the convert the >> same INode to INodeFileUnderConstruction. And we start writing to the >> same >> blocks at the same datanodes (after incrementing the GS) followed by >> either truncating the remaining blocks(if the file size decreases) or >> allocating new blocks (if the file size increases). This will decrease >> data redundancy and the job of garbage collector and will increase >> security. >> >> vidur >> >> >> >> >> -- >> This message has been scanned for viruses and >> dangerous content by MailScanner, and is >> believed to be clean. >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.