Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C4F3D10DD0 for ; Tue, 7 Jan 2014 14:07:17 +0000 (UTC) Received: (qmail 6784 invoked by uid 500); 7 Jan 2014 14:02:49 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 6597 invoked by uid 500); 7 Jan 2014 14:02:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 6391 invoked by uid 99); 7 Jan 2014 14:02:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jan 2014 14:02:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of smarty.juice@gmail.com designates 209.85.219.44 as permitted sender) Received: from [209.85.219.44] (HELO mail-oa0-f44.google.com) (209.85.219.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jan 2014 14:02:06 +0000 Received: by mail-oa0-f44.google.com with SMTP id h16so193556oag.3 for ; Tue, 07 Jan 2014 06:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=kSMRuBQWE5hjAiMbq5Ko39g/XWUsucV6DorP+v1kd+U=; b=WoFSU5VsSbrKiCd9/ItjDR309LdoSuFer0GCCJf+dYaZNkgG/whF+sn3eGG2d5dYmg KLQIa+Gi/RX0Lwkli0U/RYMDUJP0XfKyRdYJoG4R+qepT5uhC3Eg6X6Ai7r7MEKE6d0V VrqOjznh1K2+SSS130i50tSF9lQv2oXDrgv64gZbaIwkV87rhoRA3CuPkQ3HgpLRaaZv +uJ6+DzmLQDyofs3jruRTQD7CHKWzmAjSx88VIbPPtuiqUjrZcnKjSgeUb0Bef4sFFc0 dKWRc08NgInGdgQESJoL3bJmbN0+4pxMps15D7jCs93/PuRRVwVy9Vj9uZGvhA7E85eX NE5A== MIME-Version: 1.0 X-Received: by 10.182.28.134 with SMTP id b6mr76820169obh.27.1389103305716; Tue, 07 Jan 2014 06:01:45 -0800 (PST) Received: by 10.182.28.165 with HTTP; Tue, 7 Jan 2014 06:01:45 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Jan 2014 09:01:45 -0500 Message-ID: Subject: Re: Content of FSImage From: Hardik Pandya To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c2903c17fc5d04ef61d15f X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2903c17fc5d04ef61d15f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes - The entire file system namespace, including the mapping of blocks to files and file system properties, is stored in a file called the FsImage. The FsImage is stored as a file in the NameNode=92s local file system too. When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint. In the current implementation, a checkpoint only occurs when the NameNode starts up. Work is in progress to support periodic checkpointing in the near future. During Metadata Disk Failure The FsImage and the EditLog are central data structures of HDFS. A corruption of these files can cause the HDFS instance to be non-functional. For this reason, the NameNode can be configured to support maintaining multiple copies of the FsImage and EditLog. Any update to either the FsImage or EditLog causes each of the FsImages and EditLogs to get updated synchronously. This synchronous updating of multiple copies of the FsImageand EditLog may degrade the rate of namespace transactions per second that a NameNode can support. However, this degradation is acceptable because even though HDFS applications are very data intensive in nature, they are not metadata intensive. When a NameNode restarts, it selects the latest consistent FsImage and EditLog to use. Reference : HDFS Design Documentation On Tue, Jan 7, 2014 at 7:26 AM, Vishnu Viswanath < vishnu.viswanath25@gmail.com> wrote: > Hi All, > > I read that block information is stored in memory by hadoop once it > receives block report from the datanodes. > > EditLog logs the changes. > > What is exactly stored in the FSImage file? > Does it store information on the files in HDFS and how many blocks are > there etc? > > Thanks > > --001a11c2903c17fc5d04ef61d15f Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable

Yes -=A0The entire file sy= stem namespace, including the mapping of blocks to files and file system pr= operties, is stored in a file called the FsImage. The FsImage is = stored as a file in the NameNode=92s local file system too.

During Metadata Disk Failure

The FsImage = and the EditLog are central data structures of HDFS. A corruption of these = files can cause the HDFS instance to be non-functional. For this reason, th= e NameNode can be configured to support maintaining multiple copies of the = FsImage and EditLog. Any update to either the FsImage or EditLog causes eac= h of the FsImages and EditLogs to get updated synchronously. T= his synchronous updating of multiple copies of the FsImage and= EditLog may degrade the rate of namespace transactions per second that a N= ameNode can support. However, this degradation is acceptable because even t= hough HDFS applications are very data intensive in nature, they are not met= adata intensive. When a NameNode restarts, it selects the latest consistent= FsImage and EditLog to use.

Reference : HDFS Design Documentat= ion

On Tue, Jan 7, 2014 at 7:26 AM, Vishnu Viswanath <= vishnu.viswanath25@gmail.com> wrote:
Hi All,

I read that b= lock information is stored in memory by hadoop once it receives block repor= t from the datanodes.

EditLog logs = the changes.
<= br>
What is exactly stored in = the FSImage file?
Does it store information on the files in HDFS and how many blocks are = there etc?

Thanks


--001a11c2903c17fc5d04ef61d15f--