hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rang...@apache.org
Subject svn commit: r616581 - in /hadoop/core/trunk: CHANGES.txt src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml src/docs/src/documentation/content/xdocs/site.xml
Date Wed, 30 Jan 2008 01:47:46 GMT
Author: rangadi
Date: Tue Jan 29 17:47:44 2008
New Revision: 616581

URL: http://svn.apache.org/viewvc?rev=616581&view=rev
HADOOP-2529. First version of HDFS User Guide. (Raghu Angadi)


Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=616581&r1=616580&r2=616581&view=diff
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Tue Jan 29 17:47:44 2008
@@ -298,6 +298,8 @@
     HADOOP-2342. Created a micro-benchmark for measuring 
     local-file versus hdfs reads. (Owen O'Malley via nigel)
+    HADOOP-2529. First version of HDFS User Guide. (Raghu Angadi)
     HADOOP-1898.  Release the lock protecting the last time of the last stack

Added: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml?rev=616581&view=auto
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml (added)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml Tue Jan
29 17:47:44 2008
@@ -0,0 +1,404 @@
+<?xml version="1.0"?>
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+          "http://forrest.apache.org/dtd/document-v20.dtd">
+  <header>
+    <title>
+      Hadoop DFS User Guide
+    </title>
+  </header>
+  <body>
+    <section> <title>Purpose</title>
+      <p>
+ This document aims to be the starting point for users working with
+ Hadoop Distributed File System (HDFS) either as a part of a
+ <a href="http://hadoop.apache.org/">Hadoop</a>
+ cluster or as a stand-alone general purpose distributed file system.
+ While HDFS is designed to "just-work" in many environments, a working
+ knowledge of HDFS helps greatly with configuration improvements and
+ diagnostics on a specific cluster.
+      </p>
+    </section>
+    <section> <title> Overview </title>
+      <p>
+ HDFS is the primary distributed storage used by Hadoop applications. A
+ HDFS cluster primarily consists of a <em>NameNode</em> that manages the
+ filesystem metadata and Datanodes that store the actual data. The
+ architecture of HDFS is described in detail
+ <a href="hdfs_design.html">here</a>. This user guide primarily deals with 
+ interaction of users and administrators with HDFS clusters. 
+ The <a href="images/hdfsarchitecture.gif">diagram</a> from 
+ <a href="hdfs_design.html">HDFS architecture</a> depicts 
+ basic interactions among Namenode, Datanodes, and the clients. Eseentially,
+ clients contact Namenode for file metadata or file modifications and perform 
+ actual file I/O directly with the datanodes.
+      </p>
+      <p>
+ The following are some of the salient features that could be of
+ interest to many users. The terms in <em>italics</em>
+ are described in later sections.
+      </p> 
+    <ul>
+    <li>
+    	Hadoop, including HDFS, is well suited for distributed storage
+    	and distributed processing using commodity hardware. It is fault
+    	tolerant, scalable, and extremely simple to expand.
+    	<a href="mapred_tutorial.html">Map-Reduce</a>,
+    	well known for its simplicity and applicability for large set of
+    	distributed applications, is an integral part of Hadoop.
+    </li>
+    <li>
+    	HDFS is highly configurable with a default configuration well
+    	suited for many installations. Most of the time, configuration
+    	needs to be tuned only for very large clusters.
+    </li>
+    <li>
+    	It is written in Java and is supported on all major platforms.
+    </li>
+    <li>
+    	Supports <em>shell like commands</em> to interact with HDFS directly.
+    </li>
+    <li>
+    	Namenode and Datanodes have built in web servers that makes it
+    	easy to check current status of the cluster.
+    </li>
+    <li>
+    	New features and improvements are regularly implemented in HDFS.
+    	The following is a subset of useful features in HDFS:
+      <ul>
+    	<li>
+    		<em>File permissions and authentication.</em>
+    	</li>
+    	<li>
+    		<em>Rack awareness</em> : to take a node's physical location into
+    		account while scheduling tasks and allocating storage.
+    	</li>
+    	<li>
+    		<em>Safemode</em> : an administrative mode for maintanance.
+    	</li>
+    	<li>
+    		<em>fsck</em> : an utility to diagnose health of the filesystem, to
+    		find missing files or blocks.
+    	</li>
+    	<li>
+    		<em>Rebalancer</em> : tool to balance the cluster when the data is
+    		unevenly distributed among datanodes.
+    	</li>
+    	<li>
+    		<em>Upgrade and Rollback</em> : after a software upgrade, 
+            it is possible to
+    		rollback to HDFS' state before the upgrade in case of unexpected
+    		problems.
+    	</li>
+    	<li>
+    		<em>Secondary Namenode</em> : helps keep the size of file
+    		containing log of HDFS modification with in certain limit at
+    		the Namenode.
+    	</li>
+      </ul>
+    </li>
+    </ul>
+    </section> <section> <title> Pre-requisites </title>
+    <p>
+ 	The following documents describe installation and set up of a
+ 	Hadoop cluster : 
+    </p>
+ 	<ul>
+ 	<li>
+ 		<a href="quickstart.html">Hadoop Quickstart</a>
+ 		for first-time users.
+ 	</li>
+ 	<li>
+ 		<a href="cluster_setup.html">Hadoop Cluster Setup</a>
+ 		for large, distributed clusters.
+ 	</li>
+    </ul>
+    <p>
+ 	The rest of document assumes the user is able to set up and run a
+ 	HDFS with at least one Datanode. For the purpose of this document,
+ 	both Namenode and Datanode could be running on the same physical
+ 	machine.	
+    </p>
+    </section> <section> <title> Web Interface </title>
+ <p>
+ 	Namenode and Datanode each run an internal web server in order to
+ 	display basic information about the current status of the cluster.
+ 	With the default configuration, namenode front page is at
+ 	<code>http://namenode:50070/</code> .
+ 	It lists the datanodes in the cluster and basic stats of the
+ 	cluster. The web interface can also be used to browse the file
+ 	system (using "Browse the file system" link on the Namenode front
+ 	page).
+ </p> 
+    </section> <section> <title>Shell Commands</title>
+ 	<p>
+      Hadoop includes various "shell-like" commands that directly
+      interact with HDFS and other file systems that Hadoop supports.
+      The command
+      <code>bin/hadoop fs -help</code>
+      lists the commands supported by Hadoop
+      shell. Further,
+      <code>bin/hadoop fs -help command</code>
+      displays more detailed help on a command. The commands support
+      most of the normal filesystem operations like copying files,
+      changing file permissions, etc. It also supports a few HDFS
+      specific operations like changing replication of files.
+     </p>
+   <section> <title> DFSAdmin Command </title>
+   <p>
+   	<code>'bin/hadoop dfsadmin'</code>
+   	command supports a few HDFS administration related operations.
+   	<code>bin/hadoop dfsadmin -help</code>
+   	lists all the commands currently supported. For e.g.:
+   </p>  
+   	<ul>
+   	<li>
+   	    <code>-report</code>
+   	    : reports basic stats of HDFS. Some of this information is
+   	    also available on the Namenode front page.
+   	</li>
+   	<li>
+   		<code>-safemode</code>
+   		: though usually not required, an administrator can manually enter
+   		or leave <em>safemode</em>.
+   	</li>
+   	<li>
+   		<code>-finalizeUpgrade</code>
+   		: removes previous backup of the cluster made during last upgrade.
+   	</li>
+   	</ul>
+   </section>
+   </section> <section> <title> Secondary Namenode </title>
+   <p>
+     Namenode stores modifications to the filesystem as a log
+     appended to a native filesystem file (<code>edits</code>). 
+   	When a Namenode starts up, it reads HDFS state from an image
+   	file (<code>fsimage</code>) and then applies <em>edits</em> from

+    edits log file. It then writes new HDFS state to (<code>fsimage</code>)
+    and starts normal
+   	operation with an empty edits file. Since namenode merges
+   	<code>fsimage</code> and <code>edits</code> files only during
start up, 
+    edits file could get very large over time on a large cluster. 
+    Another side effect of larger edits file is that next 
+    restart of Namenade takes longer.
+   </p>
+   <p>
+     The secondary namenode merges fsimage and edits log periodically
+     and keeps edits log size with in a limit. It is usually run on a
+     different machine than the primary Namenode since its memory requirements
+     are on the same order as the primary namemode. The secondary
+     namenode is started by <code>bin/start-dfs.sh</code> on the nodes 
+     specified in <code>conf/masters</code> file.
+   </p>
+   </section> <section> <title> Rebalancer </title>
+    <p>
+      HDFS data might not always be be placed uniformly across the
+      datanode. One common reason is addition of new datanodes to an
+      existing cluster. While placing new <em>blocks</em> (data for a file is
+      stored as a series of blocks), Namenode considers various
+      parameters before choosing the datanodes to receive these blocks.
+      Some of the considerations are : 
+    </p>
+      <ul>
+      <li>
+        Policy to keep one of the replicas of a block on the same node
+        as the node that is writing the block.
+      </li>
+      <li>
+        Need to spread different replicas of a block across the racks so
+        that cluster can survive loss of whole rack.
+      </li>
+      <li>
+        One of the replicas is usually placed on the same rack as the
+        node writing to the file so that cross-rack network I/O is
+        reduced.
+      </li>
+      <li>
+        Spread HDFS data uniformly across the datanodes in the cluster.
+      </li>
+      </ul>
+    <p>
+      Due to multiple competing considerations, data might not be
+      uniformly placed across the datanodes.
+      HDFS provides a tool for administrators that analyzes block
+      placement and relanaces data across the datnodes. A brief
+      adminstrator's guide for rebalancer as a
+      <a href="http://issues.apache.org/jira/secure/attachment/12368261/RebalanceDesign6.pdf">PDF</a>
+      is attached to
+      <a href="http://issues.apache.org/jira/browse/HADOOP-1652">HADOOP-1652</a>.
+    </p>
+   </section> <section> <title> Rack Awareness </title>
+    <p>
+      Typically large Hadoop clusters are arranged in <em>racks</em> and
+      network traffic between different nodes with in the same rack is
+      much more desirable than network traffic across the racks. In
+      addition Namenode tries to place replicas of block on
+      multiple racks for improved fault tolerance. Hadoop lets the
+      cluster administrators decide which <em>rack</em> a node belongs to
+      through configuration variable <code>dfs.network.script</code>. When this
+      script is configured, each node runs the script to determine its
+      <em>rackid</em>. A default installation assumes all the nodes belong to
+      the same rack. This feature and configuration is further described
+      in <a href="http://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf">PDF</a>
+      attached to 
+      <a href="http://issues.apache.org/jira/browse/HADOOP-692">HADOOP-692</a>.
+    </p>
+   </section> <section> <title> Safemode </title>
+    <p>
+      During start up Namenode loads the filesystem state from
+      <em>fsimage</em> and <em>edits</em> log file. It then waits
for datanodes
+      to report their blocks so that it does not prematurely start
+      replicating the blocks though enough replicas already exist in the
+      cluster. During this time Namenode stays in <em>safemode</em>. A 
+      <em>Safemode</em>
+      for Namenode is essentially a read-only mode for the HDFS cluster,
+      where it does not allow any modifications to filesystem or blocks.
+      Normally Namenode gets out of safemode automatically at
+      the beginning. If required, HDFS could be placed in safemode explicitly
+      using <code>'bin/hadoop dfsadmin -safemode'</code> command. Namenode front
+      page shows whether safemode is on or off. A more detailed
+      description and configuration is maintained as JavaDoc for
+      <a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/NameNode.html#setSafeMode(org.apache.hadoop.dfs.FSConstants.SafeModeAction)"><code>setSafeMode()</code></a>.
+    </p>
+   </section> <section> <title> Fsck </title>
+     <p>    
+      HDFS supports <code>fsck</code> command to check for various 
+      inconsistencies.
+      It it is designed for reporting problems with various
+      files, for e.g. missing blocks for a file or under replicated
+      blocks. Unlike a traditional fsck utility for native filesystems,
+      this command does not correct the errors it detects. Normally Namenode
+      automatically corrects most of the recoverable failures.
+      HDFS' fsck is not a
+      Hadoop shell command. It can be run as '<code>bin/hadoop fsck</code>'.
+      Fsck can be run on the whole filesystem or on a subset of files.
+     </p>
+   </section> <section> <title> Upgrade and Rollback </title>
+     <p>
+      When Hadoop is upgraded on an existing cluster, as with any
+      software upgrade, it is possible there are new bugs or
+      incompatible changes that affect existing applications and were
+      not discovered earlier. In any non-trivial HDFS installation, it
+      is not an option to loose any data, let alone to restart HDFS from
+      scratch. HDFS allows administrators to go back to earlier version
+      of Hadoop and <em>roll back</em> the cluster to the state it was in 
+      before
+      the upgrade. HDFS upgrade is described in more detail in 
+      <a href="http://wiki.apache.org/hadoop/Hadoop%20Upgrade">upgrade wiki</a>.
+      HDFS can have one such backup at a time. Before upgrading,
+      administrators need to remove existing backup using <code>bin/hadoop
+      dfsadmin -finalizeUpgrade</code> command. The following
+      briefly describes typical upgrade procedure : 
+     </p>
+      <ul>
+      <li>
+        Before upgrading Hadoop software,
+        <em>finalize</em> if there an existing backup.
+        <code>dfsadmin -upgradeProgress status</code>
+        can tell if the cluster needs to be <em>finalized</em>.
+      </li>
+      <li>Stop the cluster and distribute new version of Hadoop.</li>
+      <li>
+        Run the new version with <code>-upgrade</code> option 
+        (<code>bin/start-dfs.sh -upgrade</code>).
+      </li>
+      <li>
+        Most of the time, cluster works just fine. Once the new HDFS is
+        considered working well (may be after a few days of operation),
+        finalize the upgrade. Note that until the cluster is finalized,
+        deleting the files that existed before the upgrade does not free
+        up real disk space on the datanodes.
+      </li>
+      <li>
+        If there is a need to move back to the old version,
+        <ul>
+          <li> stop the cluster and distribute earlier version of Hadoop. </li>
+          <li> start the cluster with rollback option. 
+            (<code>bin/start-dfs.h -rollback</code>). 
+          </li>
+        </ul>
+      </li>
+      </ul>
+   </section> <section> <title> File Permissions and Security </title>
+     <p>           
+      The file permissions are designed to be similar to file permissions on
+      other familiar platforms like Linux. Currently, security is limited
+      to simple file permissions. The user that starts Namenode is
+      treated as the <em>super user</em> for HDFS. Future versions of HDFS will
+      support network authentication protocols like Kerberos for user
+      authentication and encryption of data transfers.
+     </p>
+   </section> <section> <title> Scalability </title>
+     <p>
+      Hadoop currently runs on clusters with thousands of nodes.
+      <a href="http://wiki.apache.org/hadoop/PoweredBy">PoweredBy Hadoop</a>
+      lists some of the organizations that deploy Hadoop on large
+      clusters. HDFS has one Namenode for each cluster. Currently
+      the total memory available on Namenode is the primary scalability
+      limitation. On very large clusters, increasing average size of
+      files stored in HDFS helps with increasing cluster size without
+      increasing memory requirements on Namenode.
+      The default configuration may not suite very large clustes.
+      <a href="http://wiki.apache.org/hadoop/FAQ">Hadoop FAQ</a> page lists
+      suggested configuration improvements for large Hadoop clusters.
+     </p>
+   </section> <section> <title> Related Documentation </title>  
+      <p>
+      This user guide is intended to be a good starting point for
+      working with HDFS. While it continues to improve,
+      there is a large wealth of documentation about Hadoop and HDFS.
+      The following lists starting points for further exploration :
+      </p>
+      <ul>
+      <li>
+        <a href="http://hadoop.apache.org/">Hadoop Home Page</a>
+        : the start page for everything Hadoop.
+      </li>
+      <li>
+        <a href="http://wiki.apache.org/hadoop/FrontPage">Hadoop Wiki</a>
+        : Front page for Hadoop Wiki documentation. Unlike this
+        guide which is part of Hadoop source tree, Hadoop Wiki is
+        regularly edited by Hadoop Community.
+      </li>
+      <li> <a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a> from Hadoop
+      </li>
+      <li>
+        Hadoop <a href="http://hadoop.apache.org/core/docs/current/api/">
+          JavaDoc API</a>.
+      </li>
+      <li>
+        Hadoop User Mailing List : 
+        <a href="mailto:core-user@hadoop.apache.org">core-user[at]hadoop.apache.org</a>.
+      </li>
+      <li>
+         Explore <code>conf/hadoop-default.xml</code>. 
+         It includes brief 
+         description of most of the configuration variables available.
+      </li>
+      </ul>
+     </section>
+  </body>

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=616581&r1=616580&r2=616581&view=diff
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml Tue Jan 29 17:47:44
@@ -21,6 +21,7 @@
     <quickstart label="Quickstart"        href="quickstart.html" />
     <setup     label="Cluster Setup"      href="cluster_setup.html" />
     <hdfs      label="HDFS Architecture"  href="hdfs_design.html" />
+    <hdfs      label="HDFS User Guide"    href="hdfs_user_guide.html" />
     <mapred    label="Map-Reduce Tutorial" href="mapred_tutorial.html" />
     <mapred    label="Native Hadoop Libraries" href="native_libraries.html" />
     <streaming label="Streaming"          href="streaming.html" />

View raw message