hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s..@apache.org
Subject svn commit: r764085 - in /hadoop/core/trunk: CHANGES.txt src/docs/src/documentation/content/xdocs/commands_manual.xml src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
Date Fri, 10 Apr 2009 22:20:14 GMT
Author: shv
Date: Fri Apr 10 22:20:13 2009
New Revision: 764085

URL: http://svn.apache.org/viewvc?rev=764085&view=rev
Log:
HADOOP-5502. Documentation for backup and checkpoint nodes. Contributed by Jakob Homan and
Konstantin Shvachko.

Modified:
    hadoop/core/trunk/CHANGES.txt
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml

Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Fri Apr 10 22:20:13 2009
@@ -213,7 +213,11 @@
     (Konstantin Shvachko via szetszwo)
 
     HADOOP-4584. Improve datanode block reports and associated file system
-    scan to avoid interefering with normal datanode operations. 
+    scan to avoid interefering with normal datanode operations.
+    (Suresh Srinivas via rangadi)
+
+    HADOOP-5502. Documentation for backup and checkpoint nodes.
+    (Jakob Homan via shv)
 
   OPTIMIZATIONS
 

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml Fri Apr
10 22:20:13 2009
@@ -589,6 +589,18 @@
 				<table>
 			          <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
 			
+                <tr>
+                  <td><code>-regular</code></td>
+                  <td>Start namenode in standard, active role rather than as backup
or checkpoint node. This is the default role.</td>
+                </tr>
+                <tr>
+                  <td><code>-checkpoint</code></td>
+                  <td>Start namenode in checkpoint role, creating periodic checkpoints
of the active namenode metadata.</td>
+                </tr>
+                <tr>
+                  <td><code>-backup</code></td>
+                  <td>Start namenode in backup role, maintaining an up-to-date in-memory
copy of the namespace and creating periodic checkpoints.</td>
+                </tr>
 			           <tr>
 			          	<td><code>-format</code></td>
 			            <td>Formats the namenode. It starts the namenode, formats it and then
shut it down.</td>
@@ -609,7 +621,7 @@
 			           </tr>
 			           <tr>
 			          	<td><code>-importCheckpoint</code></td>
-			            <td>Loads image from a checkpoint directory and save it into the current
one. Checkpoint dir 
+			            <td>Loads image from a checkpoint directory and saves it into the current
one. Checkpoint directory 
 			            is read from property fs.checkpoint.dir</td>
 			           </tr>
 			     </table>
@@ -618,7 +630,10 @@
 			<section>
 				<title> secondarynamenode </title>
 				<p>
-					Runs the HDFS secondary namenode. See <a href="hdfs_user_guide.html#Secondary+Namenode">Secondary
Namenode</a> 
+					Use of the Secondary NameNode has been deprecated. Instead, consider using a 
+					<a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint node</a> or 
+					<a href="hdfs_user_guide.html#Backup+node">Backup node</a>. Runs the HDFS
secondary 
+					namenode. See <a href="hdfs_user_guide.html#Secondary+NameNode">Secondary NameNode</a>

 					for more info.
 				</p>
 				<p>

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml Fri Apr
10 22:20:13 2009
@@ -110,9 +110,25 @@
     		problems.
     	</li>
     	<li>
-    		Secondary NameNode: performs periodic checkpoints of the 
+    		Secondary NameNode (deprecated): performs periodic checkpoints of the 
     		namespace and helps keep the size of file containing log of HDFS 
     		modifications within certain limits at the NameNode.
+    		Replaced by Checkpoint node.
+    	</li>
+    	<li>
+    		Checkpoint node: performs periodic checkpoints of the namespace and
+    		helps minimize the size of the log stored at the NameNode 
+    		containing changes to the HDFS.
+    		Replaces the role previously filled by the Secondary NameNode. 
+    		NameNode allows multiple Checkpoint nodes simultaneously, 
+    		as long as there are no Backup nodes registered with the system.
+    	</li>
+    	<li>
+    		Backup node: An extension to the Checkpoint node.
+    		In addition to checkpointing it also receives a stream of edits 
+    		from the NameNode and maintains its own in-memory copy of the namespace,
+    		which is always in sync with the active NameNode namespace state.
+    		Only one Backup node may be registered with the NameNode at once.
     	</li>
       </ul>
     </li>
@@ -211,12 +227,18 @@
    	</p>  
    </section>
    
-   </section> <section> <title> Secondary NameNode </title>
+   </section> 
+	<section> <title>Secondary NameNode</title>
    <p>
+     The Secondary NameNode has been deprecated; considering using the 
+   <a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint node</a> or 
+   <a href="hdfs_user_guide.html#Backup+node">Backup node</a> instead.
+   </p>
+   <p>	
      The NameNode stores modifications to the file system as a log
-     appended to a native file system file (<code>edits</code>). 
+     appended to a native file system file, <code>edits</code>. 
    	When a NameNode starts up, it reads HDFS state from an image
-   	file (<code>fsimage</code>) and then applies edits from the
+   	file, <code>fsimage</code>, and then applies edits from the
     edits log file. It then writes new HDFS state to the <code>fsimage</code>
     and starts normal
    	operation with an empty edits file. Since NameNode merges
@@ -255,7 +277,121 @@
      read by the primary NameNode if necessary.
    </p>
    <p>
-     The latest checkpoint can be imported to the primary NameNode if
+     For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code>
command</a>.
+   </p>
+   
+   </section><section> <title> Checkpoint node </title>
+   <p>NameNode persists its namespace using two files: <code>fsimage</code>,
+      which is the latest checkpoint of the namespace and <code>edits</code>,
+      a journal (log) of changes to the namespace since the checkpoint.
+      When a NameNode starts up, it merges the <code>fsimage</code> and
+      <code>edits</code> journal to provide an up-to-date view of the
+      file system metadata.
+      The NameNode then overwrites <code>fsimage</code> with the new HDFS state

+      and begins a new <code>edits</code> journal. 
+   </p>
+   <p>
+     The Checkpoint node periodically creates checkpoints of the namespace. 
+     It downloads <code>fsimage</code> and <code>edits</code> from
the active 
+     NameNode, merges them locally, and uploads the new image back to the 
+     active NameNode.
+     The Checkpoint node usually runs on a different machine than the NameNode
+     since its memory requirements are on the same order as the NameNode.
+     The Checkpoint node is started by 
+     <code>bin/hdfs namenode -checkpoint</code> on the node 
+     specified in the configuration file.
+   </p>
+   <p>The location of the Checkpoint (or Backup) node and its accompanying 
+      web interface are configured via the <code>dfs.backup.address</code> 
+      and <code>dfs.backup.http.address</code> configuration variables.
+	 </p>
+   <p>
+     The start of the checkpoint process on the Checkpoint node is 
+     controlled by two configuration parameters.
+   </p>
+   <ul>
+      <li>
+        <code>fs.checkpoint.period</code>, set to 1 hour by default, specifies
+        the maximum delay between two consecutive checkpoints 
+      </li>
+      <li>
+        <code>fs.checkpoint.size</code>, set to 64MB by default, defines the
+        size of the edits log file that forces an urgent checkpoint even if 
+        the maximum checkpoint delay is not reached.
+      </li>
+   </ul>
+   <p>
+     The Checkpoint node stores the latest checkpoint in a  
+     directory that is structured the same as the NameNode's
+     directory. This allows the checkpointed image to be always available for
+     reading by the NameNode if necessary.
+     See <a href="hdfs_user_guide.html#Import+checkpoint">Import checkpoint</a>.
+   </p>
+   <p>Multiple checkpoint nodes may be specified in the cluster configuration file.</p>
+   <p>
+     For command usage, see
+     <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
+   </p>
+   </section>
+
+   <section> <title> Backup node </title>
+   <p>	
+    The Backup node provides the same checkpointing functionality as the 
+    Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the
+    file system namespace that is always synchronized with the active NameNode state.
+    Along with accepting a journal stream of filesystem edits from 
+    the NameNode and persisting this to disk, the Backup node also applies 
+    those edits into its own copy of the namespace in memory, thus creating 
+    a backup of the namespace.
+   </p>
+   <p>
+    The Backup node does not need to download 
+    <code>fsimage</code> and <code>edits</code> files from the active
NameNode
+    in order to create a checkpoint, as would be required with a 
+    Checkpoint node or Secondary NameNode, since it already has an up-to-date 
+    state of the namespace state in memory.
+    The Backup node checkpoint process is more efficient as it only needs to 
+    save the namespace into the local <code>fsimage</code> file and reset
+    <code>edits</code>.
+   </p> 
+   <p>
+    As the Backup node maintains a copy of the
+    namespace in memory, its RAM requirements are the same as the NameNode.
+   </p> 
+   <p>
+    The NameNode supports one Backup node at a time. No Checkpoint nodes may be
+    registered if a Backup node is in use. Using multiple Backup nodes 
+    concurrently will be supported in the future.
+   </p> 
+   <p>
+    The Backup node is configured in the same manner as the Checkpoint node.
+    It is started with <code>bin/hdfs namenode -checkpoint</code>.
+   </p>
+   <p>The location of the Backup (or Checkpoint) node and its accompanying 
+      web interface are configured via the <code>dfs.backup.address</code> 
+      and <code>dfs.backup.http.address</code> configuration variables.
+	 </p>
+   <p>
+    Use of a Backup node provides the option of running the NameNode with no 
+    persistent storage, delegating all responsibility for persisting the state
+    of the namespace to the Backup node. 
+    To do this, start the NameNode with the 
+    <code>-importCheckpoint</code> option, along with specifying no persistent
+    storage directories of type edits <code>dfs.name.edits.dir</code> 
+    for the NameNode configuration.
+   </p> 
+   <p>
+    For a complete discussion of the motivation behind the creation of the 
+    Backup node and Checkpoint node, see 
+    <a href="https://issues.apache.org/jira/browse/HADOOP-4539">HADOOP-4539</a>.
+    For command usage, see 
+    <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
+   </p>
+   </section>
+
+   <section> <title> Import checkpoint </title>
+   <p>
+     The latest checkpoint can be imported to the NameNode if
      all other copies of the image and the edits files are lost.
      In order to do that one should:
    </p>
@@ -282,10 +418,12 @@
      consistent, but does not modify it in any way.
    </p>
    <p>
-     For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code>
command</a>.
+     For command usage, see
+     <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
    </p>
-   
-   </section> <section> <title> Rebalancer </title>
+   </section>
+
+   <section> <title> Rebalancer </title>
     <p>
       HDFS data might not always be be placed uniformly across the
       DataNode. One common reason is addition of new DataNodes to an



Mime
View raw message