hadoop-hdfs-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s..@apache.org
Subject svn commit: r817449 [4/8] - in /hadoop/hdfs/branches/HDFS-265: ./ .eclipse.templates/.launches/ lib/ src/contrib/block_forensics/ src/contrib/block_forensics/client/ src/contrib/block_forensics/ivy/ src/contrib/block_forensics/src/java/org/apache/hadoo...
Date Mon, 21 Sep 2009 22:33:12 GMT
Added: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_shell.xml
URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_shell.xml?rev=817449&view=auto
==============================================================================
--- hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_shell.xml (added)
+++ hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_shell.xml Mon Sep 21 22:33:09 2009
@@ -0,0 +1,465 @@
+<?xml version="1.0"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
+<document>
+	<header>
+		<title>HDFS File System Shell Guide</title>
+	</header>
+	<body>
+		<section>
+			<title>Overview</title>
+			<p>
+      The FileSystem (FS) shell is invoked by 
+      <code>bin/hadoop fs &lt;args&gt;</code>.
+      All FS shell commands take path URIs as arguments. The URI
+      format is <em>scheme://autority/path</em>. For HDFS the scheme
+      is <em>hdfs</em>, and for the local filesystem the scheme
+      is <em>file</em>. The scheme and authority are optional. If not
+      specified, the default scheme specified in the configuration is
+      used. An HDFS file or directory such as <em>/parent/child</em>
+      can be specified as <em>hdfs://namenodehost/parent/child</em> or
+      simply as <em>/parent/child</em> (given that your configuration
+      is set to point to <em>hdfs://namenodehost</em>). Most of the
+      commands in FS shell behave like corresponding Unix
+      commands. Differences are described with each of the
+      commands. Error information is sent to <em>stderr</em> and the
+      output is sent to <em>stdout</em>.
+  </p>
+		<section>
+			<title> cat </title>
+			<p>
+				<code>Usage: hadoop fs -cat URI [URI &#x2026;]</code>
+			</p>
+			<p>
+		   Copies source paths to <em>stdout</em>. 
+		   </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2 
+		   </code>
+				</li>
+				<li>
+					<code>hadoop fs -cat file:///file3 /user/hadoop/file4 </code>
+				</li>
+			</ul>
+			<p>Exit Code:<br/>
+		   <code> Returns 0 on success and -1 on error. </code></p>
+		</section>
+		<section>
+			<title> chgrp </title>
+			<p>
+				<code>Usage: hadoop fs -chgrp [-R] GROUP URI [URI &#x2026;]</code>
+			</p>
+			<p>
+	    Change group association of files. With <code>-R</code>, make the change recursively through the directory structure. The user must be the owner of files, or else a super-user. Additional information is in the <a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>.
+	    </p>
+		</section>
+		<section>
+			<title> chmod </title>
+			<p>
+				<code>Usage: hadoop fs -chmod [-R] &lt;MODE[,MODE]... | OCTALMODE&gt; URI [URI &#x2026;]</code>
+			</p>
+			<p>
+	    Change the permissions of files. With <code>-R</code>, make the change recursively through the directory structure. The user must be the owner of the file, or else a super-user. Additional information is in the <a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>.
+	    </p>
+		</section>
+		<section>
+			<title> chown </title>
+			<p>
+				<code>Usage: hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]</code>
+			</p>
+			<p>
+	    Change the owner of files. With <code>-R</code>, make the change recursively through the directory structure. The user must be a super-user. Additional information is in the <a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>.
+	    </p>
+		</section>
+		<section>
+			<title>copyFromLocal</title>
+			<p>
+				<code>Usage: hadoop fs -copyFromLocal &lt;localsrc&gt; URI</code>
+			</p>
+			<p>Similar to <a href="#put"><strong>put</strong></a> command, except that the source is restricted to a local file reference. </p>
+		</section>
+		<section>
+			<title> copyToLocal</title>
+			<p>
+				<code>Usage: hadoop fs -copyToLocal [-ignorecrc] [-crc] URI &lt;localdst&gt;</code>
+			</p>
+			<p> Similar to <a href="#get"><strong>get</strong></a> command, except that the destination is restricted to a local file reference.</p>
+		</section>
+		<section>
+			<title> count </title>
+			<p>
+				<code>Usage: hadoop fs -count [-q]  &lt;paths&gt;</code>
+			</p>
+			<p>
+				Count the number of directories, files and bytes under the paths that match the specified file pattern. The output columns are:<br/><code>DIR_COUNT, FILE_COUNT, CONTENT_SIZE FILE_NAME</code>. <br/><br/>The output columns with <code>-q</code> are:<br/><code>QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME</code>.
+		   </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2 
+		   </code>
+				</li>
+				<li>
+					<code> hadoop fs -count -q hdfs://nn1.example.com/file1
+		   </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error.</code>
+			</p>
+		</section>
+		<section>
+			<title> cp </title>
+			<p>
+				<code>Usage: hadoop fs -cp URI [URI &#x2026;] &lt;dest&gt;</code>
+			</p>
+			<p>
+	    Copy files from source to destination. This command allows multiple sources as well in which case the destination must be a directory.
+	    <br/>
+	    Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2</code>
+				</li>
+				<li>
+					<code> hadoop fs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error.</code>
+			</p>
+		</section>
+		<section>
+			<title>du</title>
+			<p>
+				<code>Usage: hadoop fs -du URI [URI &#x2026;]</code>
+			</p>
+			<p>
+	     Displays aggregate length of  files contained in the directory or the length of a file in case its just a file.<br/>
+	     Example:<br/><code>hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1</code><br/>
+	     Exit Code:<br/><code> Returns 0 on success and -1 on error. </code><br/></p>
+		</section>
+		<section>
+			<title> dus </title>
+			<p>
+				<code>Usage: hadoop fs -dus &lt;args&gt;</code>
+			</p>
+			<p>
+	    Displays a summary of file lengths.
+	   </p>
+		</section>
+		<section>
+			<title> expunge </title>
+			<p>
+				<code>Usage: hadoop fs -expunge</code>
+			</p>
+			<p>Empty the Trash. Refer to <a href="hdfs_design.html">HDFS Architecture</a> for more information on Trash feature.
+	   </p>
+		</section>
+		<section>
+			<title> get </title>
+			<p>
+				<code>Usage: hadoop fs -get [-ignorecrc] [-crc] &lt;src&gt; &lt;localdst&gt;</code>
+				<br/>
+			</p>
+			<p>
+	   Copy files to the local file system. Files that fail the CRC check may be copied with the  
+	   <code>-ignorecrc</code> option. Files and CRCs may be copied using the 
+	   <code>-crc</code> option.
+	  </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -get /user/hadoop/file localfile </code>
+				</li>
+				<li>
+					<code> hadoop fs -get hdfs://nn.example.com/user/hadoop/file localfile</code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error. </code>
+			</p>
+		</section>
+		<section>
+			<title> getmerge </title>
+			<p>
+				<code>Usage: hadoop fs -getmerge &lt;src&gt; &lt;localdst&gt; [addnl]</code>
+			</p>
+			<p>
+	  Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally <code>addnl</code> can be set to enable adding a newline character at the end of each file.  
+	  </p>
+		</section>
+       <section>
+           <title>ls</title>
+           <p>
+               <code>Usage: hadoop fs -ls &lt;args&gt;</code>
+           </p>
+           <p>For a file returns stat on the file with the following format:</p>
+           <p>
+               <code>permissions number_of_replicas userid  groupid  filesize modification_date modification_time filename</code>
+           </p>
+           <p>For a directory it returns list of its direct children as in unix.A directory is listed as:</p>
+           <p>
+               <code>permissions userid groupid modification_date modification_time dirname</code>
+           </p>
+           <p>Example:</p>
+           <p>
+               <code>hadoop fs -ls /user/hadoop/file1 </code>
+           </p>
+           <p>Exit Code:</p>
+           <p>
+               <code>Returns 0 on success and -1 on error.</code>
+           </p>
+       </section>
+		<section>
+			<title>lsr</title>
+			<p><code>Usage: hadoop fs -lsr &lt;args&gt;</code><br/>
+	      Recursive version of <code>ls</code>. Similar to Unix <code>ls -R</code>.
+	      </p>
+		</section>
+		<section>
+			<title> mkdir </title>
+			<p>
+				<code>Usage: hadoop fs -mkdir &lt;paths&gt;</code>
+				<br/>
+			</p>
+			<p>
+	   Takes path uri's as argument and creates directories. The behavior is much like unix mkdir -p creating parent directories along the path.
+	  </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code>hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2 </code>
+				</li>
+				<li>
+					<code>hadoop fs -mkdir hdfs://nn1.example.com/user/hadoop/dir hdfs://nn2.example.com/user/hadoop/dir
+	  </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code>Returns 0 on success and -1 on error.</code>
+			</p>
+		</section>
+		<section>
+			<title> moveFromLocal </title>
+			<p>
+				<code>Usage: dfs -moveFromLocal &lt;localsrc&gt; &lt;dst&gt;</code>
+			</p>
+			<p>Similar to <a href="#put"><strong>put</strong></a> command, except that the source <code>localsrc</code> is deleted after it's copied. </p>
+		</section>
+		<section>
+			<title> moveToLocal</title>
+			<p>
+				<code>Usage: hadoop fs -moveToLocal [-crc] &lt;src&gt; &lt;dst&gt;</code>
+			</p>
+			<p>Displays a "Not implemented yet" message.</p>
+		</section>
+		<section>
+			<title> mv </title>
+			<p>
+				<code>Usage: hadoop fs -mv URI [URI &#x2026;] &lt;dest&gt;</code>
+			</p>
+			<p>
+	    Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across filesystems is not permitted.
+	    <br/>
+	    Example:
+	    </p>
+			<ul>
+				<li>
+					<code> hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2</code>
+				</li>
+				<li>
+					<code> hadoop fs -mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs://nn.example.com/dir1</code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error.</code>
+			</p>
+		</section>
+		<section>
+			<title> put </title>
+			<p>
+				<code>Usage: hadoop fs -put &lt;localsrc&gt; ... &lt;dst&gt;</code>
+			</p>
+			<p>Copy single src, or multiple srcs from local file system to the destination filesystem. Also reads input from stdin and writes to destination filesystem.<br/>
+	   </p>
+			<ul>
+				<li>
+					<code> hadoop fs -put localfile /user/hadoop/hadoopfile</code>
+				</li>
+				<li>
+					<code> hadoop fs -put localfile1 localfile2 /user/hadoop/hadoopdir</code>
+				</li>
+				<li>
+					<code> hadoop fs -put localfile hdfs://nn.example.com/hadoop/hadoopfile</code>
+				</li>
+				<li><code>hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile</code><br/>Reads the input from stdin.</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error. </code>
+			</p>
+		</section>
+		<section>
+			<title> rm </title>
+			<p>
+				<code>Usage: hadoop fs -rm URI [URI &#x2026;] </code>
+			</p>
+			<p>
+	   Delete files specified as args. Only deletes non empty directory and files. Refer to rmr for recursive deletes.<br/>
+	   Example:
+	   </p>
+			<ul>
+				<li>
+					<code> hadoop fs -rm hdfs://nn.example.com/file /user/hadoop/emptydir </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error.</code>
+			</p>
+		</section>
+		<section>
+			<title> rmr </title>
+			<p>
+				<code>Usage: hadoop fs -rmr URI [URI &#x2026;]</code>
+			</p>
+			<p>Recursive version of delete.<br/>
+	   Example:
+	   </p>
+			<ul>
+				<li>
+					<code> hadoop fs -rmr /user/hadoop/dir </code>
+				</li>
+				<li>
+					<code> hadoop fs -rmr hdfs://nn.example.com/user/hadoop/dir </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code> Returns 0 on success and -1 on error. </code>
+			</p>
+		</section>
+		<section>
+			<title> setrep </title>
+			<p>
+				<code>Usage: hadoop fs -setrep [-R] &lt;path&gt;</code>
+			</p>
+			<p>
+	   Changes the replication factor of a file. -R option is for recursively increasing the replication factor of files within a directory.
+	  </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -setrep -w 3 -R /user/hadoop/dir1 </code>
+				</li>
+			</ul>
+			<p>Exit Code:</p>
+			<p>
+				<code>Returns 0 on success and -1 on error. </code>
+			</p>
+		</section>
+		<section>
+			<title> stat </title>
+			<p>
+				<code>Usage: hadoop fs -stat URI [URI &#x2026;]</code>
+			</p>
+			<p>
+	   Returns the stat information on the path.
+	   </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -stat path </code>
+				</li>
+			</ul>
+			<p>Exit Code:<br/>
+	   <code> Returns 0 on success and -1 on error.</code></p>
+		</section>
+		<section>
+			<title> tail </title>
+			<p>
+				<code>Usage: hadoop fs -tail [-f] URI </code>
+			</p>
+			<p>
+	   Displays last kilobyte of the file to stdout. -f option can be used as in Unix.
+	   </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -tail pathname </code>
+				</li>
+			</ul>
+			<p>Exit Code: <br/>
+	   <code> Returns 0 on success and -1 on error.</code></p>
+		</section>
+		<section>
+			<title> test </title>
+			<p>
+				<code>Usage: hadoop fs -test -[ezd] URI</code>
+			</p>
+			<p>
+	   Options: <br/>
+	   -e check to see if the file exists. Return 0 if true. <br/>
+	   -z check to see if the file is zero length. Return 0 if true. <br/>
+	   -d check to see if the path is directory. Return 0 if true. <br/></p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop fs -test -e filename </code>
+				</li>
+			</ul>
+		</section>
+		<section>
+			<title> text </title>
+			<p>
+				<code>Usage: hadoop fs -text &lt;src&gt;</code>
+				<br/>
+			</p>
+			<p>
+	   Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.
+	  </p>
+		</section>
+		<section>
+			<title> touchz </title>
+			<p>
+				<code>Usage: hadoop fs -touchz URI [URI &#x2026;]</code>
+				<br/>
+			</p>
+			<p>
+	   Create a file of zero length.
+	   </p>
+			<p>Example:</p>
+			<ul>
+				<li>
+					<code> hadoop -touchz pathname </code>
+				</li>
+			</ul>
+			<p>Exit Code:<br/>
+	   <code> Returns 0 on success and -1 on error.</code></p>
+		</section>
+        </section>
+	</body>
+</document>

Propchange: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_shell.xml
------------------------------------------------------------------------------
    svn:mime-type = text/plain

Modified: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml?rev=817449&r1=817448&r2=817449&view=diff
==============================================================================
--- hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml (original)
+++ hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml Mon Sep 21 22:33:09 2009
@@ -24,7 +24,7 @@
 
   <header>
     <title>
-      HDFS Users Guide
+      HDFS User Guide
     </title>
   </header>
 
@@ -32,8 +32,9 @@
     <section> <title>Purpose</title>
       <p>
  This document is a starting point for users working with
- Hadoop Distributed File System (HDFS) either as a part of a Hadoop cluster  
- or as a stand-alone general purpose distributed file system.
+ Hadoop Distributed File System (HDFS) either as a part of a
+ <a href="http://hadoop.apache.org/">Hadoop</a>
+ cluster or as a stand-alone general purpose distributed file system.
  While HDFS is designed to "just work" in many environments, a working
  knowledge of HDFS helps greatly with configuration improvements and
  diagnostics on a specific cluster.
@@ -45,7 +46,7 @@
  HDFS is the primary distributed storage used by Hadoop applications. A
  HDFS cluster primarily consists of a NameNode that manages the
  file system metadata and DataNodes that store the actual data. The
- <a href="hdfs_design.html">HDFS Architecture Guide</a> describes HDFS in detail. This user guide primarily deals with 
+ <a href="hdfs_design.html">HDFS Architecture</a> describes HDFS in detail. This user guide primarily deals with 
  the interaction of users and administrators with HDFS clusters. 
  The <a href="images/hdfsarchitecture.gif">HDFS architecture diagram</a> depicts 
  basic interactions among NameNode, the DataNodes, and the clients. 
@@ -60,7 +61,8 @@
     <li>
     	Hadoop, including HDFS, is well suited for distributed storage
     	and distributed processing using commodity hardware. It is fault
-    	tolerant, scalable, and extremely simple to expand. MapReduce, 
+    	tolerant, scalable, and extremely simple to expand.
+    	<a href="mapred_tutorial.html">Map/Reduce</a>,
     	well known for its simplicity and applicability for large set of
     	distributed applications, is an integral part of Hadoop.
     </li>
@@ -132,17 +134,18 @@
     </li>
     </ul>
     
-    </section> <section> <title> Prerequisites </title>
+    </section> <section> <title> Pre-requisites </title>
     <p>
- 	The following documents describe how to install and set up a Hadoop cluster: 
+ 	The following documents describe installation and set up of a
+ 	Hadoop cluster : 
     </p>
  	<ul>
  	<li>
- 		<a href="http://hadoop.apache.org/common/docs/current/single_node_setup.html">Single Node Setup</a>
+ 		<a href="quickstart.html">Hadoop Quick Start</a>
  		for first-time users.
  	</li>
  	<li>
- 		<a href="http://hadoop.apache.org/common/docs/current/cluster_setup.html">Cluster Setup</a>
+ 		<a href="cluster_setup.html">Hadoop Cluster Setup</a>
  		for large, distributed clusters.
  	</li>
     </ul>
@@ -170,15 +173,14 @@
       Hadoop includes various shell-like commands that directly
       interact with HDFS and other file systems that Hadoop supports.
       The command
-      <code>bin/hdfs dfs -help</code>
+      <code>bin/hadoop fs -help</code>
       lists the commands supported by Hadoop
       shell. Furthermore, the command
-      <code>bin/hdfs dfs -help command-name</code>
+      <code>bin/hadoop fs -help command-name</code>
       displays more detailed help for a command. These commands support
-      most of the normal files system operations like copying files,
+      most of the normal files ystem operations like copying files,
       changing file permissions, etc. It also supports a few HDFS
-      specific operations like changing replication of files. 
-      For more information see <a href="http://hadoop.apache.org/common/docs/current/file_system_shell.html">File System Shell Guide</a>.
+      specific operations like changing replication of files.
      </p>
 
    <section> <title> DFSAdmin Command </title>
@@ -221,19 +223,17 @@
     </li>
    	</ul>
    	<p>
-   	  For command usage, see  
-   	  <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#dfsadmin">dfsadmin</a>.
+   	  For command usage, see <a href="commands_manual.html#dfsadmin">dfsadmin command</a>.
    	</p>  
    </section>
    
    </section> 
 	<section> <title>Secondary NameNode</title>
-   <note>
-   The Secondary NameNode has been deprecated. 
-   Instead, consider using the 
-   <a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint Node</a> or 
-   <a href="hdfs_user_guide.html#Backup+node">Backup Node</a>.
-   </note>
+   <p>
+     The Secondary NameNode has been deprecated; considering using the 
+   <a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint node</a> or 
+   <a href="hdfs_user_guide.html#Backup+node">Backup node</a> instead.
+   </p>
    <p>	
      The NameNode stores modifications to the file system as a log
      appended to a native file system file, <code>edits</code>. 
@@ -277,11 +277,10 @@
      read by the primary NameNode if necessary.
    </p>
    <p>
-     For command usage, see  
-     <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#secondarynamenode">secondarynamenode</a>.
+     For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code> command</a>.
    </p>
    
-   </section><section> <title> Checkpoint Node </title>
+   </section><section> <title> Checkpoint node </title>
    <p>NameNode persists its namespace using two files: <code>fsimage</code>,
       which is the latest checkpoint of the namespace and <code>edits</code>,
       a journal (log) of changes to the namespace since the checkpoint.
@@ -330,17 +329,17 @@
    </p>
    <p>Multiple checkpoint nodes may be specified in the cluster configuration file.</p>
    <p>
-     For command usage, see  
-     <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#namenode">namenode</a>.
+     For command usage, see
+     <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
    </p>
    </section>
 
-   <section> <title> Backup Node </title>
+   <section> <title> Backup node </title>
    <p>	
     The Backup node provides the same checkpointing functionality as the 
     Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the
     file system namespace that is always synchronized with the active NameNode state.
-    Along with accepting a journal stream of file system edits from 
+    Along with accepting a journal stream of filesystem edits from 
     the NameNode and persisting this to disk, the Backup node also applies 
     those edits into its own copy of the namespace in memory, thus creating 
     a backup of the namespace.
@@ -385,12 +384,12 @@
     For a complete discussion of the motivation behind the creation of the 
     Backup node and Checkpoint node, see 
     <a href="https://issues.apache.org/jira/browse/HADOOP-4539">HADOOP-4539</a>.
-    For command usage, see  
-     <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#namenode">namenode</a>.
+    For command usage, see 
+    <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
    </p>
    </section>
 
-   <section> <title> Import Checkpoint </title>
+   <section> <title> Import checkpoint </title>
    <p>
      The latest checkpoint can be imported to the NameNode if
      all other copies of the image and the edits files are lost.
@@ -419,8 +418,8 @@
      consistent, but does not modify it in any way.
    </p>
    <p>
-     For command usage, see  
-      <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#namenode">namenode</a>.
+     For command usage, see
+     <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
    </p>
    </section>
 
@@ -462,8 +461,7 @@
       <a href="http://issues.apache.org/jira/browse/HADOOP-1652">HADOOP-1652</a>.
     </p>
     <p>
-     For command usage, see  
-     <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#balancer">balancer</a>.
+     For command usage, see <a href="commands_manual.html#balancer">balancer command</a>.
    </p>
     
    </section> <section> <title> Rack Awareness </title>
@@ -514,8 +512,7 @@
       <code>fsck</code> ignores open files but provides an option to select all files during reporting.
       The HDFS <code>fsck</code> command is not a
       Hadoop shell command. It can be run as '<code>bin/hadoop fsck</code>'.
-      For command usage, see  
-      <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html#fsck">fsck</a>.
+      For command usage, see <a href="commands_manual.html#fsck"><code>fsck</code> command</a>. 
       <code>fsck</code> can be run on the whole file system or on a subset of files.
      </p>
      
@@ -530,7 +527,7 @@
       of Hadoop and rollback the cluster to the state it was in 
       before
       the upgrade. HDFS upgrade is described in more detail in 
-      <a href="http://wiki.apache.org/hadoop/Hadoop%20Upgrade">Hadoop Upgrade</a> Wiki page.
+      <a href="http://wiki.apache.org/hadoop/Hadoop%20Upgrade">upgrade wiki</a>.
       HDFS can have one such backup at a time. Before upgrading,
       administrators need to remove existing backup using <code>bin/hadoop
       dfsadmin -finalizeUpgrade</code> command. The following
@@ -574,13 +571,13 @@
       treated as the superuser for HDFS. Future versions of HDFS will
       support network authentication protocols like Kerberos for user
       authentication and encryption of data transfers. The details are discussed in the 
-      <a href="hdfs_permissions_guide.html">Permissions Guide</a>.
+      <a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>.
      </p>
      
    </section> <section> <title> Scalability </title>
      <p>
-      Hadoop currently runs on clusters with thousands of nodes. The  
-      <a href="http://wiki.apache.org/hadoop/PoweredBy">PoweredBy</a> Wiki page 
+      Hadoop currently runs on clusters with thousands of nodes.
+      <a href="http://wiki.apache.org/hadoop/PoweredBy">Powered By Hadoop</a>
       lists some of the organizations that deploy Hadoop on large
       clusters. HDFS has one NameNode for each cluster. Currently
       the total memory available on NameNode is the primary scalability
@@ -588,8 +585,8 @@
       files stored in HDFS helps with increasing cluster size without
       increasing memory requirements on NameNode.
    
-      The default configuration may not suite very large clustes. The 
-      <a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a> Wiki page lists
+      The default configuration may not suite very large clustes.
+      <a href="http://wiki.apache.org/hadoop/FAQ">Hadoop FAQ</a> page lists
       suggested configuration improvements for large Hadoop clusters.
      </p>
      
@@ -602,16 +599,15 @@
       </p>
       <ul>
       <li>
-        <a href="http://hadoop.apache.org/">Hadoop Site</a>: The home page for the Apache Hadoop site.
+        <a href="http://hadoop.apache.org/">Hadoop Home Page</a>: The start page for everything Hadoop.
       </li>
       <li>
-        <a href="http://wiki.apache.org/hadoop/FrontPage">Hadoop Wiki</a>:
-        The home page (FrontPage) for the Hadoop Wiki. Unlike the released documentation, 
-        which is part of Hadoop source tree, Hadoop Wiki is
+        <a href="http://wiki.apache.org/hadoop/FrontPage">Hadoop Wiki</a>
+        : Front page for Hadoop Wiki documentation. Unlike this
+        guide which is part of Hadoop source tree, Hadoop Wiki is
         regularly edited by Hadoop Community.
       </li>
-      <li> <a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>: 
-      The FAQ Wiki page.
+      <li> <a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a> from Hadoop Wiki.
       </li>
       <li>
         Hadoop <a href="http://hadoop.apache.org/core/docs/current/api/">
@@ -627,7 +623,7 @@
          description of most of the configuration variables available.
       </li>
       <li>
-        <a href="http://hadoop.apache.org/common/docs/current/commands_manual.html">Hadoop Commands Guide</a>: Hadoop commands usage.
+        <a href="commands_manual.html">Hadoop Command Guide</a>: commands usage.
       </li>
       </ul>
      </section>

Added: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_admin_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_admin_guide.xml?rev=817449&view=auto
==============================================================================
--- hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_admin_guide.xml (added)
+++ hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_admin_guide.xml Mon Sep 21 22:33:09 2009
@@ -0,0 +1,412 @@
+<?xml version="1.0"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+          "http://forrest.apache.org/dtd/document-v20.dtd">
+
+
+<document>
+
+  <header>
+    <title> 
+      HOD Administrator Guide
+    </title>
+  </header>
+
+  <body>
+<section>
+<title>Overview</title>
+<p>Hadoop On Demand (HOD) is a system for provisioning and
+managing independent Hadoop Map/Reduce and Hadoop Distributed File System (HDFS)
+instances on a shared cluster 
+of nodes. HOD is a tool that makes it easy for administrators and users to 
+quickly setup and use Hadoop. HOD is also a very useful tool for Hadoop developers 
+and testers who need to share a physical cluster for testing their own Hadoop 
+versions.
+</p>
+
+<p>HOD relies on a resource manager (RM) for allocation of nodes that it can use for
+running Hadoop instances. At present it runs with the <a href="ext:hod/torque">Torque
+resource manager</a>.
+</p>
+
+<p>
+The basic system architecture of HOD includes these components:</p>
+<ul>
+  <li>A Resource manager (possibly together with a scheduler)</li>
+  <li>Various HOD components</li>
+  <li>Hadoop Map/Reduce and HDFS daemons</li>
+</ul>
+
+<p>
+HOD provisions and maintains Hadoop Map/Reduce and, optionally, HDFS instances 
+through interaction with the above components on a given cluster of nodes. A cluster of
+nodes can be thought of as comprising two sets of nodes:</p>
+<ul>
+  <li>Submit nodes: Users use the HOD client on these nodes to allocate clusters, and then
+use the Hadoop client to submit Hadoop jobs. </li>
+  <li>Compute nodes: Using the resource manager, HOD components are run on these nodes to 
+provision the Hadoop daemons. After that Hadoop jobs run on them.</li>
+</ul>
+
+<p>
+Here is a brief description of the sequence of operations in allocating a cluster and
+running jobs on them.
+</p>
+
+<ul>
+  <li>The user uses the HOD client on the Submit node to allocate a desired number of
+cluster nodes and to provision Hadoop on them.</li>
+  <li>The HOD client uses a resource manager interface (qsub, in Torque) to submit a HOD
+process, called the RingMaster, as a Resource Manager job, to request the user's desired number 
+of nodes. This job is submitted to the central server of the resource manager (pbs_server, in Torque).</li>
+  <li>On the compute nodes, the resource manager slave daemons (pbs_moms in Torque) accept
+and run jobs that they are assigned by the central server (pbs_server in Torque). The RingMaster 
+process is started on one of the compute nodes (mother superior, in Torque).</li>
+  <li>The RingMaster then uses another resource manager interface (pbsdsh, in Torque) to run
+the second HOD component, HodRing, as distributed tasks on each of the compute
+nodes allocated.</li>
+  <li>The HodRings, after initializing, communicate with the RingMaster to get Hadoop commands, 
+and run them accordingly. Once the Hadoop commands are started, they register with the RingMaster,
+giving information about the daemons.</li>
+  <li>All the configuration files needed for Hadoop instances are generated by HOD itself, 
+some obtained from options given by user in its own configuration file.</li>
+  <li>The HOD client keeps communicating with the RingMaster to find out the location of the 
+JobTracker and HDFS daemons.</li>
+</ul>
+
+<p>This guide shows you how to get started using HOD, reviews various HOD features and command line options, and provides detailed troubleshooting help.</p>
+
+</section>
+
+<section>
+<title>Pre-requisites</title>
+<p>To use HOD, your system should include the following hardware and software
+components.</p>
+<p>Operating System: HOD is currently tested on RHEL4.<br/>
+Nodes : HOD requires a minimum of three nodes configured through a resource manager.<br/></p>
+
+<p> Software </p>
+<p>The following components must be installed on ALL nodes before using HOD:</p>
+<ul>
+ <li><a href="ext:hod/torque">Torque: Resource manager</a></li>
+ <li><a href="ext:hod/python">Python</a> : HOD requires version 2.5.1 of Python.</li>
+</ul>
+
+<p>The following components are optional and can be installed to obtain better
+functionality from HOD:</p>
+<ul>
+ <li><a href="ext:hod/twisted-python">Twisted Python</a>: This can be
+  used for improving the scalability of HOD. If this module is detected to be
+  installed, HOD uses it, else it falls back to default modules.</li>
+ <li><a href="ext:site">Hadoop</a>: HOD can automatically
+ distribute Hadoop to all nodes in the cluster. However, it can also use a
+ pre-installed version of Hadoop, if it is available on all nodes in the cluster.
+  HOD currently supports Hadoop 0.15 and above.</li>
+</ul>
+
+<p>NOTE: HOD configuration requires the location of installs of these
+components to be the same on all nodes in the cluster. It will also
+make the configuration simpler to have the same location on the submit
+nodes.
+</p>
+</section>
+
+<section>
+<title>Resource Manager</title>
+<p>  Currently HOD works with the Torque resource manager, which it uses for its node
+  allocation and job submission. Torque is an open source resource manager from
+  <a href="ext:hod/cluster-resources">Cluster Resources</a>, a community effort
+  based on the PBS project. It provides control over batch jobs and distributed compute nodes. Torque is
+  freely available for download from <a href="ext:hod/torque-download">here</a>.
+  </p>
+
+<p>  All documentation related to torque can be seen under
+  the section TORQUE Resource Manager <a
+  href="ext:hod/torque-docs">here</a>. You can
+  get wiki documentation from <a
+  href="ext:hod/torque-wiki">here</a>.
+  Users may wish to subscribe to TORQUE’s mailing list or view the archive for questions,
+  comments <a
+  href="ext:hod/torque-mailing-list">here</a>.
+</p>
+
+<p>To use HOD with Torque:</p>
+<ul>
+ <li>Install Torque components: pbs_server on one node (head node), pbs_mom on all
+  compute nodes, and PBS client tools on all compute nodes and submit
+  nodes. Perform at least a basic configuration so that the Torque system is up and
+  running, that is, pbs_server knows which machines to talk to. Look <a
+  href="ext:hod/torque-basic-config">here</a>
+  for basic configuration.
+
+  For advanced configuration, see <a
+  href="ext:hod/torque-advanced-config">here</a></li>
+ <li>Create a queue for submitting jobs on the pbs_server. The name of the queue is the
+  same as the HOD configuration parameter, resource-manager.queue. The HOD client uses this queue to
+  submit the RingMaster process as a Torque job.</li>
+ <li>Specify a cluster name as a property for all nodes in the cluster.
+  This can be done by using the qmgr command. For example:
+  <code>qmgr -c "set node node properties=cluster-name"</code>. The name of the cluster is the same as
+  the HOD configuration parameter, hod.cluster. </li>
+ <li>Make sure that jobs can be submitted to the nodes. This can be done by
+  using the qsub command. For example:
+  <code>echo "sleep 30" | qsub -l nodes=3</code></li>
+</ul>
+
+</section>
+
+<section>
+<title>Installing HOD</title>
+
+<p>Once the resource manager is set up, you can obtain and
+install HOD.</p>
+<ul>
+ <li>If you are getting HOD from the Hadoop tarball, it is available under the 
+  'contrib' section of Hadoop, under the root  directory 'hod'.</li>
+ <li>If you are building from source, you can run ant tar from the Hadoop root
+  directory to generate the Hadoop tarball, and then get HOD from there,
+  as described above.</li>
+ <li>Distribute the files under this directory to all the nodes in the
+  cluster. Note that the location where the files are copied should be
+  the same on all the nodes.</li>
+  <li>Note that compiling hadoop would build HOD with appropriate permissions 
+  set on all the required script files in HOD.</li>
+</ul>
+</section>
+
+<section>
+<title>Configuring HOD</title>
+
+<p>You can configure HOD once it is installed. The minimal configuration needed
+to run HOD is described below. More advanced configuration options are discussed
+in the HOD Configuration Guide.</p>
+<section>
+  <title>Minimal Configuration</title>
+  <p>To get started using HOD, the following minimal configuration is
+  required:</p>
+<ul>
+ <li>On the node from where you want to run HOD, edit the file hodrc
+  located in the &lt;install dir&gt;/conf directory. This file
+  contains the minimal set of values required to run hod.</li>
+ <li>
+<p>Specify values suitable to your environment for the following
+  variables defined in the configuration file. Note that some of these
+  variables are defined at more than one place in the file.</p>
+
+  <ul>
+   <li>${JAVA_HOME}: Location of Java for Hadoop. Hadoop supports Sun JDK
+    1.6.x and above.</li>
+   <li>${CLUSTER_NAME}: Name of the cluster which is specified in the
+    'node property' as mentioned in resource manager configuration.</li>
+   <li>${HADOOP_HOME}: Location of Hadoop installation on the compute and
+    submit nodes.</li>
+   <li>${RM_QUEUE}: Queue configured for submitting jobs in the resource
+    manager configuration.</li>
+   <li>${RM_HOME}: Location of the resource manager installation on the
+    compute and submit nodes.</li>
+    </ul>
+</li>
+
+<li>
+<p>The following environment variables may need to be set depending on
+  your environment. These variables must be defined where you run the
+  HOD client and must also be specified in the HOD configuration file as the
+  value of the key resource_manager.env-vars. Multiple variables can be
+  specified as a comma separated list of key=value pairs.</p>
+
+  <ul>
+   <li>HOD_PYTHON_HOME: If you install python to a non-default location
+    of the compute nodes, or submit nodes, then this variable must be
+    defined to point to the python executable in the non-standard
+    location.</li>
+    </ul>
+</li>
+</ul>
+</section>
+
+  <section>
+    <title>Advanced Configuration</title>
+    <p> You can review and modify other configuration options to suit
+ your specific needs. Refer to the <a href="hod_config_guide.html">HOD Configuration
+ Guide</a> for more information.</p>
+  </section>
+</section>
+
+  <section>
+    <title>Running HOD</title>
+    <p>You can run HOD once it is configured. Refer to the<a
+    href="hod_user_guide.html"> HOD User Guide</a> for more information.</p>
+  </section>
+
+  <section>
+    <title>Supporting Tools and Utilities</title>
+    <p>This section describes supporting tools and utilities that can be used to
+    manage HOD deployments.</p>
+    
+    <section>
+      <title>logcondense.py - Manage Log Files</title>
+      <p>As mentioned in the 
+         <a href="hod_user_guide.html#Collecting+and+Viewing+Hadoop+Logs">HOD User Guide</a>,
+         HOD can be configured to upload
+         Hadoop logs to a statically configured HDFS. Over time, the number of logs uploaded
+         to HDFS could increase. logcondense.py is a tool that helps
+         administrators to remove log files uploaded to HDFS. </p>
+      <section>
+        <title>Running logcondense.py</title>
+        <p>logcondense.py is available under hod_install_location/support folder. You can either
+        run it using python, for example, <em>python logcondense.py</em>, or give execute permissions 
+        to the file, and directly run it as <em>logcondense.py</em>. logcondense.py needs to be 
+        run by a user who has sufficient permissions to remove files from locations where log 
+        files are uploaded in the HDFS, if permissions are enabled. For example as mentioned in the
+        <a href="hod_config_guide.html#3.7+hodring+options">HOD Configuration Guide</a>, the logs could
+        be configured to come under the user's home directory in HDFS. In that case, the user
+        running logcondense.py should have super user privileges to remove the files from under
+        all user home directories.</p>
+      </section>
+      <section>
+        <title>Command Line Options for logcondense.py</title>
+        <p>The following command line options are supported for logcondense.py.</p>
+          <table>
+            <tr>
+              <td>Short Option</td>
+              <td>Long option</td>
+              <td>Meaning</td>
+              <td>Example</td>
+            </tr>
+            <tr>
+              <td>-p</td>
+              <td>--package</td>
+              <td>Complete path to the hadoop script. The version of hadoop must be the same as the 
+                  one running HDFS.</td>
+              <td>/usr/bin/hadoop</td>
+            </tr>
+            <tr>
+              <td>-d</td>
+              <td>--days</td>
+              <td>Delete log files older than the specified number of days</td>
+              <td>7</td>
+            </tr>
+            <tr>
+              <td>-c</td>
+              <td>--config</td>
+              <td>Path to the Hadoop configuration directory, under which hadoop-site.xml resides.
+              The hadoop-site.xml must point to the HDFS NameNode from where logs are to be removed.</td>
+              <td>/home/foo/hadoop/conf</td>
+            </tr>
+            <tr>
+              <td>-l</td>
+              <td>--logs</td>
+              <td>A HDFS path, this must be the same HDFS path as specified for the log-destination-uri,
+              as mentioned in the  <a href="hod_config_guide.html#3.7+hodring+options">HOD Configuration Guide</a>,
+              without the hdfs:// URI string</td>
+              <td>/user</td>
+            </tr>
+            <tr>
+              <td>-n</td>
+              <td>--dynamicdfs</td>
+              <td>If true, this will indicate that the logcondense.py script should delete HDFS logs
+              in addition to Map/Reduce logs. Otherwise, it only deletes Map/Reduce logs, which is also the
+              default if this option is not specified. This option is useful if
+              dynamic HDFS installations 
+              are being provisioned by HOD, and the static HDFS installation is being used only to collect 
+              logs - a scenario that may be common in test clusters.</td>
+              <td>false</td>
+            </tr>
+            <tr>
+              <td>-r</td>
+              <td>--retain-master-logs</td>
+              <td>If true, this will keep the JobTracker logs of job in hod-logs inside HDFS and it 
+              will delete only the TaskTracker logs. Also, this will keep the Namenode logs along with 
+              JobTracker logs and will only delete the Datanode logs if 'dynamicdfs' options is set 
+              to true. Otherwise, it will delete the complete job directory from hod-logs inside 
+              HDFS. By default it is set to false.</td>
+              <td>false</td>
+            </tr>
+          </table>
+        <p>So, for example, to delete all log files older than 7 days using a hadoop-site.xml stored in
+        ~/hadoop-conf, using the hadoop installation under ~/hadoop-0.17.0, you could say:</p>
+        <p><em>python logcondense.py -p ~/hadoop-0.17.0/bin/hadoop -d 7 -c ~/hadoop-conf -l /user</em></p>
+      </section>
+    </section>
+    <section>
+      <title>checklimits.sh - Monitor Resource Limits</title>
+      <p>checklimits.sh is a HOD tool specific to the Torque/Maui environment
+      (<a href="ext:hod/maui">Maui Cluster Scheduler</a> is an open source job
+      scheduler for clusters and supercomputers, from clusterresources). The
+      checklimits.sh script
+      updates the torque comment field when newly submitted job(s) violate or
+      exceed
+      over user limits set up in Maui scheduler. It uses qstat, does one pass
+      over the torque job-list to determine queued or unfinished jobs, runs Maui
+      tool checkjob on each job to see if user limits are violated and then
+      runs torque's qalter utility to update job attribute 'comment'. Currently
+      it updates the comment as <em>User-limits exceeded. Requested:([0-9]*)
+      Used:([0-9]*) MaxLimit:([0-9]*)</em> for those jobs that violate limits.
+      This comment field is then used by HOD to behave accordingly depending on
+      the type of violation.</p>
+      <section>
+        <title>Running checklimits.sh</title>
+        <p>checklimits.sh is available under the hod_install_location/support
+        folder. This shell script can be run directly as <em>sh
+        checklimits.sh </em>or as <em>./checklimits.sh</em> after enabling
+        execute permissions. Torque and Maui binaries should be available
+        on the machine where the tool is run and should be in the path
+        of the shell script process. To update the
+        comment field of jobs from different users, this tool must be run with
+        torque administrative privileges. This tool must be run repeatedly
+        after specific intervals of time to frequently update jobs violating
+        constraints, for example via cron. Please note that the resource manager
+        and scheduler commands used in this script can be expensive and so
+        it is better not to run this inside a tight loop without sleeping.</p>
+      </section>
+    </section>
+
+    <section>
+      <title>verify-account - Script to verify an account under which 
+             jobs are submitted</title>
+      <p>Production systems use accounting packages to charge users for using
+      shared compute resources. HOD supports a parameter 
+      <em>resource_manager.pbs-account</em> to allow users to identify the
+      account under which they would like to submit jobs. It may be necessary
+      to verify that this account is a valid one configured in an accounting
+      system. The <em>hod-install-dir/bin/verify-account</em> script 
+      provides a mechanism to plug-in a custom script that can do this
+      verification.</p>
+      
+      <section>
+        <title>Integrating the verify-account script with HOD</title>
+        <p>HOD runs the <em>verify-account</em> script passing in the
+        <em>resource_manager.pbs-account</em> value as argument to the script,
+        before allocating a cluster. Sites can write a script that verify this 
+        account against their accounting systems. Returning a non-zero exit 
+        code from this script will cause HOD to fail allocation. Also, in
+        case of an error, HOD will print the output of script to the user.
+        Any descriptive error message can be passed to the user from the
+        script in this manner.</p>
+        <p>The default script that comes with the HOD installation does not
+        do any validation, and returns a zero exit code.</p>
+        <p>If the verify-account script is not found, then HOD will treat
+        that verification is disabled, and continue allocation as is.</p>
+      </section>
+    </section>
+
+  </section>
+
+</body>
+</document>

Propchange: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_admin_guide.xml
------------------------------------------------------------------------------
    svn:mime-type = text/plain

Added: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_config_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_config_guide.xml?rev=817449&view=auto
==============================================================================
--- hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_config_guide.xml (added)
+++ hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_config_guide.xml Mon Sep 21 22:33:09 2009
@@ -0,0 +1,344 @@
+<?xml version="1.0"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN"
+          "http://forrest.apache.org/dtd/document-v20.dtd">
+
+
+<document>
+
+  <header>
+    <title> 
+      HOD Configuration Guide
+    </title>
+  </header>
+
+  <body>
+    <section>
+      <title>1. Introduction</title>
+      <p>This guide discusses Hadoop on Demand (HOD) configuration sections and shows you how to work with the most important 
+      and commonly used HOD configuration options.</p>
+      <p>Configuration options 
+      can be specified in two ways: a configuration file 
+      in the INI format, and as command line options to the HOD shell, 
+      specified in the format --section.option[=value]. If the same option is 
+      specified in both places, the value specified on the command line 
+      overrides the value in the configuration file.</p>
+      
+      <p>
+        To get a simple description of all configuration options, type:
+      </p>
+      <table><tr><td><code>$ hod --verbose-help</code></td></tr></table>
+      
+
+    </section>
+    
+    <section>
+      <title>2. Sections</title>
+    
+      <p>HOD organizes configuration options into these sections:</p>
+      
+      <ul>
+        <li>  hod:                  Options for the HOD client</li>
+        <li>  resource_manager:     Options for specifying which resource manager
+         to use, and other parameters for using that resource manager</li>
+        <li>  ringmaster:           Options for the RingMaster process, </li>
+        <li>  hodring:              Options for the HodRing processes</li>
+        <li>  gridservice-mapred:   Options for the Map/Reduce daemons</li>
+        <li>  gridservice-hdfs:     Options for the HDFS daemons.</li>
+      </ul>
+    
+    </section>
+    
+    <section>
+      <title>3. HOD Configuration Options</title>
+  
+      <p>The following section describes configuration options common to most 
+      HOD sections followed by sections that describe configuration options 
+      specific to each HOD section.</p>
+      
+      <section> 
+        <title>3.1 Common configuration options</title>
+        
+        <p>Certain configuration options are defined in most of the sections of 
+        the HOD configuration. Options defined in a section, are used by the
+        process for which that section applies. These options have the same
+        meaning, but can have different values in each section.
+        </p>
+        
+        <ul>
+          <li>temp-dir: Temporary directory for usage by the HOD processes. Make 
+                      sure that the users who will run hod have rights to create 
+                      directories under the directory specified here. If you
+                      wish to make this directory vary across allocations,
+                      you can make use of the environmental variables which will
+                      be made available by the resource manager to the HOD
+                      processes. For example, in a Torque setup, having
+                      --ringmaster.temp-dir=/tmp/hod-temp-dir.$PBS_JOBID would
+                      let ringmaster use different temp-dir for each
+                      allocation; Torque expands this variable before starting
+                      the ringmaster.</li>
+          
+          <li>debug: Numeric value from 1-4. 4 produces the most log information,
+                   and 1 the least.</li>
+          
+          <li>log-dir: Directory where log files are stored. By default, this is
+                     &lt;install-location&gt;/logs/. The restrictions and notes for the
+                     temp-dir variable apply here too.
+          </li>
+          
+          <li>xrs-port-range: Range of ports, among which an available port shall
+                            be picked for use to run an XML-RPC server.</li>
+          
+          <li>http-port-range: Range of ports, among which an available port shall
+                             be picked for use to run an HTTP server.</li>
+          
+          <li>java-home: Location of Java to be used by Hadoop.</li>
+          <li>syslog-address: Address to which a syslog daemon is bound to. The format 
+                              of the value is host:port. If configured, HOD log messages
+                              will be logged to syslog using this value.</li>
+                              
+        </ul>
+      </section>
+      
+      <section>
+        <title>3.2 hod options</title>
+        
+        <ul>
+          <li>cluster: Descriptive name given to the cluster. For Torque, this is
+                     specified as a 'Node property' for every node in the cluster.
+                     HOD uses this value to compute the number of available nodes.</li>
+          
+          <li>client-params: Comma-separated list of hadoop config parameters
+                           specified as key-value pairs. These will be used to
+                           generate a hadoop-site.xml on the submit node that 
+                           should be used for running Map/Reduce jobs.</li>
+          <li>job-feasibility-attr: Regular expression string that specifies
+                           whether and how to check job feasibility - resource
+                           manager or scheduler limits. The current
+                           implementation corresponds to the torque job
+                           attribute 'comment' and by default is disabled.
+                           When set, HOD uses it to decide what type
+                           of limit violation is triggered and either
+                           deallocates the cluster or stays in queued state
+                           according as the request is beyond maximum limits or
+                           the cumulative usage has crossed maximum limits. 
+                           The torque comment attribute may be updated
+                           periodically by an external mechanism. For example,
+                           comment attribute can be updated by running <a href=
+"hod_admin_guide.html#checklimits.sh+-+Tool+to+update+torque+comment+field+reflecting+resource+limits">
+                           checklimits.sh</a> script in hod/support directory,
+                           and then setting job-feasibility-attr equal to the
+                           value TORQUE_USER_LIMITS_COMMENT_FIELD,
+                           "User-limits exceeded. Requested:([0-9]*)
+                           Used:([0-9]*) MaxLimit:([0-9]*)", will make HOD
+                           behave accordingly.
+                           </li>
+         </ul>
+      </section>
+      
+      <section>
+        <title>3.3 resource_manager options</title>
+      
+        <ul>
+          <li>queue: Name of the queue configured in the resource manager to which
+                   jobs are to be submitted.</li>
+          
+          <li>batch-home: Install directory to which 'bin' is appended and under 
+                        which the executables of the resource manager can be 
+                        found.</li> 
+          
+          <li>env-vars: Comma-separated list of key-value pairs, 
+                      expressed as key=value, which would be passed to the jobs 
+                      launched on the compute nodes. 
+                      For example, if the python installation is 
+                      in a non-standard location, one can set the environment
+                      variable 'HOD_PYTHON_HOME' to the path to the python 
+                      executable. The HOD processes launched on the compute nodes
+                      can then use this variable.</li>
+          <li>options: Comma-separated list of key-value pairs,
+                      expressed as
+                      &lt;option&gt;:&lt;sub-option&gt;=&lt;value&gt;. When
+                      passing to the job submission program, these are expanded
+                      as -&lt;option&gt; &lt;sub-option&gt;=&lt;value&gt;. These
+                      are generally used for specifying additional resource
+                      contraints for scheduling. For instance, with a Torque
+                      setup, one can specify
+                      --resource_manager.options='l:arch=x86_64' for
+                      constraining the nodes being allocated to a particular
+                      architecture; this option will be passed to Torque's qsub
+                      command as "-l arch=x86_64".</li>
+        </ul>
+      </section>
+      
+      <section>
+        <title>3.4 ringmaster options</title>
+        
+        <ul>
+          <li>work-dirs: Comma-separated list of paths that will serve
+                       as the root for directories that HOD generates and passes
+                       to Hadoop for use to store DFS and Map/Reduce data. For
+                       example,
+                       this is where DFS data blocks will be stored. Typically,
+                       as many paths are specified as there are disks available
+                       to ensure all disks are being utilized. The restrictions
+                       and notes for the temp-dir variable apply here too.</li>
+          <li>max-master-failures: Number of times a hadoop master
+                       daemon can fail to launch, beyond which HOD will fail
+                       the cluster allocation altogether. In HOD clusters,
+                       sometimes there might be a single or few "bad" nodes due
+                       to issues like missing java, missing or incorrect version
+                       of Hadoop etc. When this configuration variable is set
+                       to a positive integer, the RingMaster returns an error
+                       to the client only when the number of times a hadoop
+                       master (JobTracker or NameNode) fails to start on these
+                       bad nodes because of above issues, exceeds the specified
+                       value. If the number is not exceeded, the next HodRing
+                       which requests for a command to launch is given the same
+                       hadoop master again. This way, HOD tries its best for a
+                       successful allocation even in the presence of a few bad
+                       nodes in the cluster.
+                       </li>
+          <li>workers_per_ring: Number of workers per service per HodRing.
+                       By default this is set to 1. If this configuration
+                       variable is set to a value 'n', the HodRing will run
+                       'n' instances of the workers (TaskTrackers or DataNodes)
+                       on each node acting as a slave. This can be used to run
+                       multiple workers per HodRing, so that the total number of
+                       workers  in a HOD cluster is not limited by the total
+                       number of nodes requested during allocation. However, note
+                       that this will mean each worker should be configured to use
+                       only a proportional fraction of the capacity of the 
+                       resources on the node. In general, this feature is only
+                       useful for testing and simulation purposes, and not for
+                       production use.</li>
+        </ul>
+      </section>
+      
+      <section>
+        <title>3.5 gridservice-hdfs options</title>
+        
+        <ul>
+          <li>external: If false, indicates that a HDFS cluster must be 
+                      bought up by the HOD system, on the nodes which it 
+                      allocates via the allocate command. Note that in that case,
+                      when the cluster is de-allocated, it will bring down the 
+                      HDFS cluster, and all the data will be lost.
+                      If true, it will try and connect to an externally configured
+                      HDFS system.
+                      Typically, because input for jobs are placed into HDFS
+                      before jobs are run, and also the output from jobs in HDFS 
+                      is required to be persistent, an internal HDFS cluster is 
+                      of little value in a production system. However, it allows 
+                      for quick testing.</li>
+          
+          <li>host: Hostname of the externally configured NameNode, if any</li>
+          
+          <li>fs_port: Port to which NameNode RPC server is bound.</li>
+          
+          <li>info_port: Port to which the NameNode web UI server is bound.</li>
+          
+          <li>pkgs: Installation directory, under which bin/hadoop executable is 
+                  located. This can be used to use a pre-installed version of
+                  Hadoop on the cluster.</li>
+          
+          <li>server-params: Comma-separated list of hadoop config parameters
+                           specified key-value pairs. These will be used to
+                           generate a hadoop-site.xml that will be used by the
+                           NameNode and DataNodes.</li>
+          
+          <li>final-server-params: Same as above, except they will be marked final.</li>
+        </ul>
+      </section>
+      
+      <section>
+        <title>3.6 gridservice-mapred options</title>
+        
+        <ul>
+          <li>external: If false, indicates that a Map/Reduce cluster must be
+                      bought up by the HOD system on the nodes which it allocates
+                      via the allocate command.
+                      If true, if will try and connect to an externally 
+                      configured Map/Reduce system.</li>
+          
+          <li>host: Hostname of the externally configured JobTracker, if any</li>
+          
+          <li>tracker_port: Port to which the JobTracker RPC server is bound</li>
+          
+          <li>info_port: Port to which the JobTracker web UI server is bound.</li>
+          
+          <li>pkgs: Installation directory, under which bin/hadoop executable is 
+                  located</li>
+          
+          <li>server-params: Comma-separated list of hadoop config parameters
+                           specified key-value pairs. These will be used to
+                           generate a hadoop-site.xml that will be used by the
+                           JobTracker and TaskTrackers</li>
+          
+          <li>final-server-params: Same as above, except they will be marked final.</li>
+        </ul>
+      </section>
+
+      <section>
+        <title>3.7 hodring options</title>
+
+        <ul>
+          <li>mapred-system-dir-root: Directory in the DFS under which HOD will
+                                      generate sub-directory names and pass the full path
+                                      as the value of the 'mapred.system.dir' configuration 
+                                      parameter to Hadoop daemons. The format of the full 
+                                      path will be value-of-this-option/userid/mapredsystem/cluster-id.
+                                      Note that the directory specified here should be such
+                                      that all users can create directories under this, if
+                                      permissions are enabled in HDFS. Setting the value of
+                                      this option to /user will make HOD use the user's
+                                      home directory to generate the mapred.system.dir value.</li>
+
+          <li>log-destination-uri: URL describing a path in an external, static DFS or the 
+                                   cluster node's local file system where HOD will upload 
+                                   Hadoop logs when a cluster is deallocated. To specify a 
+                                   DFS path, use the format 'hdfs://path'. To specify a 
+                                   cluster node's local file path, use the format 'file://path'.
+
+                                   When clusters are deallocated by HOD, the hadoop logs will
+                                   be deleted as part of HOD's cleanup process. To ensure these
+                                   logs persist, you can use this configuration option.
+
+                                   The format of the path is 
+                                   value-of-this-option/userid/hod-logs/cluster-id
+
+                                   Note that the directory you specify here must be such that all
+                                   users can create sub-directories under this. Setting this value
+                                   to hdfs://user will make the logs come in the user's home directory
+                                   in DFS.</li>
+
+          <li>pkgs: Installation directory, under which bin/hadoop executable is located. This will
+                    be used by HOD to upload logs if a HDFS URL is specified in log-destination-uri
+                    option. Note that this is useful if the users are using a tarball whose version
+                    may differ from the external, static HDFS version.</li>
+
+          <li>hadoop-port-range: Range of ports, among which an available port shall
+                             be picked for use to run a Hadoop Service, like JobTracker or TaskTracker. </li>
+          
+                                      
+        </ul>
+      </section>
+    </section>
+  </body>
+</document>
+

Propchange: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/hod_config_guide.xml
------------------------------------------------------------------------------
    svn:mime-type = text/plain



Mime
View raw message