Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8A336DD25 for ; Thu, 3 Jan 2013 21:24:37 +0000 (UTC) Received: (qmail 6092 invoked by uid 500); 3 Jan 2013 21:24:37 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 6030 invoked by uid 500); 3 Jan 2013 21:24:37 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 6019 invoked by uid 99); 3 Jan 2013 21:24:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 21:24:37 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 21:24:25 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id B9BE223889BF; Thu, 3 Jan 2013 21:24:03 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1428601 [2/2] - in /hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common: ./ dev-support/ src/main/bin/ src/main/docs/ src/main/docs/src/documentation/content/xdocs/ src/main/java/ src/main/java/org/apache/hadoop/fs/... Date: Thu, 03 Jan 2013 21:24:02 -0000 To: common-commits@hadoop.apache.org From: suresh@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20130103212403.B9BE223889BF@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm?rev=1428601&r1=1428600&r2=1428601&view=diff ============================================================================== --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm Thu Jan 3 21:23:58 2013 @@ -16,82 +16,82 @@ --- ${maven.build.timestamp} -Hadoop MapReduce Next Generation - Cluster Setup - - \[ {{{./index.html}Go Back}} \] + \[ {{{../index.html}Go Back}} \] %{toc|section=1|fromDepth=0} +Hadoop MapReduce Next Generation - Cluster Setup + * {Purpose} - This document describes how to install, configure and manage non-trivial - Hadoop clusters ranging from a few nodes to extremely large clusters + This document describes how to install, configure and manage non-trivial + Hadoop clusters ranging from a few nodes to extremely large clusters with thousands of nodes. - To play with Hadoop, you may first want to install it on a single + To play with Hadoop, you may first want to install it on a single machine (see {{{SingleCluster}Single Node Setup}}). - + * {Prerequisites} Download a stable version of Hadoop from Apache mirrors. - + * {Installation} - Installing a Hadoop cluster typically involves unpacking the software on all + Installing a Hadoop cluster typically involves unpacking the software on all the machines in the cluster or installing RPMs. - Typically one machine in the cluster is designated as the NameNode and - another machine the as ResourceManager, exclusively. These are the masters. - - The rest of the machines in the cluster act as both DataNode and NodeManager. + Typically one machine in the cluster is designated as the NameNode and + another machine the as ResourceManager, exclusively. These are the masters. + + The rest of the machines in the cluster act as both DataNode and NodeManager. These are the slaves. * {Running Hadoop in Non-Secure Mode} The following sections describe how to configure a Hadoop cluster. - * {Configuration Files} - + {Configuration Files} + Hadoop configuration is driven by two types of important configuration files: - * Read-only default configuration - <<>>, - <<>>, <<>> and + * Read-only default configuration - <<>>, + <<>>, <<>> and <<>>. - - * Site-specific configuration - <>, - <>, <> and + + * Site-specific configuration - <>, + <>, <> and <>. - Additionally, you can control the Hadoop scripts found in the bin/ - directory of the distribution, by setting site-specific values via the + Additionally, you can control the Hadoop scripts found in the bin/ + directory of the distribution, by setting site-specific values via the <> and <>. - * {Site Configuration} - - To configure the Hadoop cluster you will need to configure the - <<>> in which the Hadoop daemons execute as well as the + {Site Configuration} + + To configure the Hadoop cluster you will need to configure the + <<>> in which the Hadoop daemons execute as well as the <<>> for the Hadoop daemons. The Hadoop daemons are NameNode/DataNode and ResourceManager/NodeManager. - * {Configuring Environment of Hadoop Daemons} - - Administrators should use the <> and - <> script to do site-specific customization of the - Hadoop daemons' process environment. - - At the very least you should specify the <<>> so that it is - correctly defined on each remote node. - - In most cases you should also specify <<>> and - <<>> to point to directories that can only be - written to by the users that are going to run the hadoop daemons. - Otherwise there is the potential for a symlink attack. +** {Configuring Environment of Hadoop Daemons} - Administrators can configure individual daemons using the configuration - options shown below in the table: + Administrators should use the <> and + <> script to do site-specific customization of the + Hadoop daemons' process environment. + + At the very least you should specify the <<>> so that it is + correctly defined on each remote node. + + In most cases you should also specify <<>> and + <<>> to point to directories that can only be + written to by the users that are going to run the hadoop daemons. + Otherwise there is the potential for a symlink attack. + + Administrators can configure individual daemons using the configuration + options shown below in the table: *--------------------------------------+--------------------------------------+ || Daemon || Environment Variable | @@ -112,24 +112,25 @@ Hadoop MapReduce Next Generation - Clust *--------------------------------------+--------------------------------------+ - For example, To configure Namenode to use parallelGC, the following - statement should be added in hadoop-env.sh : - ----- - export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}" ----- - - Other useful configuration parameters that you can customize include: - - * <<>> / <<>> - The directory where the - daemons' log files are stored. They are automatically created if they - don't exist. - - * <<>> / <<>> - The maximum amount of - heapsize to use, in MB e.g. if the varibale is set to 1000 the heap - will be set to 1000MB. This is used to configure the heap - size for the daemon. By default, the value is 1000. If you want to - configure the values separately for each deamon you can use. + For example, To configure Namenode to use parallelGC, the following + statement should be added in hadoop-env.sh : + +---- + export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}" +---- + + Other useful configuration parameters that you can customize include: + + * <<>> / <<>> - The directory where the + daemons' log files are stored. They are automatically created if they + don't exist. + + * <<>> / <<>> - The maximum amount of + heapsize to use, in MB e.g. if the varibale is set to 1000 the heap + will be set to 1000MB. This is used to configure the heap + size for the daemon. By default, the value is 1000. If you want to + configure the values separately for each deamon you can use. + *--------------------------------------+--------------------------------------+ || Daemon || Environment Variable | *--------------------------------------+--------------------------------------+ @@ -141,14 +142,14 @@ Hadoop MapReduce Next Generation - Clust *--------------------------------------+--------------------------------------+ | Map Reduce Job History Server | HADOOP_JOB_HISTORYSERVER_HEAPSIZE | *--------------------------------------+--------------------------------------+ - - * {Configuring the Hadoop Daemons in Non-Secure Mode} - This section deals with important parameters to be specified in - the given configuration files: - - * <<>> - +** {Configuring the Hadoop Daemons in Non-Secure Mode} + + This section deals with important parameters to be specified in + the given configuration files: + + * <<>> + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ @@ -158,16 +159,16 @@ Hadoop MapReduce Next Generation - Clust | | | Size of read/write buffer used in SequenceFiles. | *-------------------------+-------------------------+------------------------+ - * <<>> - - * Configurations for NameNode: + * <<>> + + * Configurations for NameNode: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ -| <<>> | | | -| | Path on the local filesystem where the NameNode stores the namespace | | -| | and transactions logs persistently. | | +| <<>> | | | +| | Path on the local filesystem where the NameNode stores the namespace | | +| | and transactions logs persistently. | | | | | If this is a comma-delimited list of directories then the name table is | | | | replicated in all of the directories, for redundancy. | *-------------------------+-------------------------+------------------------+ @@ -177,28 +178,28 @@ Hadoop MapReduce Next Generation - Clust | | | datanodes. | *-------------------------+-------------------------+------------------------+ | <<>> | 268435456 | | -| | | HDFS blocksize of 256MB for large file-systems. | +| | | HDFS blocksize of 256MB for large file-systems. | *-------------------------+-------------------------+------------------------+ | <<>> | 100 | | | | | More NameNode server threads to handle RPCs from large number of | | | | DataNodes. | *-------------------------+-------------------------+------------------------+ - - * Configurations for DataNode: - + + * Configurations for DataNode: + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ | <<>> | | | -| | Comma separated list of paths on the local filesystem of a | | +| | Comma separated list of paths on the local filesystem of a | | | | <<>> where it should store its blocks. | | -| | | If this is a comma-delimited list of directories, then data will be | +| | | If this is a comma-delimited list of directories, then data will be | | | | stored in all named directories, typically on different devices. | *-------------------------+-------------------------+------------------------+ - - * <<>> - * Configurations for ResourceManager and NodeManager: + * <<>> + + * Configurations for ResourceManager and NodeManager: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -220,29 +221,29 @@ Hadoop MapReduce Next Generation - Clust *-------------------------+-------------------------+------------------------+ - * Configurations for ResourceManager: + * Configurations for ResourceManager: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | <<>> host:port for clients to submit jobs. | | | | | | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | <<>> host:port for ApplicationMasters to talk to | | | | Scheduler to obtain resources. | | | | | | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | <<>> host:port for NodeManagers. | | | | | | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | <<>> host:port for administrative commands. | | | | | | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | <<>> web-ui host:port. | | | | | | *-------------------------+-------------------------+------------------------+ @@ -258,14 +259,14 @@ Hadoop MapReduce Next Generation - Clust | | Maximum limit of memory to allocate to each container request at the <<>>. | | | | | In MBs | *-------------------------+-------------------------+------------------------+ -| <<>> / | | | -| <<>> | | | +| <<>> / | | | +| <<>> | | | | | List of permitted/excluded NodeManagers. | | -| | | If necessary, use these files to control the list of allowable | +| | | If necessary, use these files to control the list of allowable | | | | NodeManagers. | *-------------------------+-------------------------+------------------------+ - - * Configurations for NodeManager: + + * Configurations for NodeManager: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -314,7 +315,7 @@ Hadoop MapReduce Next Generation - Clust | | | Shuffle service that needs to be set for Map Reduce applications. | *-------------------------+-------------------------+------------------------+ - * Configurations for History Server (Needs to be moved elsewhere): + * Configurations for History Server (Needs to be moved elsewhere): *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -327,9 +328,9 @@ Hadoop MapReduce Next Generation - Clust - * <<>> + * <<>> - * Configurations for MapReduce Applications: + * Configurations for MapReduce Applications: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -361,7 +362,7 @@ Hadoop MapReduce Next Generation - Clust | | | from very large number of maps. | *-------------------------+-------------------------+------------------------+ - * Configurations for MapReduce JobHistory Server: + * Configurations for MapReduce JobHistory Server: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -373,51 +374,51 @@ Hadoop MapReduce Next Generation - Clust | | MapReduce JobHistory Server Web UI | Default port is 19888. | *-------------------------+-------------------------+------------------------+ | <<>> | /mr-history/tmp | | -| | | Directory where history files are written by MapReduce jobs. | +| | | Directory where history files are written by MapReduce jobs. | *-------------------------+-------------------------+------------------------+ | <<>> | /mr-history/done| | -| | | Directory where history files are managed by the MR JobHistory Server. | +| | | Directory where history files are managed by the MR JobHistory Server. | *-------------------------+-------------------------+------------------------+ - * Hadoop Rack Awareness - - The HDFS and the YARN components are rack-aware. - - The NameNode and the ResourceManager obtains the rack information of the - slaves in the cluster by invoking an API in an administrator - configured module. - - The API resolves the DNS name (also IP address) to a rack id. - - The site-specific module to use can be configured using the configuration - item <<>>. The default implementation - of the same runs a script/command configured using - <<>>. If <<>> is - not set, the rack id is returned for any passed IP address. - - * Monitoring Health of NodeManagers - - Hadoop provides a mechanism by which administrators can configure the - NodeManager to run an administrator supplied script periodically to - determine if a node is healthy or not. - - Administrators can determine if the node is in a healthy state by - performing any checks of their choice in the script. If the script - detects the node to be in an unhealthy state, it must print a line to - standard output beginning with the string ERROR. The NodeManager spawns - the script periodically and checks its output. If the script's output - contains the string ERROR, as described above, the node's status is - reported as <<>> and the node is black-listed by the - ResourceManager. No further tasks will be assigned to this node. - However, the NodeManager continues to run the script, so that if the - node becomes healthy again, it will be removed from the blacklisted nodes - on the ResourceManager automatically. The node's health along with the - output of the script, if it is unhealthy, is available to the - administrator in the ResourceManager web interface. The time since the - node was healthy is also displayed on the web interface. +* {Hadoop Rack Awareness} + + The HDFS and the YARN components are rack-aware. + + The NameNode and the ResourceManager obtains the rack information of the + slaves in the cluster by invoking an API in an administrator + configured module. + + The API resolves the DNS name (also IP address) to a rack id. + + The site-specific module to use can be configured using the configuration + item <<>>. The default implementation + of the same runs a script/command configured using + <<>>. If <<>> is + not set, the rack id is returned for any passed IP address. + +* {Monitoring Health of NodeManagers} + + Hadoop provides a mechanism by which administrators can configure the + NodeManager to run an administrator supplied script periodically to + determine if a node is healthy or not. + + Administrators can determine if the node is in a healthy state by + performing any checks of their choice in the script. If the script + detects the node to be in an unhealthy state, it must print a line to + standard output beginning with the string ERROR. The NodeManager spawns + the script periodically and checks its output. If the script's output + contains the string ERROR, as described above, the node's status is + reported as <<>> and the node is black-listed by the + ResourceManager. No further tasks will be assigned to this node. + However, the NodeManager continues to run the script, so that if the + node becomes healthy again, it will be removed from the blacklisted nodes + on the ResourceManager automatically. The node's health along with the + output of the script, if it is unhealthy, is available to the + administrator in the ResourceManager web interface. The time since the + node was healthy is also displayed on the web interface. - The following parameters can be used to control the node health - monitoring script in <<>>. + The following parameters can be used to control the node health + monitoring script in <<>>. *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -439,164 +440,163 @@ Hadoop MapReduce Next Generation - Clust | | | Timeout for health script execution. | *-------------------------+-------------------------+------------------------+ - The health checker script is not supposed to give ERROR if only some of the - local disks become bad. NodeManager has the ability to periodically check - the health of the local disks (specifically checks nodemanager-local-dirs - and nodemanager-log-dirs) and after reaching the threshold of number of - bad directories based on the value set for the config property - yarn.nodemanager.disk-health-checker.min-healthy-disks, the whole node is - marked unhealthy and this info is sent to resource manager also. The boot - disk is either raided or a failure in the boot disk is identified by the - health checker script. - - * {Slaves file} - - Typically you choose one machine in the cluster to act as the NameNode and - one machine as to act as the ResourceManager, exclusively. The rest of the - machines act as both a DataNode and NodeManager and are referred to as - . - - List all slave hostnames or IP addresses in your <<>> file, - one per line. - - * {Logging} - - Hadoop uses the Apache log4j via the Apache Commons Logging framework for - logging. Edit the <<>> file to customize the - Hadoop daemons' logging configuration (log-formats and so on). - - * {Operating the Hadoop Cluster} + The health checker script is not supposed to give ERROR if only some of the + local disks become bad. NodeManager has the ability to periodically check + the health of the local disks (specifically checks nodemanager-local-dirs + and nodemanager-log-dirs) and after reaching the threshold of number of + bad directories based on the value set for the config property + yarn.nodemanager.disk-health-checker.min-healthy-disks, the whole node is + marked unhealthy and this info is sent to resource manager also. The boot + disk is either raided or a failure in the boot disk is identified by the + health checker script. + +* {Slaves file} + + Typically you choose one machine in the cluster to act as the NameNode and + one machine as to act as the ResourceManager, exclusively. The rest of the + machines act as both a DataNode and NodeManager and are referred to as + . + + List all slave hostnames or IP addresses in your <<>> file, + one per line. + +* {Logging} + + Hadoop uses the Apache log4j via the Apache Commons Logging framework for + logging. Edit the <<>> file to customize the + Hadoop daemons' logging configuration (log-formats and so on). - Once all the necessary configuration is complete, distribute the files to the +* {Operating the Hadoop Cluster} + + Once all the necessary configuration is complete, distribute the files to the <<>> directory on all the machines. - * Hadoop Startup - - To start a Hadoop cluster you will need to start both the HDFS and YARN - cluster. +** Hadoop Startup + + To start a Hadoop cluster you will need to start both the HDFS and YARN + cluster. + + Format a new distributed filesystem: - Format a new distributed filesystem: - ---- - $ $HADOOP_PREFIX/bin/hdfs namenode -format +$ $HADOOP_PREFIX/bin/hdfs namenode -format ---- - Start the HDFS with the following command, run on the designated NameNode: - + Start the HDFS with the following command, run on the designated NameNode: + ---- - $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode ----- +$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode +---- - Run a script to start DataNodes on all slaves: + Run a script to start DataNodes on all slaves: ---- - $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode ----- - - Start the YARN with the following command, run on the designated - ResourceManager: - +$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode +---- + + Start the YARN with the following command, run on the designated + ResourceManager: + ---- - $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager ----- +$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager +---- - Run a script to start NodeManagers on all slaves: + Run a script to start NodeManagers on all slaves: ---- - $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager ----- +$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager +---- - Start a standalone WebAppProxy server. If multiple servers - are used with load balancing it should be run on each of them: + Start a standalone WebAppProxy server. If multiple servers + are used with load balancing it should be run on each of them: ---- - $ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR +$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR ---- - Start the MapReduce JobHistory Server with the following command, run on the - designated server: - + Start the MapReduce JobHistory Server with the following command, run on the + designated server: + ---- - $ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR ----- +$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR +---- + +** Hadoop Shutdown - * Hadoop Shutdown + Stop the NameNode with the following command, run on the designated + NameNode: - Stop the NameNode with the following command, run on the designated - NameNode: - ---- - $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode ----- +$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode +---- - Run a script to stop DataNodes on all slaves: + Run a script to stop DataNodes on all slaves: ---- - $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode ----- - - Stop the ResourceManager with the following command, run on the designated - ResourceManager: - +$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode +---- + + Stop the ResourceManager with the following command, run on the designated + ResourceManager: + ---- - $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager ----- +$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager +---- - Run a script to stop NodeManagers on all slaves: + Run a script to stop NodeManagers on all slaves: ---- - $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager ----- +$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager +---- - Stop the WebAppProxy server. If multiple servers are used with load - balancing it should be run on each of them: + Stop the WebAppProxy server. If multiple servers are used with load + balancing it should be run on each of them: ---- - $ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR +$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR ---- - Stop the MapReduce JobHistory Server with the following command, run on the - designated server: - + Stop the MapReduce JobHistory Server with the following command, run on the + designated server: + ---- - $ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR ----- +$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR +---- - * {Running Hadoop in Secure Mode} - This section deals with important parameters to be specified in + This section deals with important parameters to be specified in to run Hadoop in <> with strong, Kerberos-based authentication. - + * <<>> - + Ensure that HDFS and YARN daemons run as different Unix users, for e.g. <<>> and <<>>. Also, ensure that the MapReduce JobHistory - server runs as user <<>>. - + server runs as user <<>>. + It's recommended to have them share a Unix group, for e.g. <<>>. - -*--------------------------------------+----------------------------------------------------------------------+ -|| User:Group || Daemons | -*--------------------------------------+----------------------------------------------------------------------+ -| hdfs:hadoop | NameNode, Secondary NameNode, Checkpoint Node, Backup Node, DataNode | -*--------------------------------------+----------------------------------------------------------------------+ -| yarn:hadoop | ResourceManager, NodeManager | -*--------------------------------------+----------------------------------------------------------------------+ -| mapred:hadoop | MapReduce JobHistory Server | -*--------------------------------------+----------------------------------------------------------------------+ - + +*---------------+----------------------------------------------------------------------+ +|| User:Group || Daemons | +*---------------+----------------------------------------------------------------------+ +| hdfs:hadoop | NameNode, Secondary NameNode, Checkpoint Node, Backup Node, DataNode | +*---------------+----------------------------------------------------------------------+ +| yarn:hadoop | ResourceManager, NodeManager | +*---------------+----------------------------------------------------------------------+ +| mapred:hadoop | MapReduce JobHistory Server | +*---------------+----------------------------------------------------------------------+ + * <<>> - + The following table lists various paths on HDFS and local filesystems (on all nodes) and recommended permissions: - + *-------------------+-------------------+------------------+------------------+ || Filesystem || Path || User:Group || Permissions | *-------------------+-------------------+------------------+------------------+ -| local | <<>> | hdfs:hadoop | drwx------ | +| local | <<>> | hdfs:hadoop | drwx------ | *-------------------+-------------------+------------------+------------------+ | local | <<>> | hdfs:hadoop | drwx------ | *-------------------+-------------------+------------------+------------------+ @@ -621,123 +621,111 @@ Hadoop MapReduce Next Generation - Clust | hdfs | <<>> | yarn:hadoop | drwxrwxrwxt | *-------------------+-------------------+------------------+------------------+ | hdfs | <<>> | mapred:hadoop | | -| | | | drwxrwxrwxt | +| | | | drwxrwxrwxt | *-------------------+-------------------+------------------+------------------+ | hdfs | <<>> | mapred:hadoop | | -| | | | drwxr-x--- | +| | | | drwxr-x--- | *-------------------+-------------------+------------------+------------------+ * Kerberos Keytab files - + * HDFS - - The NameNode keytab file, on the NameNode host, should look like the + + The NameNode keytab file, on the NameNode host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nn.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nn.service.keytab Keytab name: FILE:/etc/security/keytab/nn.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- - The Secondary NameNode keytab file, on that host, should look like the + The Secondary NameNode keytab file, on that host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/sn.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/sn.service.keytab Keytab name: FILE:/etc/security/keytab/sn.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- The DataNode keytab file, on each host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/dn.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/dn.service.keytab Keytab name: FILE:/etc/security/keytab/dn.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- - + * YARN - - The ResourceManager keytab file, on the ResourceManager host, should look + + The ResourceManager keytab file, on the ResourceManager host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/rm.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/rm.service.keytab Keytab name: FILE:/etc/security/keytab/rm.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- The NodeManager keytab file, on each host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nm.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nm.service.keytab Keytab name: FILE:/etc/security/keytab/nm.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- - + * MapReduce JobHistory Server - The MapReduce JobHistory Server keytab file, on that host, should look + The MapReduce JobHistory Server keytab file, on that host, should look like the following: - ----- -$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/jhs.service.keytab +---- +$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/jhs.service.keytab Keytab name: FILE:/etc/security/keytab/jhs.service.keytab KVNO Timestamp Principal - 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) - 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) - + 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC) + 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) ---- - - * Configuration in Secure Mode - - * <<>> + +** Configuration in Secure Mode + + * <<>> *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -748,10 +736,10 @@ KVNO Timestamp Principal | | | Enable RPC service-level authorization. | *-------------------------+-------------------------+------------------------+ - * <<>> - - * Configurations for NameNode: - + * <<>> + + * Configurations for NameNode: + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ @@ -774,8 +762,8 @@ KVNO Timestamp Principal | | | HTTPS Kerberos principal name for the NameNode. | *-------------------------+-------------------------+------------------------+ - * Configurations for Secondary NameNode: - + * Configurations for Secondary NameNode: + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ @@ -783,7 +771,7 @@ KVNO Timestamp Principal *-------------------------+-------------------------+------------------------+ | <<>> | <50470> | | *-------------------------+-------------------------+------------------------+ -| <<>> | | | +| <<>> | | | | | | | | | | Kerberos keytab file for the NameNode. | *-------------------------+-------------------------+------------------------+ @@ -795,7 +783,7 @@ KVNO Timestamp Principal | | | HTTPS Kerberos principal name for the Secondary NameNode. | *-------------------------+-------------------------+------------------------+ - * Configurations for DataNode: + * Configurations for DataNode: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -817,15 +805,15 @@ KVNO Timestamp Principal | | | HTTPS Kerberos principal name for the DataNode. | *-------------------------+-------------------------+------------------------+ - * <<>> - - * WebAppProxy + * <<>> - The <<>> provides a proxy between the web applications - exported by an application and an end user. If security is enabled - it will warn users before accessing a potentially unsafe web application. - Authentication and authorization using the proxy is handled just like - any other privileged web application. + * WebAppProxy + + The <<>> provides a proxy between the web applications + exported by an application and an end user. If security is enabled + it will warn users before accessing a potentially unsafe web application. + Authentication and authorization using the proxy is handled just like + any other privileged web application. *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -844,12 +832,12 @@ KVNO Timestamp Principal | | | Kerberos principal name for the WebAppProxy. | *-------------------------+-------------------------+------------------------+ - * LinuxContainerExecutor - - A <<>> used by YARN framework which define how any - launched and controlled. - - The following are the available in Hadoop YARN: + * LinuxContainerExecutor + + A <<>> used by YARN framework which define how any + launched and controlled. + + The following are the available in Hadoop YARN: *--------------------------------------+--------------------------------------+ || ContainerExecutor || Description | @@ -859,7 +847,7 @@ KVNO Timestamp Principal | | The container process has the same Unix user as the NodeManager. | *--------------------------------------+--------------------------------------+ | <<>> | | -| | Supported only on GNU/Linux, this executor runs the containers as the | +| | Supported only on GNU/Linux, this executor runs the containers as the | | | user who submitted the application. It requires all user accounts to be | | | created on the cluster nodes where the containers are launched. It uses | | | a executable that is included in the Hadoop distribution. | @@ -874,53 +862,53 @@ KVNO Timestamp Principal | | localized as part of the distributed cache. | *--------------------------------------+--------------------------------------+ - To build the LinuxContainerExecutor executable run: - + To build the LinuxContainerExecutor executable run: + ---- $ mvn package -Dcontainer-executor.conf.dir=/etc/hadoop/ ---- - - The path passed in <<<-Dcontainer-executor.conf.dir>>> should be the - path on the cluster nodes where a configuration file for the setuid - executable should be located. The executable should be installed in - $HADOOP_YARN_HOME/bin. - - The executable must have specific permissions: 6050 or --Sr-s--- - permissions user-owned by (super-user) and group-owned by a - special group (e.g. <<>>) of which the NodeManager Unix user is - the group member and no ordinary application user is. If any application - user belongs to this special group, security will be compromised. This - special group name should be specified for the configuration property - <<>> in both - <<>> and <<>>. - - For example, let's say that the NodeManager is run as user who is - part of the groups users and , any of them being the primary group. - Let also be that has both and another user - (application submitter) as its members, and does not - belong to . Going by the above description, the setuid/setgid - executable should be set 6050 or --Sr-s--- with user-owner as and - group-owner as which has as its member (and not - which has also as its member besides ). - - The LinuxTaskController requires that paths including and leading up to - the directories specified in <<>> and - <<>> to be set 755 permissions as described - above in the table on permissions on directories. - - * <<>> - - The executable requires a configuration file called - <<>> to be present in the configuration - directory passed to the mvn target mentioned above. - - The configuration file must be owned by the user running NodeManager - (user <<>> in the above example), group-owned by anyone and - should have the permissions 0400 or r--------. - - The executable requires following configuration items to be present - in the <<>> file. The items should be - mentioned as simple key=value pairs, one per-line: + + The path passed in <<<-Dcontainer-executor.conf.dir>>> should be the + path on the cluster nodes where a configuration file for the setuid + executable should be located. The executable should be installed in + $HADOOP_YARN_HOME/bin. + + The executable must have specific permissions: 6050 or --Sr-s--- + permissions user-owned by (super-user) and group-owned by a + special group (e.g. <<>>) of which the NodeManager Unix user is + the group member and no ordinary application user is. If any application + user belongs to this special group, security will be compromised. This + special group name should be specified for the configuration property + <<>> in both + <<>> and <<>>. + + For example, let's say that the NodeManager is run as user who is + part of the groups users and , any of them being the primary group. + Let also be that has both and another user + (application submitter) as its members, and does not + belong to . Going by the above description, the setuid/setgid + executable should be set 6050 or --Sr-s--- with user-owner as and + group-owner as which has as its member (and not + which has also as its member besides ). + + The LinuxTaskController requires that paths including and leading up to + the directories specified in <<>> and + <<>> to be set 755 permissions as described + above in the table on permissions on directories. + + * <<>> + + The executable requires a configuration file called + <<>> to be present in the configuration + directory passed to the mvn target mentioned above. + + The configuration file must be owned by the user running NodeManager + (user <<>> in the above example), group-owned by anyone and + should have the permissions 0400 or r--------. + + The executable requires following configuration items to be present + in the <<>> file. The items should be + mentioned as simple key=value pairs, one per-line: *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | @@ -930,16 +918,16 @@ KVNO Timestamp Principal | | | binary should be this group. Should be same as the | | | | value with which the NodeManager is configured. This configuration is | | | | required for validating the secure access of the | -| | | binary. | +| | | binary. | *-------------------------+-------------------------+------------------------+ | <<>> | hfds,yarn,mapred,bin | Banned users. | *-------------------------+-------------------------+------------------------+ -| <<>> | 1000 | Prevent other super-users. | +| <<>> | 1000 | Prevent other super-users. | *-------------------------+-------------------------+------------------------+ - To re-cap, here are the local file-ssytem permissions required for the + To re-cap, here are the local file-sysytem permissions required for the various paths related to the <<>>: - + *-------------------+-------------------+------------------+------------------+ || Filesystem || Path || User:Group || Permissions | *-------------------+-------------------+------------------+------------------+ @@ -951,9 +939,9 @@ KVNO Timestamp Principal *-------------------+-------------------+------------------+------------------+ | local | <<>> | yarn:hadoop | drwxr-xr-x | *-------------------+-------------------+------------------+------------------+ - + * Configurations for ResourceManager: - + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ @@ -964,9 +952,9 @@ KVNO Timestamp Principal | <<>> | rm/_HOST@REALM.TLD | | | | | Kerberos principal name for the ResourceManager. | *-------------------------+-------------------------+------------------------+ - + * Configurations for NodeManager: - + *-------------------------+-------------------------+------------------------+ || Parameter || Value || Notes | *-------------------------+-------------------------+------------------------+ @@ -977,15 +965,15 @@ KVNO Timestamp Principal | | | Kerberos principal name for the NodeManager. | *-------------------------+-------------------------+------------------------+ | <<>> | | | -| | <<>> | -| | | Use LinuxContainerExecutor. | +| | <<>> | +| | | Use LinuxContainerExecutor. | *-------------------------+-------------------------+------------------------+ | <<>> | | | | | | Unix group of the NodeManager. | *-------------------------+-------------------------+------------------------+ * <<>> - + * Configurations for MapReduce JobHistory Server: *-------------------------+-------------------------+------------------------+ @@ -1002,116 +990,116 @@ KVNO Timestamp Principal | | | Kerberos principal name for the MapReduce JobHistory Server. | *-------------------------+-------------------------+------------------------+ - - * {Operating the Hadoop Cluster} - Once all the necessary configuration is complete, distribute the files to the +* {Operating the Hadoop Cluster} + + Once all the necessary configuration is complete, distribute the files to the <<>> directory on all the machines. This section also describes the various Unix users who should be starting the various components and uses the same Unix accounts and groups used previously: - - * Hadoop Startup - - To start a Hadoop cluster you will need to start both the HDFS and YARN + +** Hadoop Startup + + To start a Hadoop cluster you will need to start both the HDFS and YARN cluster. Format a new distributed filesystem as : - + ---- [hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format ---- Start the HDFS with the following command, run on the designated NameNode as : - + ---- [hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode ----- +---- Run a script to start DataNodes on all slaves as with a special environment variable <<>> set to : ---- [root]$ HADOOP_SECURE_DN_USER=hdfs $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode ----- - - Start the YARN with the following command, run on the designated +---- + + Start the YARN with the following command, run on the designated ResourceManager as : - + ---- -[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager ----- +[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager +---- Run a script to start NodeManagers on all slaves as : ---- -[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager ----- +[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager +---- - Start a standalone WebAppProxy server. Run on the WebAppProxy + Start a standalone WebAppProxy server. Run on the WebAppProxy server as . If multiple servers are used with load balancing it should be run on each of them: ---- -[yarn]$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR ----- +[yarn]$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR +---- - Start the MapReduce JobHistory Server with the following command, run on the + Start the MapReduce JobHistory Server with the following command, run on the designated server as : - + ---- -[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR ----- +[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR +---- - * Hadoop Shutdown +** Hadoop Shutdown + + Stop the NameNode with the following command, run on the designated NameNode + as : - Stop the NameNode with the following command, run on the designated NameNode - as : - ---- [hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode ----- +---- - Run a script to stop DataNodes on all slaves as : + Run a script to stop DataNodes on all slaves as : ---- [root]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode ----- - - Stop the ResourceManager with the following command, run on the designated - ResourceManager as : - +---- + + Stop the ResourceManager with the following command, run on the designated + ResourceManager as : + ---- -[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager ----- +[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager +---- - Run a script to stop NodeManagers on all slaves as : + Run a script to stop NodeManagers on all slaves as : ---- -[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager ----- +[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager +---- - Stop the WebAppProxy server. Run on the WebAppProxy server as - . If multiple servers are used with load balancing it - should be run on each of them: + Stop the WebAppProxy server. Run on the WebAppProxy server as + . If multiple servers are used with load balancing it + should be run on each of them: ---- -[yarn]$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR +[yarn]$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR ---- - Stop the MapReduce JobHistory Server with the following command, run on the - designated server as : + Stop the MapReduce JobHistory Server with the following command, run on the + designated server as : ---- -[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR ----- - -* {Web Interfaces} - - Once the Hadoop cluster is up and running check the web-ui of the - components as described below: - +[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR +---- + +* {Web Interfaces} + + Once the Hadoop cluster is up and running check the web-ui of the + components as described below: + *-------------------------+-------------------------+------------------------+ || Daemon || Web Interface || Notes | *-------------------------+-------------------------+------------------------+ @@ -1122,5 +1110,5 @@ KVNO Timestamp Principal | MapReduce JobHistory Server | http:/// | | | | | Default HTTP port is 19888. | *-------------------------+-------------------------+------------------------+ - - + + Propchange: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/core/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core:r1423068-1428155 Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDU.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDU.java?rev=1428601&r1=1428600&r2=1428601&view=diff ============================================================================== --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDU.java (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDU.java Thu Jan 3 21:23:58 2013 @@ -103,4 +103,12 @@ public class TestDU extends TestCase { duSize >= writtenSize && writtenSize <= (duSize + slack)); } + public void testDUGetUsedWillNotReturnNegative() throws IOException { + File file = new File(DU_DIR, "data"); + assertTrue(file.createNewFile()); + DU du = new DU(file, 10000); + du.decDfsUsed(Long.MAX_VALUE); + long duSize = du.getUsed(); + assertTrue(String.valueOf(duSize), duSize >= 0L); + } } Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileStatus.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileStatus.java?rev=1428601&r1=1428600&r2=1428601&view=diff ============================================================================== --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileStatus.java (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileStatus.java Thu Jan 3 21:23:58 2013 @@ -25,18 +25,34 @@ import java.io.DataInput; import java.io.DataInputStream; import java.io.DataOutput; import java.io.DataOutputStream; +import java.io.IOException; import org.junit.Test; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; public class TestFileStatus { private static final Log LOG = LogFactory.getLog(TestFileStatus.class); - + + /** Values for creating {@link FileStatus} in some tests */ + static final int LENGTH = 1; + static final int REPLICATION = 2; + static final long BLKSIZE = 3; + static final long MTIME = 4; + static final long ATIME = 5; + static final String OWNER = "owner"; + static final String GROUP = "group"; + static final FsPermission PERMISSION = FsPermission.valueOf("-rw-rw-rw-"); + static final Path PATH = new Path("path"); + + /** + * Check that the write and readField methods work correctly. + */ @Test public void testFileStatusWritable() throws Exception { FileStatus[] tests = { @@ -68,4 +84,181 @@ public class TestFileStatus { iterator++; } } + + /** + * Check that the full parameter constructor works correctly. + */ + @Test + public void constructorFull() throws IOException { + boolean isdir = false; + Path symlink = new Path("symlink"); + FileStatus fileStatus = new FileStatus(LENGTH, isdir, REPLICATION, BLKSIZE, + MTIME, ATIME, PERMISSION, OWNER, GROUP, symlink, PATH); + + validateAccessors(fileStatus, LENGTH, isdir, REPLICATION, BLKSIZE, MTIME, + ATIME, PERMISSION, OWNER, GROUP, symlink, PATH); + } + + /** + * Check that the non-symlink constructor works correctly. + */ + @Test + public void constructorNoSymlink() throws IOException { + boolean isdir = true; + FileStatus fileStatus = new FileStatus(LENGTH, isdir, REPLICATION, BLKSIZE, + MTIME, ATIME, PERMISSION, OWNER, GROUP, PATH); + validateAccessors(fileStatus, LENGTH, isdir, REPLICATION, BLKSIZE, MTIME, + ATIME, PERMISSION, OWNER, GROUP, null, PATH); + } + + /** + * Check that the constructor without owner, group and permissions works + * correctly. + */ + @Test + public void constructorNoOwner() throws IOException { + boolean isdir = true; + FileStatus fileStatus = new FileStatus(LENGTH, isdir, + REPLICATION, BLKSIZE, MTIME, PATH); + validateAccessors(fileStatus, LENGTH, isdir, REPLICATION, BLKSIZE, MTIME, + 0, FsPermission.getDefault(), "", "", null, PATH); + } + + /** + * Check that the no parameter constructor works correctly. + */ + @Test + public void constructorBlank() throws IOException { + FileStatus fileStatus = new FileStatus(); + validateAccessors(fileStatus, 0, false, 0, 0, 0, + 0, FsPermission.getDefault(), "", "", null, null); + } + + /** + * Check that FileStatus are equal if their paths are equal. + */ + @Test + public void testEquals() { + Path path = new Path("path"); + FileStatus fileStatus1 = new FileStatus(1, true, 1, 1, 1, 1, + FsPermission.valueOf("-rw-rw-rw-"), "one", "one", null, path); + FileStatus fileStatus2 = new FileStatus(2, true, 2, 2, 2, 2, + FsPermission.valueOf("---x--x--x"), "two", "two", null, path); + assertEquals(fileStatus1, fileStatus2); + } + + /** + * Check that FileStatus are not equal if their paths are not equal. + */ + @Test + public void testNotEquals() { + Path path1 = new Path("path1"); + Path path2 = new Path("path2"); + FileStatus fileStatus1 = new FileStatus(1, true, 1, 1, 1, 1, + FsPermission.valueOf("-rw-rw-rw-"), "one", "one", null, path1); + FileStatus fileStatus2 = new FileStatus(1, true, 1, 1, 1, 1, + FsPermission.valueOf("-rw-rw-rw-"), "one", "one", null, path2); + assertFalse(fileStatus1.equals(fileStatus2)); + assertFalse(fileStatus2.equals(fileStatus1)); + } + + /** + * Check that toString produces the expected output for a file. + */ + @Test + public void toStringFile() throws IOException { + boolean isdir = false; + FileStatus fileStatus = new FileStatus(LENGTH, isdir, REPLICATION, BLKSIZE, + MTIME, ATIME, PERMISSION, OWNER, GROUP, null, PATH); + validateToString(fileStatus); + } + + /** + * Check that toString produces the expected output for a directory. + */ + @Test + public void toStringDir() throws IOException { + FileStatus fileStatus = new FileStatus(LENGTH, true, REPLICATION, BLKSIZE, + MTIME, ATIME, PERMISSION, OWNER, GROUP, null, PATH); + validateToString(fileStatus); + } + + /** + * Check that toString produces the expected output for a symlink. + */ + @Test + public void toStringSymlink() throws IOException { + boolean isdir = false; + Path symlink = new Path("symlink"); + FileStatus fileStatus = new FileStatus(LENGTH, isdir, REPLICATION, BLKSIZE, + MTIME, ATIME, PERMISSION, OWNER, GROUP, symlink, PATH); + validateToString(fileStatus); + } + + /** + * Validate the accessors for FileStatus. + * @param fileStatus FileStatus to checked + * @param length expected length + * @param isdir expected isDirectory + * @param replication expected replication + * @param blocksize expected blocksize + * @param mtime expected modification time + * @param atime expected access time + * @param permission expected permission + * @param owner expected owner + * @param group expected group + * @param symlink expected symlink + * @param path expected path + */ + private void validateAccessors(FileStatus fileStatus, + long length, boolean isdir, int replication, long blocksize, long mtime, + long atime, FsPermission permission, String owner, String group, + Path symlink, Path path) throws IOException { + + assertEquals(length, fileStatus.getLen()); + assertEquals(isdir, fileStatus.isDirectory()); + assertEquals(replication, fileStatus.getReplication()); + assertEquals(blocksize, fileStatus.getBlockSize()); + assertEquals(mtime, fileStatus.getModificationTime()); + assertEquals(atime, fileStatus.getAccessTime()); + assertEquals(permission, fileStatus.getPermission()); + assertEquals(owner, fileStatus.getOwner()); + assertEquals(group, fileStatus.getGroup()); + if(symlink == null) { + assertFalse(fileStatus.isSymlink()); + } else { + assertTrue(fileStatus.isSymlink()); + assertEquals(symlink, fileStatus.getSymlink()); + } + assertEquals(path, fileStatus.getPath()); + } + + /** + * Validates the toString method for FileStatus. + * @param fileStatus FileStatus to be validated + */ + private void validateToString(FileStatus fileStatus) throws IOException { + StringBuilder expected = new StringBuilder(); + expected.append("FileStatus{"); + expected.append("path=").append(fileStatus.getPath()).append("; "); + expected.append("isDirectory=").append(fileStatus.isDirectory()).append("; "); + if(!fileStatus.isDirectory()) { + expected.append("length=").append(fileStatus.getLen()).append("; "); + expected.append("replication=").append(fileStatus.getReplication()).append("; "); + expected.append("blocksize=").append(fileStatus.getBlockSize()).append("; "); + } + expected.append("modification_time=").append(fileStatus.getModificationTime()).append("; "); + expected.append("access_time=").append(fileStatus.getAccessTime()).append("; "); + expected.append("owner=").append(fileStatus.getOwner()).append("; "); + expected.append("group=").append(fileStatus.getGroup()).append("; "); + expected.append("permission=").append(fileStatus.getPermission()).append("; "); + if(fileStatus.isSymlink()) { + expected.append("isSymlink=").append(true).append("; "); + expected.append("symlink=").append(fileStatus.getSymlink()).append("}"); + } else { + expected.append("isSymlink=").append(false).append("}"); + } + + assertEquals(expected.toString(), fileStatus.toString()); + } } Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java?rev=1428601&r1=1428600&r2=1428601&view=diff ============================================================================== --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java Thu Jan 3 21:23:58 2013 @@ -357,6 +357,66 @@ public class TestFsShellCopy { assertEquals(0, exit); assertEquals("f1\ndf1\ndf2\ndf3\nf2\n", readFile("out")); } + + + @Test + public void testMoveFileFromLocal() throws Exception { + Path testRoot = new Path(testRootDir, "testPutFile"); + lfs.delete(testRoot, true); + lfs.mkdirs(testRoot); + + Path target = new Path(testRoot, "target"); + Path srcFile = new Path(testRoot, new Path("srcFile")); + lfs.createNewFile(srcFile); + + int exit = shell.run(new String[]{ + "-moveFromLocal", srcFile.toString(), target.toString() }); + assertEquals(0, exit); + assertFalse(lfs.exists(srcFile)); + assertTrue(lfs.exists(target)); + assertTrue(lfs.isFile(target)); + } + + @Test + public void testMoveDirFromLocal() throws Exception { + Path testRoot = new Path(testRootDir, "testPutDir"); + lfs.delete(testRoot, true); + lfs.mkdirs(testRoot); + + Path srcDir = new Path(testRoot, "srcDir"); + lfs.mkdirs(srcDir); + Path targetDir = new Path(testRoot, "target"); + + int exit = shell.run(new String[]{ + "-moveFromLocal", srcDir.toString(), targetDir.toString() }); + assertEquals(0, exit); + assertFalse(lfs.exists(srcDir)); + assertTrue(lfs.exists(targetDir)); + } + + @Test + public void testMoveDirFromLocalDestExists() throws Exception { + Path testRoot = new Path(testRootDir, "testPutDir"); + lfs.delete(testRoot, true); + lfs.mkdirs(testRoot); + + Path srcDir = new Path(testRoot, "srcDir"); + lfs.mkdirs(srcDir); + Path targetDir = new Path(testRoot, "target"); + lfs.mkdirs(targetDir); + + int exit = shell.run(new String[]{ + "-moveFromLocal", srcDir.toString(), targetDir.toString() }); + assertEquals(0, exit); + assertFalse(lfs.exists(srcDir)); + assertTrue(lfs.exists(new Path(targetDir, srcDir.getName()))); + + lfs.mkdirs(srcDir); + exit = shell.run(new String[]{ + "-moveFromLocal", srcDir.toString(), targetDir.toString() }); + assertEquals(1, exit); + assertTrue(lfs.exists(srcDir)); + } private void createFile(Path ... paths) throws IOException { for (Path path : paths) { Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java?rev=1428601&r1=1428600&r2=1428601&view=diff ============================================================================== --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java Thu Jan 3 21:23:58 2013 @@ -662,4 +662,15 @@ public class ViewFileSystemBaseTest { public void testInternalSetOwner() throws IOException { fsView.setOwner(new Path("/internalDir"), "foo", "bar"); } + + @Test + public void testCreateNonRecursive() throws IOException { + Path path = FileSystemTestHelper.getTestRootPath(fsView, "/user/foo"); + fsView.createNonRecursive(path, false, 1024, (short)1, 1024L, null); + FileStatus status = fsView.getFileStatus(new Path("/user/foo")); + Assert.assertTrue("Created file should be type file", + fsView.isFile(new Path("/user/foo"))); + Assert.assertTrue("Target of created file should be type file", + fsTarget.isFile(new Path(targetTestRoot,"user/foo"))); + } }