hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HowToConfigure" by ArpitAgarwal
Date Wed, 20 Jan 2016 05:36:15 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HowToConfigure" page has been changed by ArpitAgarwal:
https://wiki.apache.org/hadoop/HowToConfigure?action=diff&rev1=17&rev2=18

Comment:
Flag page as outdated.

+ = Deprecated =
+ '''Most of the information in this wiki page is outdated or inapplicable to Hadoop 2.x and
will be deleted soon. Refer to the Configuration section of the [[https://hadoop.apache.org/docs/stable/|latest
2.x stable release]] docs instead.'''
+ 
+ ----
+  
- = How To Configure Hadoop - ''hadoop-0.15.0 and later'' =
+ == How To Configure Hadoop - ''hadoop-0.15.0 and later'' ==
  
  Hadoop's configuration has been significantly changed in the hadoop-0.15.0 release (http://issues.apache.org/jira/browse/HADOOP-785).
  
@@ -13, +18 @@

  See QuickStart and 
  [[http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configurationml | Hadoop
Cluster Setup/Configuration]] for a description of Hadoop configuration for 0.21.0.
  
- = How To Configure Hadoop - ''pre hadoop-0.15.0'' =
+ == How To Configure Hadoop - ''pre hadoop-0.15.0'' ==
  
- == Primary XML Files ==
+ === Primary XML Files ===
  
  Hadoop is configured with a set of files. The files are loaded in the order listed in the
table below, with the lower files in the table overriding the higher ones:
  
@@ -25, +30 @@

  || job.xml || Configuration for a specific map/reduce job ||
  || hadoop-site.xml || Site specific value that can not be modified by the job ||
  
- === Look up path ===
+ ==== Look up path ====
  
  Configuration files are found via Java's Classpath. Only the first instance of each file
is used. The $HADOOP_CONF_DIR is added by the bin/hadoop script to the front of the path.
When installing Hadoop on a cluster, it is best to use a conf directory outside of the distribution.
That allows you to easily update the release on the cluster without changing your configuration
by mistake.
  
- === hadoop-default.xml ===
+ ==== hadoop-default.xml ====
  
  This file has the default values for many of the configuration variables that are used by
Hadoop. This file should never be in $HADOOP_CONF_DIR so that the version in the hadoop-*-core.jar
is used. (Otherwise, if a variable is added to this file in a new release, you won't have
it defined.)
  
- === mapred-default.xml ===
+ ==== mapred-default.xml ====
  
  This file should contain the majority of your site's customization of Hadoop. Although this
file name is prefixed with mapred, the default settings for the user maps and reduces are
controlled by it.
  
@@ -53, +58 @@

  
  [[http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuration+Files|The
rest of the valid property names and their default values]] can be found in the current docs.
  
- === job.xml ===
+ ==== job.xml ====
  
  This file is never created explicitly by the user. The map/reduce application creates a

  [[http://wiki.apache.org/hadoop/JobConfFile|JobConf]], which is serialized when the job
is submitted.
  
- === hadoop-site.xml ===
+ ==== hadoop-site.xml ====
  
  This file overrides any settings in the job.xml and therefore should be very minimal. Usually
it just contains the address of the NameNode, the address of the JobTracker, and the port
and working directories for the various servers.
  
- == Environment Variables ==
+ === Environment Variables ===
  
  For the most part, you should only need to define $HADOOP_CONF_DIR. Other environment variables
are defined in $HADOOP_CONF_DIR/hadoop-env.sh. 
  
@@ -80, +85 @@

  || HADOOP_SECURITY_LOGGER || Logging configuration for hadoop.security.logger. default:
"INFO,NullAppender" ||
  || HDFS_AUDIT_LOGGER || Logging configuration for hdfs.audit.logger. default: "INFO,NullAppender"
||
  
- == Log4j Configuration ==
+ === Log4j Configuration ===
  
  Hadoop logs messages to Log4j by default. Log4j is configured via log4j.properties on the
classpath. This file defines both what is logged and where. For applications, the default
root logger is "INFO,console", which logs all message at level INFO and above to the console's
stderr. Servers log to the "INFO,DRFA", which logs to a file that is rolled daily. Log files
are named $HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-<server>.log. 
  
@@ -88, +93 @@

   log4j.logger.org.apache.hadoop.mapred.!TaskTracker=DEBUG
  in your log4j.properties.
  
- === Audit Logging ===
+ ==== Audit Logging ====
  
  In 0.18 and later, one can enable audit logging from the Namenode. By default, events logged
to this appender are forwarded to the Namenode log, which will radically increase the number
of events emitted from that interface (see example). Audit events are emitted as a set of
key=value pairs for the following keys:
  

Mime
View raw message