Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Date: Mon, 8 Feb 2016 11:23:11 +0100
Message-ID: 
 <CAEgyCiZiU5CpvHSeasbCEONbu_p_2sO4FOQmT4w-=h-tvhFE1g@mail.gmail.com>
Subject: How can I see the LOG.debug output from CopyMapper of DistCp?
From: Emre Sevinc <emre.sevinc@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b3a8054846540052b3f96b0

--047d7b3a8054846540052b3f96b0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hello,

I'm using a recent version of Hadoop with YARN, and after running a
`distcp` job successfully, I'm trying to see the output of LOG.debug lines
from CopyMapper.java, but even though I've enabled DEBUG logging in
log4j.properties (and of course copied this file to all the nodes in my
cluster), I cannot see the output of these lines.

The LOG.debug statements I'm interested are:

  LOG.debug("DistCpMapper::map(): Received " + sourcePath + ", " + relPath)=
;

  (from:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp=
/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java?revision=3D16=
19197&view=3Dmarkup#l196
)

  LOG.debug("Copying " + sourceFileStatus.getPath() + " to " + target);
  LOG.debug("Target file path: " + targetPath);

  (from
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp=
/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java=
?revision=3D1596931&view=3Dmarkup#l113
)

The `distcp` job copies about 20 files from one cluster to another and
reports success. Then I check the YARN WEB UI and see that job is listed
under FINISHED jobs. When I click on that, application_1454924704123_0001
in my case, I see only 1 entry in the list, such as

        appattempt_1454924704123_0001_000001     Mon Feb 8 10:51:27 +0100
2016    http://hadoop10:8042       Logs

And when I click on the "Logs" I see that there's a "syslog : Total file
length is 165516 bytes." And when I examine its contents I *don't* see any
DEBUG lines, I also don't see any strings such as "DistCpMapper" or "Target
file path" that should have been produced by CopyMapper.java and
RetriableFileCopyCommand.java.

I also SSHed into `hadoop10` node, and did a `grep` but still couldn't find
such DEBUG output, e.g.:

  grep -r "Target file" /var/log/hadoop/

return no result.

In my log4j.propertie, I have lines such as:

  hadoop.root.logger=3DDEBUG,console,RFA
  log4j.logger.org.apache.hadoop.tools.mapred=3DDEBUG

And in my hadoop-env.sh I have the following line:

  export HADOOP_DAEMON_ROOT_LOGGER=3DDEBUG,RFA

Is this not enough to see the output of all LOG.debug statements from all
of the classes in `org.apache.hadoop.tools.mapred` package such as
`CopyMapper` and `RetriableFileCopyCommand`? Or am I looking at the wrong
directory?

You can see the contents of my log4j.properties and hadoop-env.sh files at
the end of this message, I made sure that they are the same on all of the
nodes in the cluster.


log4j.properties
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Define some default values that can be overridden by system properties
#hadoop.root.logger=3DINFO,console
#hadoop.root.logger=3DINFO,console,RFA
hadoop.root.logger=3DDEBUG,console,RFA
hadoop.log.dir=3D.
hadoop.log.file=3Dhadoop.log

# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=3D${hadoop.root.logger}, EventCounter

# Logging Threshold
log4j.threshold=3DALL

# Null Appender
log4j.appender.NullAppender=3Dorg.apache.log4j.varia.NullAppender

#
# Rolling File Appender - cap space usage at 5gb.
#
hadoop.log.maxfilesize=3D256MB
hadoop.log.maxbackupindex=3D20
log4j.appender.RFA=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.RFA.File=3D${hadoop.log.dir}/${hadoop.log.file}

log4j.appender.RFA.MaxFileSize=3D${hadoop.log.maxfilesize}
log4j.appender.RFA.MaxBackupIndex=3D${hadoop.log.maxbackupindex}

log4j.appender.RFA.layout=3Dorg.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage
log4j.appender.RFA.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n
# Debugging Pattern format
#log4j.appender.RFA.layout.ConversionPattern=3D%d{ISO8601} %-5p %c{2}
(%F:%M(%L)) - %m%n


#
# Daily Rolling File Appender
#

log4j.appender.DRFA=3Dorg.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA.File=3D${hadoop.log.dir}/${hadoop.log.file}

# Rollover at midnight
log4j.appender.DRFA.DatePattern=3D.yyyy-MM-dd

log4j.appender.DRFA.layout=3Dorg.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage
log4j.appender.DRFA.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n
# Debugging Pattern format
#log4j.appender.DRFA.layout.ConversionPattern=3D%d{ISO8601} %-5p %c{2}
(%F:%M(%L)) - %m%n


#
# console
# Add "console" to rootlogger above if you want to use this
#

log4j.appender.console=3Dorg.apache.log4j.ConsoleAppender
log4j.appender.console.target=3DSystem.err
log4j.appender.console.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=3D%d{yy/MM/dd HH:mm:ss} %p
%c{2}: %m%n

#
# TaskLog Appender
#

#Default values
hadoop.tasklog.taskid=3Dnull
hadoop.tasklog.iscleanup=3Dfalse
hadoop.tasklog.noKeepSplits=3D4
hadoop.tasklog.totalLogFileSize=3D100
hadoop.tasklog.purgeLogSplits=3Dtrue
hadoop.tasklog.logsRetainHours=3D12

log4j.appender.TLA=3Dorg.apache.hadoop.mapred.TaskLogAppender
log4j.appender.TLA.taskId=3D${hadoop.tasklog.taskid}
log4j.appender.TLA.isCleanup=3D${hadoop.tasklog.iscleanup}
log4j.appender.TLA.totalLogFileSize=3D${hadoop.tasklog.totalLogFileSize}

log4j.appender.TLA.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n

#
# HDFS block state change log from block manager
#
# Uncomment the following to suppress normal block state change
# messages from BlockManager in NameNode.
#log4j.logger.BlockStateChange=3DWARN

#
#Security appender
#
hadoop.security.logger=3DINFO,NullAppender
hadoop.security.log.maxfilesize=3D256MB
hadoop.security.log.maxbackupindex=3D20
log4j.category.SecurityLogger=3D${hadoop.security.logger}
hadoop.security.log.file=3DSecurityAuth-${user.name}.audit
log4j.appender.RFAS=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.RFAS.File=3D${hadoop.log.dir}/${hadoop.security.log.file}
log4j.appender.RFAS.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.RFAS.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n
log4j.appender.RFAS.MaxFileSize=3D${hadoop.security.log.maxfilesize}
log4j.appender.RFAS.MaxBackupIndex=3D${hadoop.security.log.maxbackupindex}

#
# Daily Rolling Security appender
#
log4j.appender.DRFAS=3Dorg.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAS.File=3D${hadoop.log.dir}/${hadoop.security.log.file}
log4j.appender.DRFAS.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.DRFAS.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n
log4j.appender.DRFAS.DatePattern=3D.yyyy-MM-dd

#
# hadoop configuration logging
#

# Uncomment the following line to turn off configuration deprecation
warnings.
# log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=3DWARN

#
# hdfs audit logging
#
hdfs.audit.logger=3DINFO,NullAppender
hdfs.audit.log.maxfilesize=3D256MB
hdfs.audit.log.maxbackupindex=3D20
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=3D${=
hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=
=3Dfalse
log4j.appender.RFAAUDIT=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.RFAAUDIT.File=3D${hadoop.log.dir}/hdfs-audit.log
log4j.appender.RFAAUDIT.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.RFAAUDIT.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m=
%n
log4j.appender.RFAAUDIT.MaxFileSize=3D${hdfs.audit.log.maxfilesize}
log4j.appender.RFAAUDIT.MaxBackupIndex=3D${hdfs.audit.log.maxbackupindex}

#
# NameNode metrics logging.
# The default is to retain two namenode-metrics.log files up to 64MB each.
#
namenode.metrics.logger=3DINFO,NullAppender
log4j.logger.NameNodeMetricsLog=3D${namenode.metrics.logger}
log4j.additivity.NameNodeMetricsLog=3Dfalse
log4j.appender.NNMETRICSRFA=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.NNMETRICSRFA.File=3D${hadoop.log.dir}/namenode-metrics.log
log4j.appender.NNMETRICSRFA.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.NNMETRICSRFA.layout.ConversionPattern=3D%d{ISO8601} %m%n
log4j.appender.NNMETRICSRFA.MaxBackupIndex=3D1
log4j.appender.NNMETRICSRFA.MaxFileSize=3D64MB

#
# DataNode metrics logging.
# The default is to retain two datanode-metrics.log files up to 64MB each.
#
datanode.metrics.logger=3DINFO,NullAppender
log4j.logger.DataNodeMetricsLog=3D${datanode.metrics.logger}
log4j.additivity.DataNodeMetricsLog=3Dfalse
log4j.appender.DNMETRICSRFA=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.DNMETRICSRFA.File=3D${hadoop.log.dir}/datanode-metrics.log
log4j.appender.DNMETRICSRFA.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.DNMETRICSRFA.layout.ConversionPattern=3D%d{ISO8601} %m%n
log4j.appender.DNMETRICSRFA.MaxBackupIndex=3D1
log4j.appender.DNMETRICSRFA.MaxFileSize=3D64MB

#
# mapred audit logging
#
mapred.audit.logger=3DINFO,NullAppender
mapred.audit.log.maxfilesize=3D256MB
mapred.audit.log.maxbackupindex=3D20
log4j.logger.org.apache.hadoop.mapred.AuditLogger=3D${mapred.audit.logger}
log4j.additivity.org.apache.hadoop.mapred.AuditLogger=3Dfalse
log4j.appender.MRAUDIT=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.MRAUDIT.File=3D${hadoop.log.dir}/mapred-audit.log
log4j.appender.MRAUDIT.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.MRAUDIT.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m%=
n
log4j.appender.MRAUDIT.MaxFileSize=3D${mapred.audit.log.maxfilesize}
log4j.appender.MRAUDIT.MaxBackupIndex=3D${mapred.audit.log.maxbackupindex}

# Custom Logging levels

#log4j.logger.org.apache.hadoop.mapred.JobTracker=3DDEBUG
#log4j.logger.org.apache.hadoop.mapred.TaskTracker=3DDEBUG
#log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=3DD=
EBUG

# Jets3t library
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=3DERROR

# AWS SDK & S3A FileSystem
log4j.logger.com.amazonaws=3DERROR
log4j.logger.com.amazonaws.http.AmazonHttpClient=3DERROR
#log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=3DWARN

log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=3DDEBUG
log4j.logger.org.apache.hadoop.tools.mapred=3DDEBUG
#log4j.logger.org.apache.hadoop=3DDEBUG

#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop
Metrics.
#
log4j.appender.EventCounter=3Dorg.apache.hadoop.log.metrics.EventCounter

#
# Job Summary Appender
#
# Use following logger to send summary to separate file defined by
# hadoop.mapreduce.jobsummary.log.file :
# hadoop.mapreduce.jobsummary.logger=3DINFO,JSA
#
hadoop.mapreduce.jobsummary.logger=3D${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=3Dhadoop-mapreduce.jobsummary.log
hadoop.mapreduce.jobsummary.log.maxfilesize=3D256MB
hadoop.mapreduce.jobsummary.log.maxbackupindex=3D20
log4j.appender.JSA=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.JSA.File=3D${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.l=
og.file}
log4j.appender.JSA.MaxFileSize=3D${hadoop.mapreduce.jobsummary.log.maxfiles=
ize}
log4j.appender.JSA.MaxBackupIndex=3D${hadoop.mapreduce.jobsummary.log.maxba=
ckupindex}
log4j.appender.JSA.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=3D%d{yy/MM/dd HH:mm:ss} %p %c{2=
}:
%m%n
log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=3D${hadoop.m=
apreduce.jobsummary.logger}
log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=3Dfalse

#
# shuffle connection log from shuffleHandler
# Uncomment the following line to enable logging of shuffle connections
# log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=3DDEBUG

#
# Yarn ResourceManager Application Summary Log
#
# Set the ResourceManager summary log filename
yarn.server.resourcemanager.appsummary.log.file=3Drm-appsummary.log
# Set the ResourceManager summary log level and appender
yarn.server.resourcemanager.appsummary.logger=3D${hadoop.root.logger}
#yarn.server.resourcemanager.appsummary.logger=3DINFO,RMSUMMARY

# To enable AppSummaryLogging for the RM,
# set yarn.server.resourcemanager.appsummary.logger to
# <LEVEL>,RMSUMMARY in hadoop-env.sh

# Appender for ResourceManager Application Summary Log
# Requires the following properties to be set
#    - hadoop.log.dir (Hadoop Log directory)
#    - yarn.server.resourcemanager.appsummary.log.file (resource manager
app summary log filename)
#    - yarn.server.resourcemanager.appsummary.logger (resource manager app
summary log level and appender)

log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$App=
licationSummary=3D${yarn.server.resourcemanager.appsummary.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager=
$ApplicationSummary=3Dfalse
log4j.appender.RMSUMMARY=3Dorg.apache.log4j.RollingFileAppender
log4j.appender.RMSUMMARY.File=3D${hadoop.log.dir}/${yarn.server.resourceman=
ager.appsummary.log.file}
log4j.appender.RMSUMMARY.MaxFileSize=3D256MB
log4j.appender.RMSUMMARY.MaxBackupIndex=3D20
log4j.appender.RMSUMMARY.layout=3Dorg.apache.log4j.PatternLayout
log4j.appender.RMSUMMARY.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %=
m%n

# HS audit log configs
#mapreduce.hs.audit.logger=3DINFO,HSAUDIT
#log4j.logger.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=3D${mapreduce=
.hs.audit.logger}
#log4j.additivity.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=3Dfalse
#log4j.appender.HSAUDIT=3Dorg.apache.log4j.DailyRollingFileAppender
#log4j.appender.HSAUDIT.File=3D${hadoop.log.dir}/hs-audit.log
#log4j.appender.HSAUDIT.layout=3Dorg.apache.log4j.PatternLayout
#log4j.appender.HSAUDIT.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m=
%n
#log4j.appender.HSAUDIT.DatePattern=3D.yyyy-MM-dd

# Http Server Request Logs
#log4j.logger.http.requests.namenode=3DINFO,namenoderequestlog
#log4j.appender.namenoderequestlog=3Dorg.apache.hadoop.http.HttpRequestLogA=
ppender
#log4j.appender.namenoderequestlog.Filename=3D${hadoop.log.dir}/jetty-namen=
ode-yyyy_mm_dd.log
#log4j.appender.namenoderequestlog.RetainDays=3D3

#log4j.logger.http.requests.datanode=3DINFO,datanoderequestlog
#log4j.appender.datanoderequestlog=3Dorg.apache.hadoop.http.HttpRequestLogA=
ppender
#log4j.appender.datanoderequestlog.Filename=3D${hadoop.log.dir}/jetty-datan=
ode-yyyy_mm_dd.log
#log4j.appender.datanoderequestlog.RetainDays=3D3

#log4j.logger.http.requests.resourcemanager=3DINFO,resourcemanagerrequestlo=
g
#log4j.appender.resourcemanagerrequestlog=3Dorg.apache.hadoop.http.HttpRequ=
estLogAppender
#log4j.appender.resourcemanagerrequestlog.Filename=3D${hadoop.log.dir}/jett=
y-resourcemanager-yyyy_mm_dd.log
#log4j.appender.resourcemanagerrequestlog.RetainDays=3D3

#log4j.logger.http.requests.jobhistory=3DINFO,jobhistoryrequestlog
#log4j.appender.jobhistoryrequestlog=3Dorg.apache.hadoop.http.HttpRequestLo=
gAppender
#log4j.appender.jobhistoryrequestlog.Filename=3D${hadoop.log.dir}/jetty-job=
history-yyyy_mm_dd.log
#log4j.appender.jobhistoryrequestlog.RetainDays=3D3

#log4j.logger.http.requests.nodemanager=3DINFO,nodemanagerrequestlog
#log4j.appender.nodemanagerrequestlog=3Dorg.apache.hadoop.http.HttpRequestL=
ogAppender
#log4j.appender.nodemanagerrequestlog.Filename=3D${hadoop.log.dir}/jetty-no=
demanager-yyyy_mm_dd.log
#log4j.appender.nodemanagerrequestlog.RetainDays=3D3

# Appender for viewing information for errors and warnings
yarn.ewma.cleanupInterval=3D300
yarn.ewma.messageAgeLimitSeconds=3D86400
yarn.ewma.maxUniqueMessages=3D250
log4j.appender.EWMA=3Dorg.apache.hadoop.yarn.util.Log4jWarningErrorMetricsA=
ppender
log4j.appender.EWMA.cleanupInterval=3D${yarn.ewma.cleanupInterval}
log4j.appender.EWMA.messageAgeLimitSeconds=3D${yarn.ewma.messageAgeLimitSec=
onds}
log4j.appender.EWMA.maxUniqueMessages=3D${yarn.ewma.maxUniqueMessages}
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


hadoop-env.sh
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

##
## THIS FILE ACTS AS THE MASTER FILE FOR ALL HADOOP PROJECTS.
## SETTINGS HERE WILL BE READ BY ALL HADOOP COMMANDS.  THEREFORE,
## ONE CAN USE THIS FILE TO SET YARN, HDFS, AND MAPREDUCE
## CONFIGURATION OPTIONS INSTEAD OF xxx-env.sh.
##
## Precedence rules:
##
## {yarn-env.sh|hdfs-env.sh} > hadoop-env.sh > hard-coded defaults
##
## {YARN_xyz|HDFS_xyz} > HADOOP_xyz > hard-coded defaults
##

# Many of the options here are built from the perspective that users
# may want to provide OVERWRITING values on the command line.
# For example:
#
#  JAVA_HOME=3D/usr/java/testing hdfs dfs -ls
#
# Therefore, the vast majority (BUT NOT ALL!) of these defaults
# are configured for substitution and not append.  If append
# is preferable, modify this file accordingly.

###
# Generic settings for HADOOP
###

# Technically, the only required environment variable is JAVA_HOME.
# All others are optional.  However, the defaults are probably not
# preferred.  Many sites configure these options outside of Hadoop,
# such as in /etc/profile.d

# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=3D/usr/lib/jvm/java-8-oracle

# Location of Hadoop.  By default, Hadoop will attempt to determine
# this location based upon its execution path.
export HADOOP_PREFIX=3D/opt/hadoop-3.0.0-SNAPSHOT

# Location of Hadoop's configuration information.  i.e., where this
# file is probably living. Many sites will also set this in the
# same location where JAVA_HOME is defined.  If this is not defined
# Hadoop will attempt to locate it based upon its execution
# path.
export HADOOP_CONF_DIR=3D$HADOOP_PREFIX/etc/hadoop

# The maximum amount of heap to use (Java -Xmx).  If no unit
# is provided, it will be converted to MB.  Daemons will
# prefer any Xmx setting in their respective _OPT variable.
# There is no default; the JVM will autoscale based upon machine
# memory size.
# export HADOOP_HEAPSIZE_MAX=3D

# The minimum amount of heap to use (Java -Xms).  If no unit
# is provided, it will be converted to MB.  Daemons will
# prefer any Xms setting in their respective _OPT variable.
# There is no default; the JVM will autoscale based upon machine
# memory size.
# export HADOOP_HEAPSIZE_MIN=3D

# Extra Java runtime options for all Hadoop commands. We don't support
# IPv6 yet/still, so by default the preference is set to IPv4.
# export HADOOP_OPTS=3D"-Djava.net.preferIPv4Stack=3Dtrue"

# Some parts of the shell code may do special things dependent upon
# the operating system.  We have to set this here. See the next
# section as to why....
export HADOOP_OS_TYPE=3D${HADOOP_OS_TYPE:-$(uname -s)}


# Under certain conditions, Java on OS X will throw SCDynamicStore errors
# in the system logs.
# See HADOOP-8719 for more information.  If one needs Kerberos
# support on OS X, one will want to change/remove this extra bit.
case ${HADOOP_OS_TYPE} in
  Darwin*)
    export HADOOP_OPTS=3D"${HADOOP_OPTS} -Djava.security.krb5.realm=3D "
    export HADOOP_OPTS=3D"${HADOOP_OPTS} -Djava.security.krb5.kdc=3D "
    export HADOOP_OPTS=3D"${HADOOP_OPTS} -Djava.security.krb5.conf=3D "
  ;;
esac

# Extra Java runtime options for some Hadoop commands
# and clients (i.e., hdfs dfs -blah).  These get appended to HADOOP_OPTS
for
# such commands.  In most cases, # this should be left empty and
# let users supply it on the command line.
# export HADOOP_CLIENT_OPTS=3D""

#
# A note about classpaths.
#
# The classpath is configured such that entries are stripped prior
# to handing to Java based either upon duplication or non-existence.
# Wildcards and/or directories are *NOT* expanded as the
# de-duplication is fairly simple.  So if two directories are in
# the classpath that both contain awesome-methods-1.0.jar,
# awesome-methods-1.0.jar will still be seen by java.  But if
# the classpath specifically has awesome-methods-1.0.jar from the
# same directory listed twice, the last one will be removed.
#

# An additional, custom CLASSPATH.  This is really meant for
# end users, but as an administrator, one might want to push
# something extra in here too, such as the jar to the topology
# method.  Just be sure to append to the existing HADOOP_USER_CLASSPATH
# so end users have a way to add stuff.
# export HADOOP_USER_CLASSPATH=3D"/some/cool/path/on/your/machine"

# Should HADOOP_USER_CLASSPATH be first in the official CLASSPATH?
# export HADOOP_USER_CLASSPATH_FIRST=3D"yes"

# If HADOOP_USE_CLIENT_CLASSLOADER is set, HADOOP_CLASSPATH along with the
main
# jar are handled by a separate isolated client classloader. If it is set,
# HADOOP_USER_CLASSPATH_FIRST is ignored. Can be defined by doing
# export HADOOP_USE_CLIENT_CLASSLOADER=3Dtrue

# HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES overrides the default definition
of
# system classes for the client classloader when
HADOOP_USE_CLIENT_CLASSLOADER
# is enabled. Names ending in '.' (period) are treated as package names, an=
d
# names starting with a '-' are treated as negative matches. For example,
# export
HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES=3D"-org.apache.hadoop.UserClass,ja=
va.,javax.,org.apache.hadoop."

# You need the hadoop-aws-3.0.0-SNAPSHOT.jar (or similar) in your CLASSPATH
# otherwise you might get the following error:
# java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.s3a.S3AFileSystem not found
export
HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$HADOOP_PREFIX/share/hadoop/tools/lib/=
*

###
# Options for remote shell connectivity
###

# There are some optional components of hadoop that allow for
# command and control of remote hosts.  For example,
# start-dfs.sh will attempt to bring up all NNs, DNS, etc.

# Options to pass to SSH when one of the "log into a host and
# start/stop daemons" scripts is executed
# export HADOOP_SSH_OPTS=3D"-o BatchMode=3Dyes -o StrictHostKeyChecking=3Dn=
o -o
ConnectTimeout=3D10s"

# The built-in ssh handler will limit itself to 10 simultaneous connections=
.
# For pdsh users, this sets the fanout size ( -f )
# Change this to increase/decrease as necessary.
# export HADOOP_SSH_PARALLEL=3D10

# Filename which contains all of the hosts for any remote execution
# helper scripts # such as slaves.sh, start-dfs.sh, etc.
# export HADOOP_SLAVES=3D"${HADOOP_CONF_DIR}/slaves"

###
# Options for all daemons
###
#

#
# Many options may also be specified as Java properties.  It is
# very common, and in many cases, desirable, to hard-set these
# in daemon _OPTS variables.  Where applicable, the appropriate
# Java property is also identified.  Note that many are re-used
# or set differently in certain contexts (e.g., secure vs
# non-secure)
#

# Where (primarily) daemon log files are stored.  # $HADOOP_PREFIX/logs
# by default.
# Java property: hadoop.log.dir
export HADOOP_LOG_DIR=3D/var/log/hadoop

# A string representing this instance of hadoop. $USER by default.
# This is used in writing log and pid files, so keep that in mind!
# Java property: hadoop.id.str
# export HADOOP_IDENT_STRING=3D$USER

# How many seconds to pause after stopping a daemon
# export HADOOP_STOP_TIMEOUT=3D5

# Where pid files are stored.  /tmp by default.
# export HADOOP_PID_DIR=3D/tmp

# Default log4j setting for interactive commands
# Java property: hadoop.root.logger
# export HADOOP_ROOT_LOGGER=3DINFO,console

# Default log4j setting for daemons spawned explicitly by
# --daemon option of hadoop, hdfs, mapred and yarn command.
# Java property: hadoop.root.logger
#export HADOOP_DAEMON_ROOT_LOGGER=3DINFO,RFA
export HADOOP_DAEMON_ROOT_LOGGER=3DDEBUG,RFA

# Default log level and output location for security-related messages.
# You will almost certainly want to change this on a per-daemon basis via
# the Java property (i.e., -Dhadoop.security.logger=3Dfoo). (Note that the
# defaults for the NN and 2NN override this by default.)
# Java property: hadoop.security.logger
# export HADOOP_SECURITY_LOGGER=3DINFO,NullAppender

# Default log level for file system audit messages.
# Generally, this is specifically set in the namenode-specific
# options line.
# Java property: hdfs.audit.logger
# export HADOOP_AUDIT_LOGGER=3DINFO,NullAppender

# Default process priority level
# Note that sub-processes will also run at this level!
# export HADOOP_NICENESS=3D0

# Default name for the service level authorization file
# Java property: hadoop.policy.file
# export HADOOP_POLICYFILE=3D"hadoop-policy.xml"

#
# NOTE: this is not used by default!  <-----
# You can define variables right here and then re-use them later on.
# For example, it is common to use the same garbage collection settings
# for all the daemons.  So one could define:
#
# export HADOOP_GC_SETTINGS=3D"-verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
#
# .. and then use it as per the b option under the namenode.

###
# Secure/privileged execution
###

#
# Out of the box, Hadoop uses jsvc from Apache Commons to launch daemons
# on privileged ports.  This functionality can be replaced by providing
# custom functions.  See hadoop-functions.sh for more information.
#

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol.  Jsvc is not required if SASL is configured for authentication
of
# data transfer protocol using non-privileged ports.
# export JSVC_HOME=3D/usr/bin

#
# This directory contains pids for secure and privileged processes.
#export HADOOP_SECURE_PID_DIR=3D${HADOOP_PID_DIR}

#
# This directory contains the logs for secure and privileged processes.
# Java property: hadoop.log.dir
# export HADOOP_SECURE_LOG=3D${HADOOP_LOG_DIR}

#
# When running a secure daemon, the default value of HADOOP_IDENT_STRING
# ends up being a bit bogus.  Therefore, by default, the code will
# replace HADOOP_IDENT_STRING with HADOOP_SECURE_xx_USER.  If one wants
# to keep HADOOP_IDENT_STRING untouched, then uncomment this line.
# export HADOOP_SECURE_IDENT_PRESERVE=3D"true"

###
# NameNode specific parameters
###

# Default log level and output location for file system related change
# messages. For non-namenode daemons, the Java property must be set in
# the appropriate _OPTS if one wants something other than INFO,NullAppender
# Java property: hdfs.audit.logger
# export HDFS_AUDIT_LOGGER=3DINFO,NullAppender

# Specify the JVM options to be used when starting the NameNode.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# a) Set JMX options
# export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote=3Dtrue
-Dcom.sun.management.jmxremote.authenticate=3Dfalse
-Dcom.sun.management.jmxremote.ssl=3Dfalse
-Dcom.sun.management.jmxremote.port=3D1026"
#
# b) Set garbage collection logs
# export HADOOP_NAMENODE_OPTS=3D"${HADOOP_GC_SETTINGS}
-Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"
#
# c) ... or set them directly
# export HADOOP_NAMENODE_OPTS=3D"-verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"

# this is the default:
# export HADOOP_NAMENODE_OPTS=3D"-Dhadoop.security.logger=3DINFO,RFAS"

###
# SecondaryNameNode specific parameters
###
# Specify the JVM options to be used when starting the SecondaryNameNode.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# This is the default:
# export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dhadoop.security.logger=3DINFO,R=
FAS"

###
# DataNode specific parameters
###
# Specify the JVM options to be used when starting the DataNode.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# This is the default:
# export HADOOP_DATANODE_OPTS=3D"-Dhadoop.security.logger=3DERROR,RFAS"

# On secure datanodes, user to run the datanode as after dropping
privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged
ports
# to provide authentication of data transfer protocol.  This **MUST NOT** b=
e
# defined if SASL is configured for authentication of data transfer protoco=
l
# using non-privileged ports.
# This will replace the hadoop.id.str Java property in secure mode.
# export HADOOP_SECURE_DN_USER=3Dhdfs

# Supplemental options for secure datanodes
# By default, Hadoop uses jsvc which needs to know to launch a
# server jvm.
# export HADOOP_DN_SECURE_EXTRA_OPTS=3D"-jvm server"

# Where datanode log files are stored in the secure data environment.
# This will replace the hadoop.log.dir Java property in secure mode.
# export HADOOP_SECURE_DN_LOG_DIR=3D${HADOOP_SECURE_LOG_DIR}

# Where datanode pid files are stored in the secure data environment.
# export HADOOP_SECURE_DN_PID_DIR=3D${HADOOP_SECURE_PID_DIR}

###
# NFS3 Gateway specific parameters
###
# Specify the JVM options to be used when starting the NFS3 Gateway.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_NFS3_OPTS=3D""

# Specify the JVM options to be used when starting the Hadoop portmapper.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_PORTMAP_OPTS=3D"-Xmx512m"

# Supplemental options for priviliged gateways
# By default, Hadoop uses jsvc which needs to know to launch a
# server jvm.
# export HADOOP_NFS3_SECURE_EXTRA_OPTS=3D"-jvm server"

# On privileged gateways, user to run the gateway as after dropping
privileges
# This will replace the hadoop.id.str Java property in secure mode.
# export HADOOP_PRIVILEGED_NFS_USER=3Dnfsserver

###
# ZKFailoverController specific parameters
###
# Specify the JVM options to be used when starting the ZKFailoverController=
.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_ZKFC_OPTS=3D""

###
# QuorumJournalNode specific parameters
###
# Specify the JVM options to be used when starting the QuorumJournalNode.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_JOURNALNODE_OPTS=3D""

###
# HDFS Balancer specific parameters
###
# Specify the JVM options to be used when starting the HDFS Balancer.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_BALANCER_OPTS=3D""

###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=3D""

###
# Advanced Users Only!
###

#
# When building Hadoop, one can add the class paths to the commands
# via this special env var:
# export HADOOP_ENABLE_BUILD_PATHS=3D"true"

#
# To prevent accidents, shell commands be (superficially) locked
# to only allow certain users to execute certain subcommands.
#
# For example, to limit who can execute the namenode command,
# export HADOOP_namenode_USER=3Dhdfs
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


--
Emre Sevin=C3=A7

--047d7b3a8054846540052b3f96b0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Hello,<br><br>I&#39;m using a recent versio=
n of Hadoop with YARN, and after running a `distcp` job successfully, I&#39=
;m trying to see the output of LOG.debug lines from CopyMapper.java, but ev=
en though I&#39;ve enabled DEBUG logging in log4j.properties (and of course=
 copied this file to all the nodes in my cluster), I cannot see the output =
of these lines.<br><br>The LOG.debug statements I&#39;m interested are:<br>=
<br>=C2=A0 LOG.debug(&quot;DistCpMapper::map(): Received &quot; + sourcePat=
h + &quot;, &quot; + relPath);<br><br>=C2=A0 (from: <a href=3D"http://svn.a=
pache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/ja=
va/org/apache/hadoop/tools/mapred/CopyMapper.java?revision=3D1619197&amp;vi=
ew=3Dmarkup#l196">http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-t=
ools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.=
java?revision=3D1619197&amp;view=3Dmarkup#l196</a> )<br><br>=C2=A0 LOG.debu=
g(&quot;Copying &quot; + sourceFileStatus.getPath() + &quot; to &quot; + ta=
rget);<br>=C2=A0 LOG.debug(&quot;Target file path: &quot; + targetPath);<br=
><br>=C2=A0 (from <a href=3D"http://svn.apache.org/viewvc/hadoop/common/tru=
nk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/=
RetriableFileCopyCommand.java?revision=3D1596931&amp;view=3Dmarkup#l113">ht=
tp://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp/s=
rc/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java?r=
evision=3D1596931&amp;view=3Dmarkup#l113</a> )<br><br>The `distcp` job copi=
es about 20 files from one cluster to another and reports success. Then I c=
heck the YARN WEB UI and see that job is listed under FINISHED jobs. When I=
 click on that, application_1454924704123_0001 in my case, I see only 1 ent=
ry in the list, such as<br><br>=C2=A0 =C2=A0=C2=A0 =C2=A0=C2=A0 appattempt_=
1454924704123_0001_000001=C2=A0=C2=A0=C2=A0=C2=A0 Mon Feb 8 10:51:27 +0100 =
2016=C2=A0=C2=A0=C2=A0 <a href=3D"http://hadoop10:8042">http://hadoop10:804=
2</a>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Logs<br><br></div>And when I clic=
k on the &quot;Logs&quot; I see that there&#39;s a &quot;syslog : Total fil=
e length is 165516 bytes.&quot; And when I examine its contents I *don&#39;=
t* see any DEBUG lines, I also don&#39;t see any strings such as &quot;Dist=
CpMapper&quot; or &quot;Target file path&quot; that should have been produc=
ed by CopyMapper.java and RetriableFileCopyCommand.java.<br><br></div><div>=
I also SSHed into `hadoop10` node, and did a `grep` but still couldn&#39;t =
find such DEBUG output, e.g.:<br><br>=C2=A0 grep -r &quot;Target file&quot;=
 /var/log/hadoop/<br><br></div><div>return no result.<br></div><div><br></d=
iv>In my log4j.propertie, I have lines such as:<br><br>=C2=A0 hadoop.root.l=
ogger=3DDEBUG,console,RFA<br>=C2=A0 log4j.logger.org.apache.hadoop.tools.ma=
pred=3DDEBUG<br><br></div>And in my hadoop-env.sh I have the following line=
:<br><br>=C2=A0 export HADOOP_DAEMON_ROOT_LOGGER=3DDEBUG,RFA<br><div> <div>=
<br><div>Is this not enough to see the output of all LOG.debug statements f=
rom all of the classes in `org.apache.hadoop.tools.mapred` package such as =
`CopyMapper` and `RetriableFileCopyCommand`? Or am I looking at the wrong d=
irectory?<br></div><div><br>You can see the contents of my log4j.properties=
 and hadoop-env.sh files at the end of this message, I made sure that they =
are the same on all of the nodes in the cluster.<br><br></div><div><br></di=
v><div>log4j.properties<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D<br># Licensed to the Apache Software Foundation (ASF) un=
der one<br># or more contributor license agreements.=C2=A0 See the NOTICE f=
ile<br># distributed with this work for additional information<br># regardi=
ng copyright ownership.=C2=A0 The ASF licenses this file<br># to you under =
the Apache License, Version 2.0 (the<br># &quot;License&quot;); you may not=
 use this file except in compliance<br># with the License.=C2=A0 You may ob=
tain a copy of the License at<br>#<br>#=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"=
http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/=
LICENSE-2.0</a><br>#<br># Unless required by applicable law or agreed to in=
 writing, software<br># distributed under the License is distributed on an =
&quot;AS IS&quot; BASIS,<br># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,=
 either express or implied.<br># See the License for the specific language =
governing permissions and<br># limitations under the License.<br><br># Defi=
ne some default values that can be overridden by system properties<br>#hado=
op.root.logger=3DINFO,console<br>#hadoop.root.logger=3DINFO,console,RFA<br>=
hadoop.root.logger=3DDEBUG,console,RFA<br>hadoop.log.dir=3D.<br>hadoop.log.=
file=3Dhadoop.log<br><br># Define the root logger to the system property &q=
uot;hadoop.root.logger&quot;.<br>log4j.rootLogger=3D${hadoop.root.logger}, =
EventCounter<br><br># Logging Threshold<br>log4j.threshold=3DALL<br><br># N=
ull Appender<br>log4j.appender.NullAppender=3Dorg.apache.log4j.varia.NullAp=
pender<br><br>#<br># Rolling File Appender - cap space usage at 5gb.<br>#<b=
r>hadoop.log.maxfilesize=3D256MB<br>hadoop.log.maxbackupindex=3D20<br>log4j=
.appender.RFA=3Dorg.apache.log4j.RollingFileAppender<br>log4j.appender.RFA.=
File=3D${hadoop.log.dir}/${hadoop.log.file}<br><br>log4j.appender.RFA.MaxFi=
leSize=3D${hadoop.log.maxfilesize}<br>log4j.appender.RFA.MaxBackupIndex=3D$=
{hadoop.log.maxbackupindex}<br><br>log4j.appender.RFA.layout=3Dorg.apache.l=
og4j.PatternLayout<br><br># Pattern format: Date LogLevel LoggerName LogMes=
sage<br>log4j.appender.RFA.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m=
%n<br># Debugging Pattern format<br>#log4j.appender.RFA.layout.ConversionPa=
ttern=3D%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n<br><br><br>#<br># Daily R=
olling File Appender<br>#<br><br>log4j.appender.DRFA=3Dorg.apache.log4j.Dai=
lyRollingFileAppender<br>log4j.appender.DRFA.File=3D${hadoop.log.dir}/${had=
oop.log.file}<br><br># Rollover at midnight<br>log4j.appender.DRFA.DatePatt=
ern=3D.yyyy-MM-dd<br><br>log4j.appender.DRFA.layout=3Dorg.apache.log4j.Patt=
ernLayout<br><br># Pattern format: Date LogLevel LoggerName LogMessage<br>l=
og4j.appender.DRFA.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n<br># =
Debugging Pattern format<br>#log4j.appender.DRFA.layout.ConversionPattern=
=3D%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n<br><br><br>#<br># console<br>#=
 Add &quot;console&quot; to rootlogger above if you want to use this<br>#<b=
r><br>log4j.appender.console=3Dorg.apache.log4j.ConsoleAppender<br>log4j.ap=
pender.console.target=3DSystem.err<br>log4j.appender.console.layout=3Dorg.a=
pache.log4j.PatternLayout<br>log4j.appender.console.layout.ConversionPatter=
n=3D%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n<br><br>#<br># TaskLog Appender<br>=
#<br><br>#Default values<br>hadoop.tasklog.taskid=3Dnull<br>hadoop.tasklog.=
iscleanup=3Dfalse<br>hadoop.tasklog.noKeepSplits=3D4<br>hadoop.tasklog.tota=
lLogFileSize=3D100<br>hadoop.tasklog.purgeLogSplits=3Dtrue<br>hadoop.tasklo=
g.logsRetainHours=3D12<br><br>log4j.appender.TLA=3Dorg.apache.hadoop.mapred=
.TaskLogAppender<br>log4j.appender.TLA.taskId=3D${hadoop.tasklog.taskid}<br=
>log4j.appender.TLA.isCleanup=3D${hadoop.tasklog.iscleanup}<br>log4j.append=
er.TLA.totalLogFileSize=3D${hadoop.tasklog.totalLogFileSize}<br><br>log4j.a=
ppender.TLA.layout=3Dorg.apache.log4j.PatternLayout<br>log4j.appender.TLA.l=
ayout.ConversionPattern=3D%d{ISO8601} %p %c: %m%n<br><br>#<br># HDFS block =
state change log from block manager<br>#<br># Uncomment the following to su=
ppress normal block state change<br># messages from BlockManager in NameNod=
e.<br>#log4j.logger.BlockStateChange=3DWARN<br><br>#<br>#Security appender<=
br>#<br>hadoop.security.logger=3DINFO,NullAppender<br>hadoop.security.log.m=
axfilesize=3D256MB<br>hadoop.security.log.maxbackupindex=3D20<br>log4j.cate=
gory.SecurityLogger=3D${hadoop.security.logger}<br>hadoop.security.log.file=
=3DSecurityAuth-${<a href=3D"http://user.name">user.name</a>}.audit<br>log4=
j.appender.RFAS=3Dorg.apache.log4j.RollingFileAppender<br>log4j.appender.RF=
AS.File=3D${hadoop.log.dir}/${hadoop.security.log.file}<br>log4j.appender.R=
FAS.layout=3Dorg.apache.log4j.PatternLayout<br>log4j.appender.RFAS.layout.C=
onversionPattern=3D%d{ISO8601} %p %c: %m%n<br>log4j.appender.RFAS.MaxFileSi=
ze=3D${hadoop.security.log.maxfilesize}<br>log4j.appender.RFAS.MaxBackupInd=
ex=3D${hadoop.security.log.maxbackupindex}<br><br>#<br># Daily Rolling Secu=
rity appender<br>#<br>log4j.appender.DRFAS=3Dorg.apache.log4j.DailyRollingF=
ileAppender<br>log4j.appender.DRFAS.File=3D${hadoop.log.dir}/${hadoop.secur=
ity.log.file}<br>log4j.appender.DRFAS.layout=3Dorg.apache.log4j.PatternLayo=
ut<br>log4j.appender.DRFAS.layout.ConversionPattern=3D%d{ISO8601} %p %c: %m=
%n<br>log4j.appender.DRFAS.DatePattern=3D.yyyy-MM-dd<br><br>#<br># hadoop c=
onfiguration logging<br>#<br><br># Uncomment the following line to turn off=
 configuration deprecation warnings.<br># log4j.logger.org.apache.hadoop.co=
nf.Configuration.deprecation=3DWARN<br><br>#<br># hdfs audit logging<br>#<b=
r>hdfs.audit.logger=3DINFO,NullAppender<br>hdfs.audit.log.maxfilesize=3D256=
MB<br>hdfs.audit.log.maxbackupindex=3D20<br>log4j.logger.org.apache.hadoop.=
hdfs.server.namenode.FSNamesystem.audit=3D${hdfs.audit.logger}<br>log4j.add=
itivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=3Dfalse<b=
r>log4j.appender.RFAAUDIT=3Dorg.apache.log4j.RollingFileAppender<br>log4j.a=
ppender.RFAAUDIT.File=3D${hadoop.log.dir}/hdfs-audit.log<br>log4j.appender.=
RFAAUDIT.layout=3Dorg.apache.log4j.PatternLayout<br>log4j.appender.RFAAUDIT=
.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m%n<br>log4j.appender.RF=
AAUDIT.MaxFileSize=3D${hdfs.audit.log.maxfilesize}<br>log4j.appender.RFAAUD=
IT.MaxBackupIndex=3D${hdfs.audit.log.maxbackupindex}<br><br>#<br># NameNode=
 metrics logging.<br># The default is to retain two namenode-metrics.log fi=
les up to 64MB each.<br>#<br>namenode.metrics.logger=3DINFO,NullAppender<br=
>log4j.logger.NameNodeMetricsLog=3D${namenode.metrics.logger}<br>log4j.addi=
tivity.NameNodeMetricsLog=3Dfalse<br>log4j.appender.NNMETRICSRFA=3Dorg.apac=
he.log4j.RollingFileAppender<br>log4j.appender.NNMETRICSRFA.File=3D${hadoop=
.log.dir}/namenode-metrics.log<br>log4j.appender.NNMETRICSRFA.layout=3Dorg.=
apache.log4j.PatternLayout<br>log4j.appender.NNMETRICSRFA.layout.Conversion=
Pattern=3D%d{ISO8601} %m%n<br>log4j.appender.NNMETRICSRFA.MaxBackupIndex=3D=
1<br>log4j.appender.NNMETRICSRFA.MaxFileSize=3D64MB<br><br>#<br># DataNode =
metrics logging.<br># The default is to retain two datanode-metrics.log fil=
es up to 64MB each.<br>#<br>datanode.metrics.logger=3DINFO,NullAppender<br>=
log4j.logger.DataNodeMetricsLog=3D${datanode.metrics.logger}<br>log4j.addit=
ivity.DataNodeMetricsLog=3Dfalse<br>log4j.appender.DNMETRICSRFA=3Dorg.apach=
e.log4j.RollingFileAppender<br>log4j.appender.DNMETRICSRFA.File=3D${hadoop.=
log.dir}/datanode-metrics.log<br>log4j.appender.DNMETRICSRFA.layout=3Dorg.a=
pache.log4j.PatternLayout<br>log4j.appender.DNMETRICSRFA.layout.ConversionP=
attern=3D%d{ISO8601} %m%n<br>log4j.appender.DNMETRICSRFA.MaxBackupIndex=3D1=
<br>log4j.appender.DNMETRICSRFA.MaxFileSize=3D64MB<br><br>#<br># mapred aud=
it logging<br>#<br>mapred.audit.logger=3DINFO,NullAppender<br>mapred.audit.=
log.maxfilesize=3D256MB<br>mapred.audit.log.maxbackupindex=3D20<br>log4j.lo=
gger.org.apache.hadoop.mapred.AuditLogger=3D${mapred.audit.logger}<br>log4j=
.additivity.org.apache.hadoop.mapred.AuditLogger=3Dfalse<br>log4j.appender.=
MRAUDIT=3Dorg.apache.log4j.RollingFileAppender<br>log4j.appender.MRAUDIT.Fi=
le=3D${hadoop.log.dir}/mapred-audit.log<br>log4j.appender.MRAUDIT.layout=3D=
org.apache.log4j.PatternLayout<br>log4j.appender.MRAUDIT.layout.ConversionP=
attern=3D%d{ISO8601} %p %c{2}: %m%n<br>log4j.appender.MRAUDIT.MaxFileSize=
=3D${mapred.audit.log.maxfilesize}<br>log4j.appender.MRAUDIT.MaxBackupIndex=
=3D${mapred.audit.log.maxbackupindex}<br><br># Custom Logging levels<br><br=
>#log4j.logger.org.apache.hadoop.mapred.JobTracker=3DDEBUG<br>#log4j.logger=
.org.apache.hadoop.mapred.TaskTracker=3DDEBUG<br>#log4j.logger.org.apache.h=
adoop.hdfs.server.namenode.FSNamesystem.audit=3DDEBUG<br><br># Jets3t libra=
ry<br>log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=3D=
ERROR<br><br># AWS SDK &amp; S3A FileSystem<br>log4j.logger.com.amazonaws=
=3DERROR<br>log4j.logger.com.amazonaws.http.AmazonHttpClient=3DERROR<br>#lo=
g4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=3DWARN<br><br>log4j.logge=
r.org.apache.hadoop.fs.s3a.S3AFileSystem=3DDEBUG<br>log4j.logger.org.apache=
.hadoop.tools.mapred=3DDEBUG<br>#log4j.logger.org.apache.hadoop=3DDEBUG<br>=
<br>#<br># Event Counter Appender<br># Sends counts of logging messages at =
different severity levels to Hadoop Metrics.<br>#<br>log4j.appender.EventCo=
unter=3Dorg.apache.hadoop.log.metrics.EventCounter<br><br>#<br># Job Summar=
y Appender<br>#<br># Use following logger to send summary to separate file =
defined by<br># hadoop.mapreduce.jobsummary.log.file :<br># hadoop.mapreduc=
e.jobsummary.logger=3DINFO,JSA<br># <br>hadoop.mapreduce.jobsummary.logger=
=3D${hadoop.root.logger}<br>hadoop.mapreduce.jobsummary.log.file=3Dhadoop-m=
apreduce.jobsummary.log<br>hadoop.mapreduce.jobsummary.log.maxfilesize=3D25=
6MB<br>hadoop.mapreduce.jobsummary.log.maxbackupindex=3D20<br>log4j.appende=
r.JSA=3Dorg.apache.log4j.RollingFileAppender<br>log4j.appender.JSA.File=3D$=
{hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}<br>log4j.appender.=
JSA.MaxFileSize=3D${hadoop.mapreduce.jobsummary.log.maxfilesize}<br>log4j.a=
ppender.JSA.MaxBackupIndex=3D${hadoop.mapreduce.jobsummary.log.maxbackupind=
ex}<br>log4j.appender.JSA.layout=3Dorg.apache.log4j.PatternLayout<br>log4j.=
appender.JSA.layout.ConversionPattern=3D%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%=
n<br>log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=3D${had=
oop.mapreduce.jobsummary.logger}<br>log4j.additivity.org.apache.hadoop.mapr=
ed.JobInProgress$JobSummary=3Dfalse<br><br>#<br># shuffle connection log fr=
om shuffleHandler<br># Uncomment the following line to enable logging of sh=
uffle connections<br># log4j.logger.org.apache.hadoop.mapred.ShuffleHandler=
.audit=3DDEBUG<br><br>#<br># Yarn ResourceManager Application Summary Log<b=
r>#<br># Set the ResourceManager summary log filename<br>yarn.server.resour=
cemanager.appsummary.log.file=3Drm-appsummary.log<br># Set the ResourceMana=
ger summary log level and appender<br>yarn.server.resourcemanager.appsummar=
y.logger=3D${hadoop.root.logger}<br>#yarn.server.resourcemanager.appsummary=
.logger=3DINFO,RMSUMMARY<br><br># To enable AppSummaryLogging for the RM,<b=
r># set yarn.server.resourcemanager.appsummary.logger to<br># &lt;LEVEL&gt;=
,RMSUMMARY in hadoop-env.sh<br><br># Appender for ResourceManager Applicati=
on Summary Log<br># Requires the following properties to be set<br>#=C2=A0=
=C2=A0=C2=A0 - hadoop.log.dir (Hadoop Log directory)<br>#=C2=A0=C2=A0=C2=A0=
 - yarn.server.resourcemanager.appsummary.log.file (resource manager app su=
mmary log filename)<br>#=C2=A0=C2=A0=C2=A0 - yarn.server.resourcemanager.ap=
psummary.logger (resource manager app summary log level and appender)<br><b=
r>log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$A=
pplicationSummary=3D${yarn.server.resourcemanager.appsummary.logger}<br>log=
4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$Ap=
plicationSummary=3Dfalse<br>log4j.appender.RMSUMMARY=3Dorg.apache.log4j.Rol=
lingFileAppender<br>log4j.appender.RMSUMMARY.File=3D${hadoop.log.dir}/${yar=
n.server.resourcemanager.appsummary.log.file}<br>log4j.appender.RMSUMMARY.M=
axFileSize=3D256MB<br>log4j.appender.RMSUMMARY.MaxBackupIndex=3D20<br>log4j=
.appender.RMSUMMARY.layout=3Dorg.apache.log4j.PatternLayout<br>log4j.append=
er.RMSUMMARY.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m%n<br><br>#=
 HS audit log configs<br>#mapreduce.hs.audit.logger=3DINFO,HSAUDIT<br>#log4=
j.logger.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=3D${mapreduce.hs.a=
udit.logger}<br>#log4j.additivity.org.apache.hadoop.mapreduce.v2.hs.HSAudit=
Logger=3Dfalse<br>#log4j.appender.HSAUDIT=3Dorg.apache.log4j.DailyRollingFi=
leAppender<br>#log4j.appender.HSAUDIT.File=3D${hadoop.log.dir}/hs-audit.log=
<br>#log4j.appender.HSAUDIT.layout=3Dorg.apache.log4j.PatternLayout<br>#log=
4j.appender.HSAUDIT.layout.ConversionPattern=3D%d{ISO8601} %p %c{2}: %m%n<b=
r>#log4j.appender.HSAUDIT.DatePattern=3D.yyyy-MM-dd<br><br># Http Server Re=
quest Logs<br>#log4j.logger.http.requests.namenode=3DINFO,namenoderequestlo=
g<br>#log4j.appender.namenoderequestlog=3Dorg.apache.hadoop.http.HttpReques=
tLogAppender<br>#log4j.appender.namenoderequestlog.Filename=3D${hadoop.log.=
dir}/jetty-namenode-yyyy_mm_dd.log<br>#log4j.appender.namenoderequestlog.Re=
tainDays=3D3<br><br>#log4j.logger.http.requests.datanode=3DINFO,datanodereq=
uestlog<br>#log4j.appender.datanoderequestlog=3Dorg.apache.hadoop.http.Http=
RequestLogAppender<br>#log4j.appender.datanoderequestlog.Filename=3D${hadoo=
p.log.dir}/jetty-datanode-yyyy_mm_dd.log<br>#log4j.appender.datanoderequest=
log.RetainDays=3D3<br><br>#log4j.logger.http.requests.resourcemanager=3DINF=
O,resourcemanagerrequestlog<br>#log4j.appender.resourcemanagerrequestlog=3D=
org.apache.hadoop.http.HttpRequestLogAppender<br>#log4j.appender.resourcema=
nagerrequestlog.Filename=3D${hadoop.log.dir}/jetty-resourcemanager-yyyy_mm_=
dd.log<br>#log4j.appender.resourcemanagerrequestlog.RetainDays=3D3<br><br>#=
log4j.logger.http.requests.jobhistory=3DINFO,jobhistoryrequestlog<br>#log4j=
.appender.jobhistoryrequestlog=3Dorg.apache.hadoop.http.HttpRequestLogAppen=
der<br>#log4j.appender.jobhistoryrequestlog.Filename=3D${hadoop.log.dir}/je=
tty-jobhistory-yyyy_mm_dd.log<br>#log4j.appender.jobhistoryrequestlog.Retai=
nDays=3D3<br><br>#log4j.logger.http.requests.nodemanager=3DINFO,nodemanager=
requestlog<br>#log4j.appender.nodemanagerrequestlog=3Dorg.apache.hadoop.htt=
p.HttpRequestLogAppender<br>#log4j.appender.nodemanagerrequestlog.Filename=
=3D${hadoop.log.dir}/jetty-nodemanager-yyyy_mm_dd.log<br>#log4j.appender.no=
demanagerrequestlog.RetainDays=3D3<br><br># Appender for viewing informatio=
n for errors and warnings<br>yarn.ewma.cleanupInterval=3D300<br>yarn.ewma.m=
essageAgeLimitSeconds=3D86400<br>yarn.ewma.maxUniqueMessages=3D250<br>log4j=
.appender.EWMA=3Dorg.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppend=
er<br>log4j.appender.EWMA.cleanupInterval=3D${yarn.ewma.cleanupInterval}<br=
>log4j.appender.EWMA.messageAgeLimitSeconds=3D${yarn.ewma.messageAgeLimitSe=
conds}<br>log4j.appender.EWMA.maxUniqueMessages=3D${yarn.ewma.maxUniqueMess=
ages}<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
<br><br><br>hadoop-env.sh<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D<br>#<br># Licensed to the Apache Software Foundation =
(ASF) under one<br># or more contributor license agreements.=C2=A0 See the =
NOTICE file<br># distributed with this work for additional information<br>#=
 regarding copyright ownership.=C2=A0 The ASF licenses this file<br># to yo=
u under the Apache License, Version 2.0 (the<br># &quot;License&quot;); you=
 may not use this file except in compliance<br># with the License.=C2=A0 Yo=
u may obtain a copy of the License at<br>#<br>#=C2=A0=C2=A0=C2=A0=C2=A0 <a =
href=3D"http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/l=
icenses/LICENSE-2.0</a><br>#<br># Unless required by applicable law or agre=
ed to in writing, software<br># distributed under the License is distribute=
d on an &quot;AS IS&quot; BASIS,<br># WITHOUT WARRANTIES OR CONDITIONS OF A=
NY KIND, either express or implied.<br># See the License for the specific l=
anguage governing permissions and<br># limitations under the License.<br><b=
r># Set Hadoop-specific environment variables here.<br><br>##<br>## THIS FI=
LE ACTS AS THE MASTER FILE FOR ALL HADOOP PROJECTS.<br>## SETTINGS HERE WIL=
L BE READ BY ALL HADOOP COMMANDS.=C2=A0 THEREFORE,<br>## ONE CAN USE THIS F=
ILE TO SET YARN, HDFS, AND MAPREDUCE<br>## CONFIGURATION OPTIONS INSTEAD OF=
 xxx-env.sh.<br>##<br>## Precedence rules:<br>##<br>## {yarn-env.sh|hdfs-en=
v.sh} &gt; hadoop-env.sh &gt; hard-coded defaults<br>##<br>## {YARN_xyz|HDF=
S_xyz} &gt; HADOOP_xyz &gt; hard-coded defaults<br>##<br><br># Many of the =
options here are built from the perspective that users<br># may want to pro=
vide OVERWRITING values on the command line.<br># For example:<br>#<br>#=C2=
=A0 JAVA_HOME=3D/usr/java/testing hdfs dfs -ls<br>#<br># Therefore, the vas=
t majority (BUT NOT ALL!) of these defaults<br># are configured for substit=
ution and not append.=C2=A0 If append<br># is preferable, modify this file =
accordingly.<br><br>###<br># Generic settings for HADOOP<br>###<br><br># Te=
chnically, the only required environment variable is JAVA_HOME.<br># All ot=
hers are optional.=C2=A0 However, the defaults are probably not<br># prefer=
red.=C2=A0 Many sites configure these options outside of Hadoop,<br># such =
as in /etc/profile.d<br><br># The java implementation to use. By default, t=
his environment <br># variable is REQUIRED on ALL platforms except OS X!<br=
>export JAVA_HOME=3D/usr/lib/jvm/java-8-oracle<br><br># Location of Hadoop.=
=C2=A0 By default, Hadoop will attempt to determine<br># this location base=
d upon its execution path.<br>export HADOOP_PREFIX=3D/opt/hadoop-3.0.0-SNAP=
SHOT<br><br># Location of Hadoop&#39;s configuration information.=C2=A0 i.e=
., where this<br># file is probably living. Many sites will also set this i=
n the<br># same location where JAVA_HOME is defined.=C2=A0 If this is not d=
efined<br># Hadoop will attempt to locate it based upon its execution<br># =
path.<br>export HADOOP_CONF_DIR=3D$HADOOP_PREFIX/etc/hadoop<br><br># The ma=
ximum amount of heap to use (Java -Xmx).=C2=A0 If no unit <br># is provided=
, it will be converted to MB.=C2=A0 Daemons will <br># prefer any Xmx setti=
ng in their respective _OPT variable.<br># There is no default; the JVM wil=
l autoscale based upon machine<br># memory size.<br># export HADOOP_HEAPSIZ=
E_MAX=3D<br><br># The minimum amount of heap to use (Java -Xms).=C2=A0 If n=
o unit <br># is provided, it will be converted to MB.=C2=A0 Daemons will <b=
r># prefer any Xms setting in their respective _OPT variable.<br># There is=
 no default; the JVM will autoscale based upon machine<br># memory size.<br=
># export HADOOP_HEAPSIZE_MIN=3D<br><br># Extra Java runtime options for al=
l Hadoop commands. We don&#39;t support<br># IPv6 yet/still, so by default =
the preference is set to IPv4.<br># export HADOOP_OPTS=3D&quot;-Djava.net.p=
referIPv4Stack=3Dtrue&quot;<br><br># Some parts of the shell code may do sp=
ecial things dependent upon<br># the operating system.=C2=A0 We have to set=
 this here. See the next<br># section as to why....<br>export HADOOP_OS_TYP=
E=3D${HADOOP_OS_TYPE:-$(uname -s)}<br><br><br># Under certain conditions, J=
ava on OS X will throw SCDynamicStore errors<br># in the system logs.<br># =
See HADOOP-8719 for more information.=C2=A0 If one needs Kerberos<br># supp=
ort on OS X, one will want to change/remove this extra bit.<br>case ${HADOO=
P_OS_TYPE} in<br>=C2=A0 Darwin*)<br>=C2=A0=C2=A0=C2=A0 export HADOOP_OPTS=
=3D&quot;${HADOOP_OPTS} -Djava.security.krb5.realm=3D &quot;<br>=C2=A0=C2=
=A0=C2=A0 export HADOOP_OPTS=3D&quot;${HADOOP_OPTS} -Djava.security.krb5.kd=
c=3D &quot;<br>=C2=A0=C2=A0=C2=A0 export HADOOP_OPTS=3D&quot;${HADOOP_OPTS}=
 -Djava.security.krb5.conf=3D &quot;<br>=C2=A0 ;;<br>esac<br><br># Extra Ja=
va runtime options for some Hadoop commands<br># and clients (i.e., hdfs df=
s -blah).=C2=A0 These get appended to HADOOP_OPTS for <br># such commands.=
=C2=A0 In most cases, # this should be left empty and <br># let users suppl=
y it on the command line.<br># export HADOOP_CLIENT_OPTS=3D&quot;&quot;<br>=
<br>#<br># A note about classpaths.<br>#<br># The classpath is configured s=
uch that entries are stripped prior<br># to handing to Java based either up=
on duplication or non-existence.<br># Wildcards and/or directories are *NOT=
* expanded as the<br># de-duplication is fairly simple.=C2=A0 So if two dir=
ectories are in<br># the classpath that both contain awesome-methods-1.0.ja=
r,<br># awesome-methods-1.0.jar will still be seen by java.=C2=A0 But if<br=
># the classpath specifically has awesome-methods-1.0.jar from the<br># sam=
e directory listed twice, the last one will be removed.<br>#<br><br># An ad=
ditional, custom CLASSPATH.=C2=A0 This is really meant for<br># end users, =
but as an administrator, one might want to push<br># something extra in her=
e too, such as the jar to the topology<br># method.=C2=A0 Just be sure to a=
ppend to the existing HADOOP_USER_CLASSPATH<br># so end users have a way to=
 add stuff.<br># export HADOOP_USER_CLASSPATH=3D&quot;/some/cool/path/on/yo=
ur/machine&quot;<br><br># Should HADOOP_USER_CLASSPATH be first in the offi=
cial CLASSPATH?<br># export HADOOP_USER_CLASSPATH_FIRST=3D&quot;yes&quot;<b=
r><br># If HADOOP_USE_CLIENT_CLASSLOADER is set, HADOOP_CLASSPATH along wit=
h the main<br># jar are handled by a separate isolated client classloader. =
If it is set,<br># HADOOP_USER_CLASSPATH_FIRST is ignored. Can be defined b=
y doing<br># export HADOOP_USE_CLIENT_CLASSLOADER=3Dtrue<br><br># HADOOP_CL=
IENT_CLASSLOADER_SYSTEM_CLASSES overrides the default definition of<br># sy=
stem classes for the client classloader when HADOOP_USE_CLIENT_CLASSLOADER<=
br># is enabled. Names ending in &#39;.&#39; (period) are treated as packag=
e names, and<br># names starting with a &#39;-&#39; are treated as negative=
 matches. For example,<br># export HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES=
=3D&quot;-org.apache.hadoop.UserClass,java.,javax.,org.apache.hadoop.&quot;=
<br><br># You need the hadoop-aws-3.0.0-SNAPSHOT.jar (or similar) in your C=
LASSPATH<br># otherwise you might get the following error:<br># java.lang.C=
lassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not fou=
nd<br>export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$HADOOP_PREFIX/share/hado=
op/tools/lib/*<br><br>###<br># Options for remote shell connectivity<br>###=
<br><br># There are some optional components of hadoop that allow for<br># =
command and control of remote hosts.=C2=A0 For example,<br># start-dfs.sh w=
ill attempt to bring up all NNs, DNS, etc.<br><br># Options to pass to SSH =
when one of the &quot;log into a host and<br># start/stop daemons&quot; scr=
ipts is executed<br># export HADOOP_SSH_OPTS=3D&quot;-o BatchMode=3Dyes -o =
StrictHostKeyChecking=3Dno -o ConnectTimeout=3D10s&quot;<br><br># The built=
-in ssh handler will limit itself to 10 simultaneous connections.<br># For =
pdsh users, this sets the fanout size ( -f )<br># Change this to increase/d=
ecrease as necessary.<br># export HADOOP_SSH_PARALLEL=3D10<br><br># Filenam=
e which contains all of the hosts for any remote execution<br># helper scri=
pts # such as slaves.sh, start-dfs.sh, etc.<br># export HADOOP_SLAVES=3D&qu=
ot;${HADOOP_CONF_DIR}/slaves&quot;<br><br>###<br># Options for all daemons<=
br>###<br>#<br><br>#<br># Many options may also be specified as Java proper=
ties.=C2=A0 It is<br># very common, and in many cases, desirable, to hard-s=
et these<br># in daemon _OPTS variables.=C2=A0 Where applicable, the approp=
riate<br># Java property is also identified.=C2=A0 Note that many are re-us=
ed<br># or set differently in certain contexts (e.g., secure vs<br># non-se=
cure)<br>#<br><br># Where (primarily) daemon log files are stored.=C2=A0 # =
$HADOOP_PREFIX/logs <br># by default.<br># Java property: hadoop.log.dir<br=
>export HADOOP_LOG_DIR=3D/var/log/hadoop<br><br># A string representing thi=
s instance of hadoop. $USER by default.<br># This is used in writing log an=
d pid files, so keep that in mind!<br># Java property: hadoop.id.str<br># e=
xport HADOOP_IDENT_STRING=3D$USER<br><br># How many seconds to pause after =
stopping a daemon<br># export HADOOP_STOP_TIMEOUT=3D5<br><br># Where pid fi=
les are stored.=C2=A0 /tmp by default.<br># export HADOOP_PID_DIR=3D/tmp<br=
><br># Default log4j setting for interactive commands<br># Java property: h=
adoop.root.logger<br># export HADOOP_ROOT_LOGGER=3DINFO,console<br><br># De=
fault log4j setting for daemons spawned explicitly by <br># --daemon option=
 of hadoop, hdfs, mapred and yarn command.<br># Java property: hadoop.root.=
logger<br>#export HADOOP_DAEMON_ROOT_LOGGER=3DINFO,RFA<br>export HADOOP_DAE=
MON_ROOT_LOGGER=3DDEBUG,RFA<br><br># Default log level and output location =
for security-related messages.<br># You will almost certainly want to chang=
e this on a per-daemon basis via<br># the Java property (i.e., -Dhadoop.sec=
urity.logger=3Dfoo). (Note that the<br># defaults for the NN and 2NN overri=
de this by default.)<br># Java property: hadoop.security.logger<br># export=
 HADOOP_SECURITY_LOGGER=3DINFO,NullAppender<br><br># Default log level for =
file system audit messages.<br># Generally, this is specifically set in the=
 namenode-specific<br># options line.<br># Java property: hdfs.audit.logger=
<br># export HADOOP_AUDIT_LOGGER=3DINFO,NullAppender<br><br># Default proce=
ss priority level<br># Note that sub-processes will also run at this level!=
<br># export HADOOP_NICENESS=3D0<br><br># Default name for the service leve=
l authorization file<br># Java property: hadoop.policy.file<br># export HAD=
OOP_POLICYFILE=3D&quot;hadoop-policy.xml&quot;<br><br>#<br># NOTE: this is =
not used by default!=C2=A0 &lt;-----<br># You can define variables right he=
re and then re-use them later on.<br># For example, it is common to use the=
 same garbage collection settings<br># for all the daemons.=C2=A0 So one co=
uld define:<br>#<br># export HADOOP_GC_SETTINGS=3D&quot;-verbose:gc -XX:+Pr=
intGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps&quot;<br>#<br># =
.. and then use it as per the b option under the namenode.<br><br>###<br># =
Secure/privileged execution<br>###<br><br>#<br># Out of the box, Hadoop use=
s jsvc from Apache Commons to launch daemons<br># on privileged ports.=C2=
=A0 This functionality can be replaced by providing<br># custom functions.=
=C2=A0 See hadoop-functions.sh for more information.<br>#<br><br># The jsvc=
 implementation to use. Jsvc is required to run secure datanodes<br># that =
bind to privileged ports to provide authentication of data transfer<br># pr=
otocol.=C2=A0 Jsvc is not required if SASL is configured for authentication=
 of<br># data transfer protocol using non-privileged ports.<br># export JSV=
C_HOME=3D/usr/bin<br><br>#<br># This directory contains pids for secure and=
 privileged processes.<br>#export HADOOP_SECURE_PID_DIR=3D${HADOOP_PID_DIR}=
<br><br>#<br># This directory contains the logs for secure and privileged p=
rocesses.<br># Java property: hadoop.log.dir<br># export HADOOP_SECURE_LOG=
=3D${HADOOP_LOG_DIR}<br><br>#<br># When running a secure daemon, the defaul=
t value of HADOOP_IDENT_STRING<br># ends up being a bit bogus.=C2=A0 Theref=
ore, by default, the code will<br># replace HADOOP_IDENT_STRING with HADOOP=
_SECURE_xx_USER.=C2=A0 If one wants<br># to keep HADOOP_IDENT_STRING untouc=
hed, then uncomment this line.<br># export HADOOP_SECURE_IDENT_PRESERVE=3D&=
quot;true&quot;<br><br>###<br># NameNode specific parameters<br>###<br><br>=
# Default log level and output location for file system related change<br>#=
 messages. For non-namenode daemons, the Java property must be set in<br># =
the appropriate _OPTS if one wants something other than INFO,NullAppender<b=
r># Java property: hdfs.audit.logger<br># export HDFS_AUDIT_LOGGER=3DINFO,N=
ullAppender<br><br># Specify the JVM options to be used when starting the N=
ameNode.<br># These options will be appended to the options specified as HA=
DOOP_OPTS<br># and therefore may override any similar flags set in HADOOP_O=
PTS<br>#<br># a) Set JMX options<br># export HADOOP_NAMENODE_OPTS=3D&quot;-=
Dcom.sun.management.jmxremote=3Dtrue -Dcom.sun.management.jmxremote.authent=
icate=3Dfalse -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.sun.manageme=
nt.jmxremote.port=3D1026&quot;<br>#<br># b) Set garbage collection logs<br>=
# export HADOOP_NAMENODE_OPTS=3D&quot;${HADOOP_GC_SETTINGS} -Xloggc:${HADOO=
P_LOG_DIR}/gc-rm.log-$(date +&#39;%Y%m%d%H%M&#39;)&quot;<br>#<br># c) ... o=
r set them directly<br># export HADOOP_NAMENODE_OPTS=3D&quot;-verbose:gc -X=
X:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:${H=
ADOOP_LOG_DIR}/gc-rm.log-$(date +&#39;%Y%m%d%H%M&#39;)&quot;<br><br># this =
is the default:<br># export HADOOP_NAMENODE_OPTS=3D&quot;-Dhadoop.security.=
logger=3DINFO,RFAS&quot;<br><br>###<br># SecondaryNameNode specific paramet=
ers<br>###<br># Specify the JVM options to be used when starting the Second=
aryNameNode.<br># These options will be appended to the options specified a=
s HADOOP_OPTS<br># and therefore may override any similar flags set in HADO=
OP_OPTS<br>#<br># This is the default:<br># export HADOOP_SECONDARYNAMENODE=
_OPTS=3D&quot;-Dhadoop.security.logger=3DINFO,RFAS&quot;<br><br>###<br># Da=
taNode specific parameters<br>###<br># Specify the JVM options to be used w=
hen starting the DataNode.<br># These options will be appended to the optio=
ns specified as HADOOP_OPTS<br># and therefore may override any similar fla=
gs set in HADOOP_OPTS<br>#<br># This is the default:<br># export HADOOP_DAT=
ANODE_OPTS=3D&quot;-Dhadoop.security.logger=3DERROR,RFAS&quot;<br><br># On =
secure datanodes, user to run the datanode as after dropping privileges.<br=
># This **MUST** be uncommented to enable secure HDFS if using privileged p=
orts<br># to provide authentication of data transfer protocol.=C2=A0 This *=
*MUST NOT** be<br># defined if SASL is configured for authentication of dat=
a transfer protocol<br># using non-privileged ports.<br># This will replace=
 the hadoop.id.str Java property in secure mode.<br># export HADOOP_SECURE_=
DN_USER=3Dhdfs<br><br># Supplemental options for secure datanodes<br># By d=
efault, Hadoop uses jsvc which needs to know to launch a<br># server jvm.<b=
r># export HADOOP_DN_SECURE_EXTRA_OPTS=3D&quot;-jvm server&quot;<br><br># W=
here datanode log files are stored in the secure data environment.<br># Thi=
s will replace the hadoop.log.dir Java property in secure mode.<br># export=
 HADOOP_SECURE_DN_LOG_DIR=3D${HADOOP_SECURE_LOG_DIR}<br><br># Where datanod=
e pid files are stored in the secure data environment.<br># export HADOOP_S=
ECURE_DN_PID_DIR=3D${HADOOP_SECURE_PID_DIR}<br><br>###<br># NFS3 Gateway sp=
ecific parameters<br>###<br># Specify the JVM options to be used when start=
ing the NFS3 Gateway.<br># These options will be appended to the options sp=
ecified as HADOOP_OPTS<br># and therefore may override any similar flags se=
t in HADOOP_OPTS<br>#<br># export HADOOP_NFS3_OPTS=3D&quot;&quot;<br><br># =
Specify the JVM options to be used when starting the Hadoop portmapper.<br>=
# These options will be appended to the options specified as HADOOP_OPTS<br=
># and therefore may override any similar flags set in HADOOP_OPTS<br>#<br>=
# export HADOOP_PORTMAP_OPTS=3D&quot;-Xmx512m&quot;<br><br># Supplemental o=
ptions for priviliged gateways<br># By default, Hadoop uses jsvc which need=
s to know to launch a<br># server jvm.<br># export HADOOP_NFS3_SECURE_EXTRA=
_OPTS=3D&quot;-jvm server&quot;<br><br># On privileged gateways, user to ru=
n the gateway as after dropping privileges<br># This will replace the hadoo=
p.id.str Java property in secure mode.<br># export HADOOP_PRIVILEGED_NFS_US=
ER=3Dnfsserver<br><br>###<br># ZKFailoverController specific parameters<br>=
###<br># Specify the JVM options to be used when starting the ZKFailoverCon=
troller.<br># These options will be appended to the options specified as HA=
DOOP_OPTS<br># and therefore may override any similar flags set in HADOOP_O=
PTS<br>#<br># export HADOOP_ZKFC_OPTS=3D&quot;&quot;<br><br>###<br># Quorum=
JournalNode specific parameters<br>###<br># Specify the JVM options to be u=
sed when starting the QuorumJournalNode.<br># These options will be appende=
d to the options specified as HADOOP_OPTS<br># and therefore may override a=
ny similar flags set in HADOOP_OPTS<br>#<br># export HADOOP_JOURNALNODE_OPT=
S=3D&quot;&quot;<br><br>###<br># HDFS Balancer specific parameters<br>###<b=
r># Specify the JVM options to be used when starting the HDFS Balancer.<br>=
# These options will be appended to the options specified as HADOOP_OPTS<br=
># and therefore may override any similar flags set in HADOOP_OPTS<br>#<br>=
# export HADOOP_BALANCER_OPTS=3D&quot;&quot;<br><br>###<br># HDFS Mover spe=
cific parameters<br>###<br># Specify the JVM options to be used when starti=
ng the HDFS Mover.<br># These options will be appended to the options speci=
fied as HADOOP_OPTS<br># and therefore may override any similar flags set i=
n HADOOP_OPTS<br>#<br># export HADOOP_MOVER_OPTS=3D&quot;&quot;<br><br>###<=
br># Advanced Users Only!<br>###<br><br>#<br># When building Hadoop, one ca=
n add the class paths to the commands<br># via this special env var:<br># e=
xport HADOOP_ENABLE_BUILD_PATHS=3D&quot;true&quot;<br><br>#<br># To prevent=
 accidents, shell commands be (superficially) locked<br># to only allow cer=
tain users to execute certain subcommands.<br>#<br># For example, to limit =
who can execute the namenode command,<br># export HADOOP_namenode_USER=3Dhd=
fs<br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br=
><br><br><br>--<br>Emre Sevin=C3=A7</div></div></div></div>

--047d7b3a8054846540052b3f96b0--