chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CHUKWA-757) Supply example log4j.properties to stream Hadoop logs to Chukwa
Date Fri, 19 Jun 2015 01:52:00 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eric Yang updated CHUKWA-757:
-----------------------------
    Attachment: CHUKWA-757.patch

In Hadoop-env.sh, user also needs to define:

{code}
export HADOOP_NAMENODE_OPTS="-Dhadoop.log.port=9096 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS}
-Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.log.port=9098 -Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.log.port=9097 -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS}
-Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
{code}

In yarn-env.sh:
{code}
export YARN_RESOURCEMANAGER_OPTS="-Dhadoop.log.port=9099"
export YARN_NODEMANAGER_OPTS="-Dhadoop.log.port=9100"
{code}

The ports need to align with port number in initial_adaptors for this to work correctly.

> Supply example log4j.properties to stream Hadoop logs to Chukwa
> ---------------------------------------------------------------
>
>                 Key: CHUKWA-757
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-757
>             Project: Chukwa
>          Issue Type: Sub-task
>          Components: Data Collection
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: CHUKWA-757.patch
>
>
> $CHUKWA_CONF_DIR/hadoop-log4j.properties file contains a list of appender to stream Hadoop
logs to Chukwa.  This was designed for Hadoop 0.20.x.  For supporting Hadoop 2.x, we need
to update the log4j.properties example, and hadoop-metrics2.properties files as examples of
how to stream logs and metrics to Chukwa.
> On Chukwa side, the initial_adaptors file also need to register SocketAdaptors to stream
the data to Chukwa.
> The define ports to stream data are:
> || Port || Description ||
> | 9095 | Hadoop Metrics |
> | 9096 | Hadoop NN Log |
> | 9097 | Hadoop SNN Log |
> | 9098 | Hadoop DN Log |
> | 9099 | Hadoop RM Log |
> | 9100 | Hadoop NM Log |
> | 9101 | Hadoop HS Log |
> | 9102 | Hadoop RM Summary |
> | 9103 | Hadoop TLA  |



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message