hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramya Sunil (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3307) Improve logging on the console during job execution
Date Fri, 28 Oct 2011 19:27:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138671#comment-13138671
] 

Ramya Sunil commented on MAPREDUCE-3307:
----------------------------------------

The redundant information is highlighted below:

{color:red} 
11/10/28 02:41:52 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
{color}
11/10/28 02:41:52 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at <hostname>/<ip:port>
{color:red} 
11/10/28 02:41:52 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol
interface org.apache.hadoop.yarn.api.ClientRMProtocol
{color}
11/10/28 02:41:52 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at <hostname>/<ip:port>
Running 100 maps.
Job started: Fri Oct 28 02:41:52 UTC 2011
{color:red} 
11/10/28 02:41:52 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/10/28 02:41:52 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager <hostname>/<ip:port>
11/10/28 02:41:52 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol
interface org.apache.hadoop.yarn.api.ClientRMProtocol
11/10/28 02:41:52 INFO mapred.ResourceMgrDelegate: Connected to ResourceManager at <hostname>/<ip:port>
{color}
11/10/28 02:41:53 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token <id> for
<user> on <ip:port>
11/10/28 02:41:53 INFO security.TokenCache: Got dt for hdfs://<host:port>;uri=<ip:port>;t.service=<ip:port>
{color:red} 
11/10/28 02:41:53 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token <id> for
<user> on <ip:port>
11/10/28 02:41:53 INFO security.TokenCache: Got dt for hdfs://<host:port>;uri=<ip:port>;t.service=<ip:port>
{color}
11/10/28 02:41:53 INFO mapreduce.JobSubmitter: number of splits:100
11/10/28 02:41:53 INFO mapred.YARNRunner: AppMaster capability = memory: 2048,
11/10/28 02:41:53 INFO mapred.YARNRunner: Command to launch container for ApplicationMaster
is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=<LOG_DIR>
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
11/10/28 02:41:53 INFO mapred.ResourceMgrDelegate: Submitted application <appID> to
ResourceManager
11/10/28 02:41:53 INFO mapred.ClientCache: Connecting to HistoryServer at: <host:port>
{color:red} 
11/10/28 02:41:53 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
{color}
11/10/28 02:41:53 INFO mapred.ClientCache: Connected to HistoryServer at: <host:port>
{color:red} 
11/10/28 02:41:53 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol
interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol
{color}
11/10/28 02:41:53 INFO mapreduce.Job: Running job: job_ID
11/10/28 02:41:54 INFO mapreduce.Job:  map 0% reduce 0%
11/10/28 02:42:05 INFO mapred.ClientServiceDelegate: Tracking Url of JOB is <url>
11/10/28 02:42:05 INFO mapred.ClientServiceDelegate: Connecting to <host:port>
{color:red} 
11/10/28 02:42:05 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/10/28 02:42:05 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol
interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocol
{color}
11/10/28 02:42:05 INFO security.ApplicationTokenSelector: Looking for a token with service
<ip:port>
11/10/28 02:42:05 INFO security.ApplicationTokenSelector: Token kind is YARN_APPLICATION_TOKEN
and the token's service name is <ip:port>
11/10/28 02:43:44 INFO mapreduce.Job:  map 2% reduce 0%
11/10/28 02:43:51 INFO mapreduce.Job:  map 3% reduce 0%
11/10/28 02:43:52 INFO mapreduce.Job:  map 5% reduce 0%
11/10/28 02:43:56 INFO mapreduce.Job:  map 10% reduce 0%

Also,
Printing, "Running job: job_ID" and then trying to connect to AM is misleading. This should
be fixed too.
                
> Improve logging on the console during job execution
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-3307
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3307
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Ramya Sunil
>             Fix For: 0.23.0
>
>
> There is a lot of redundant information being printed on the console and a not so intuitive
flow of events. We should improve the logging on console during job execution. More details
in the next comment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message