hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yeshavora (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5180) Running wordcount with "-Ddfs.client.read.shortcircuit=true/false" fails to get proper message on syslogs
Date Fri, 26 Apr 2013 19:12:15 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643146#comment-13643146
] 

yeshavora commented on MAPREDUCE-5180:
--------------------------------------

Adding 'log4j.logger.org.apache.hadoop':'DEBUG' in task-log4j.properties resolves this issue.

                
> Running wordcount with "-Ddfs.client.read.shortcircuit=true/false" fails to get proper
message on syslogs
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5180
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5180
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 1.3.0
>            Reporter: yeshavora
>             Fix For: 1.3.0
>
>         Attachments: Screen Shot 2013-04-24 at 1.12.31 PM.png
>
>
> Running wordcount job with -Ddfs.client.read.shortcircuit=true/false fails to mention
"hdfs.DFSClient: Short circuit read is true" or "hdfs.DFSClient: Short circuit read is false"
messages in syslogs. 
> Attaching screen shot of syslog output for Hadoop 1.1.2. The above message was present
in the logs earlier.
> Syslog Output of Hadoop 1.3
> 2013-04-18 13:07:08,265 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library
> 2013-04-18 13:07:10,002 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit
code 0
> 2013-04-18 13:07:10,577 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin
: org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4edc41c5
> 2013-04-18 13:07:10,682 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://node1:port1/input1.txt:0+215754
> 2013-04-18 13:07:10,706 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 200
> 2013-04-18 13:07:10,910 INFO org.apache.hadoop.mapred.MapTask: data buffer = 150994944/167772160
> 2013-04-18 13:07:10,910 INFO org.apache.hadoop.mapred.MapTask: record buffer = 2359296/2621440
> 2013-04-18 13:07:10,920 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy
native library is available
> 2013-04-18 13:07:10,920 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy
native library loaded
> 2013-04-18 13:07:10,934 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native
gpl library
> 2013-04-18 13:07:10,947 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded
&amp; initialized native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
> 2013-04-18 13:07:11,414 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map
output
> 2013-04-18 13:07:11,586 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
> 2013-04-18 13:07:11,962 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
> 2013-04-18 13:07:12,034 INFO org.apache.hadoop.mapred.Task: Task:attempt_201304181305_0001_m_000000_0
is done. And is in the process of commiting
> 2013-04-18 13:07:12,106 INFO org.apache.hadoop.mapred.Task: Task &apos;attempt_201304181305_0001_m_000000_0&apos;
done.
> 2013-04-18 13:07:12,152 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing
logs&apos; truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-04-18 13:07:12,637 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache
for UID to User mapping with a cache timeout of 14400 seconds.
> 2013-04-18 13:07:12,637 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName mapred
for UID 2002 from the native implementation

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message