hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Rosenstrauch <dar...@darose.net>
Subject Task inititalizations logging to console in MRv2
Date Mon, 14 Apr 2014 17:10:00 GMT
Running into an issue with some early attempts at converting over to MRv2.

I'm submitting a job using an MRv1 client installation, onto an MRv2 
cluster.  However, the client is displaying initialization log messages 
for every single map task, like the log entries pasted below.  In 
addition to being unnecessary and wasteful, this is also resulting in 
the job taking a long time to get submitted.  (It's writing log output 
for several thousand map tasks before the job started.

MRv1 didn't used to do this.  Anyone know if there's any way to disable?

Thanks,

DR

---

14/04/14 16:56:19 INFO mapred.LocalJobRunner: Starting task: 
attempt_local438339852_0001_m_000127_0
14/04/14 16:56:19 WARN mapreduce.Counters: Group 
org.apache.hadoop.mapred.Task$Counter is deprecated. Use 
org.apache.hadoop.mapreduce.TaskCounter instead
14/04/14 16:56:19 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@245eca5e
14/04/14 16:56:19 INFO mapred.MapTask: Processing split: 
s3n://<my-s3-bucket>/merging/eventsBids/20140408/part-r-00157:0+10081976
14/04/14 16:56:19 INFO mapred.MapTask: Map output collector class = 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/04/14 16:56:19 INFO mapred.MapTask: io.sort.mb = 100
14/04/14 16:56:19 INFO mapred.LocalJobRunner: Starting task: 
attempt_local438339852_0001_m_000128_0
14/04/14 16:56:19 WARN mapreduce.Counters: Group 
org.apache.hadoop.mapred.Task$Counter is deprecated. Use 
org.apache.hadoop.mapreduce.TaskCounter instead
14/04/14 16:56:19 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d59a8ed
14/04/14 16:56:19 INFO mapred.MapTask: Processing split: 
s3n://<my-s3-bucket>/merging/eventsBids/20140408/part-r-00021:0+10081963
14/04/14 16:56:19 INFO mapred.MapTask: Map output collector class = 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/04/14 16:56:19 INFO mapred.MapTask: io.sort.mb = 100
...

Mime
View raw message