hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "W.P. McNeill" <bill...@gmail.com>
Subject Is something wrong when I'm shuffling 2 bytes into RAM?
Date Tue, 31 May 2011 16:37:05 GMT
For some of my jobs I'll see long stretches of log files that look like
this:

INFO mapred.ReduceTask: Shuffling 2 bytes (14 raw bytes) into RAM from
attempt_201105041713_5850_m_002764_0
INFO mapred.ReduceTask: Read 2 bytes from map-output for
attempt_201105041713_5850_m_002764_0
INFO mapred.ReduceTask: attempt_201105041713_5850_r_000017_2 Scheduled 1
outputs (0 slow hosts and74 dup hosts)
INFO mapred.ReduceTask: Rec #1 from attempt_201105041713_5850_m_002764_0 ->
(-1, -1) from hnode52.tuk2.intelius.com
INFO mapred.ReduceTask: header: attempt_201105041713_5850_m_002729_0,
compressed len: 14, decompressed len: 2
INFO mapred.ReduceTask: Shuffling 2 bytes (14 raw bytes) into RAM from
attempt_201105041713_5850_m_002729_0
INFO mapred.ReduceTask: Read 2 bytes from map-output for
attempt_201105041713_5850_m_002729_0
INFO mapred.ReduceTask: Rec #1 from attempt_201105041713_5850_m_002729_0 ->
(-1, -1) from hnode42.tuk2.intelius.com
INFO mapred.ReduceTask: attempt_201105041713_5850_r_000017_2 Scheduled 1
outputs (0 slow hosts and70 dup hosts)
INFO mapred.ReduceTask: attempt_201105041713_5850_r_000017_2 Scheduled 1
outputs (0 slow hosts and64 dup hosts)
INFO mapred.ReduceTask: header: attempt_201105041713_5850_m_003036_0,
compressed len: 14, decompressed len: 2

This looks really wrong to me. Am I correct in thinking that when I'm
shuffling 2 bytes into memory at a time I've got a real performance problem?
Does anyone have ideas as to what might be going on here?

The jobs that hit this sometimes work and sometimes fail for reasons that
may or may not be related to the logs excerpted above.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message