hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: does reduce > copy (at 0.52 MB/s) means network or other IO problem?
Date Tue, 05 Oct 2010 16:12:02 GMT
The reduce begins copying map outputs as they complete (starting at 5% of
them) and this transfer may be very meagre and thus the low rate of

Observe once all maps finish or near completion at their last wave, if the
network status shown is still slow then there is a problem, whose common
side effect would be failing reducers or long time waits before the sort
phase kicks in even if all mappers are already done.

Otherwise this isn't an issue. You can also increase the parallel fetching
factor of each reducer :)

On Oct 5, 2010 6:49 PM, "Vitaliy Semochkin" <vitaliy.se@gmail.com> wrote:


I often see reduce > copy (at 0.52 MB/s)  phase with such speed.
Despite in my cluster all 5 nodes are in same rack.
Does it mean any network or other IO problems, or other reasons can
cause such slow speed?

Thanks in Advance,
Vitaliy S

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message