hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: reduce copy rate
Date Fri, 15 Apr 2011 16:52:25 GMT
Hello Baran,

On Fri, Apr 15, 2011 at 8:19 PM, baran cakici <barancakici@gmail.com> wrote:
> Hi,
> I have a question about copy speed by a MapReduce Job.I have a Cluster with
> 4 slave and 1 master, computers connected each other with one 8-Port-Switch
> (up to 1000Mbps).  Copy speed is by my Job 1,6 - 1,8MB.  Is it not too
> slow?

As Juwei has pointed out, the Reducer's copy phase is a progressive
one by default (Controlled by [1]). Reducers may be initialized when
5% of your maps may have reported completion, and begin copying early
outputs as they become available to it. Thus, you won't really see the
maximum copy rate possible all the time (since the data is pulled
gradually, but grows in size as map waves begin completing further).

[1] - The mapred.reduce.slowstart.completed.maps property @

Harsh J

View raw message