hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From praveenesh kumar <praveen...@gmail.com>
Subject Re: Reduce > copy at 0.00 MB/s
Date Wed, 25 Jan 2012 14:15:42 GMT
Yeah , I am doing it, currently its on 20 %, I guess I have to raise it
more.
Funny thing is, its still happening after map is 100% completed.
when map is completed, it should not wait, right. But I see it still give
same message, for some time.

Thanks,
Praveenesh

On Wed, Jan 25, 2012 at 7:29 PM, Harsh J <harsh@cloudera.com> wrote:

> The copy phase fetches the map outputs. It may hang for a while if
> there are no newly completed map outputs to fetch yet.
>
> You can raise your reducers' slowstart value to have it not spend so
> many cycles waiting but rather start at 80-90% of map completions,
> instead of default 5%. This helps your MR performance overall, if you
> run multiple jobs at a time, as the reduce slots aren't wasted.
>
> On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <praveenesh@gmail.com>
> wrote:
> > Hey,
> >
> > Can anyone explain me what is reduce > copy phase in the reducer section
> ?
> > The (K,List(V)), is passed to the reducer. Is reduce > copy representing
> > copying of K,List(V) on the reducer from all mappers ?
> >
> > I am monitoring my jobs on the cluster, using Jobtracker url.
> > I am seeing for most of  my reducing jobs, something like this :
> >
> > task_201201250352_0001_r_000000    31.05% reduce > copy (395 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000001    30.73% reduce > copy (391 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> > task_201201250352_0001_r_000002    30.89% reduce > copy (393 of 424 at
> 0.00
> > MB/s) > 25-Jan-2012 03:54:06
> >
> >
> > ............................
> >
> > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > Is it because this reduce > copy is happening on the same machine.
> >
> > Thanks,
> > Praveenesh
>
>
>
> --
> Harsh J
> Customer Ops. Engineer, Cloudera
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message