hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hadoop hive <hadooph...@gmail.com>
Subject Re: Reduce > copy at 0.00 MB/s
Date Thu, 02 Feb 2012 06:58:42 GMT
Hey ,

Can any1 help me with this, i have increases the reduce slowstart to .25
but its still hangs after copy .
tell me what else i can change it to make it working fine.

regards
Vikas Srivastava

On Wed, Jan 25, 2012 at 7:45 PM, praveenesh kumar <praveenesh@gmail.com>wrote:

> Yeah , I am doing it, currently its on 20 %, I guess I have to raise it
> more.
> Funny thing is, its still happening after map is 100% completed.
> when map is completed, it should not wait, right. But I see it still give
> same message, for some time.
>
> Thanks,
> Praveenesh
>
> On Wed, Jan 25, 2012 at 7:29 PM, Harsh J <harsh@cloudera.com> wrote:
>
> > The copy phase fetches the map outputs. It may hang for a while if
> > there are no newly completed map outputs to fetch yet.
> >
> > You can raise your reducers' slowstart value to have it not spend so
> > many cycles waiting but rather start at 80-90% of map completions,
> > instead of default 5%. This helps your MR performance overall, if you
> > run multiple jobs at a time, as the reduce slots aren't wasted.
> >
> > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar <praveenesh@gmail.com>
> > wrote:
> > > Hey,
> > >
> > > Can anyone explain me what is reduce > copy phase in the reducer
> section
> > ?
> > > The (K,List(V)), is passed to the reducer. Is reduce > copy
> representing
> > > copying of K,List(V) on the reducer from all mappers ?
> > >
> > > I am monitoring my jobs on the cluster, using Jobtracker url.
> > > I am seeing for most of  my reducing jobs, something like this :
> > >
> > > task_201201250352_0001_r_000000    31.05% reduce > copy (395 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000001    30.73% reduce > copy (391 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > > task_201201250352_0001_r_000002    30.89% reduce > copy (393 of 424 at
> > 0.00
> > > MB/s) > 25-Jan-2012 03:54:06
> > >
> > >
> > > ............................
> > >
> > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine.
> > > Is it because this reduce > copy is happening on the same machine.
> > >
> > > Thanks,
> > > Praveenesh
> >
> >
> >
> > --
> > Harsh J
> > Customer Ops. Engineer, Cloudera
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message