hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: I've probably hit some system limits
Date Wed, 12 Aug 2009 20:19:12 GMT
On Wed, Aug 12, 2009 at 1:15 PM, Mayuran Yogarajah <
mayuran.yogarajah@casalemedia.com> wrote:

> I had 3 jobs running and I saw something a bit odd.  Two of the tasks
> are reducing, one of them is using all the reducers so the other is
> waiting, this is OK.  However the 3rd job is still in the mapping phase
> and even though the web interface shows map capacity at 96 I only
> see about 7-12 mappers actually running.  I'm wondering if theres
> some setting I need to change, perhaps I've hit some system limit.  Can
> someone point me in the right direction please?

Are there any pending mappers remaining? Are you using any scheduler?

> The other thing was that with the two jobs that are in the reducing phase,
> the reducer for one job wouldn't actually start until all the mappers of
> the
> _other_ job completed which seems kind of odd.  Is this expected ?

Reducers dont really start the "reduce" phase till the mappers are
completed. However, the process gets spawned off and the copying of the
intermediate keys from the mapper starts off.

> thanks,
> M

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message