hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Vivancos <pedro.vivan...@vocali.net>
Subject Re: Mappers become less utilized as time goes on?
Date Wed, 04 Mar 2009 08:34:56 GMT
I have the same problem... Please let me know if you solve it and how to.

Thanks.



On Wed, Mar 4, 2009 at 2:49 AM, Nathan Marz <nathan@rapleaf.com> wrote:

> Nope... and there were no failed tasks.
>
>
>
> On Mar 3, 2009, at 5:16 PM, Runping Qi wrote:
>
>  Were task Trackers black-listed?
>>
>>
>> On Tue, Mar 3, 2009 at 3:25 PM, Nathan Marz <nathan@rapleaf.com> wrote:
>>
>>  I'm seeing some really bizarre behavior from Hadoop 0.19.1. I have a
>>> fairly
>>> large job with about 29000 map tasks and 72 reducers. there are 304 map
>>> task
>>> slots in the cluster. When the job starts, it runs 304 map tasks at a
>>> time.
>>> As time goes on the number of map tasks run concurrently drops. For at
>>> least
>>> half of the execution exactly 152 mappers were run at a time. Towards,
>>> the
>>> end , when there were only 100 or so tasks remaining, the number of
>>> concurrent mappers quickly fell to 2 at a time, bringing the end of the
>>> map
>>> phase to a crawl. This was the only job running on the cluster. Has
>>> anyone
>>> else seen behavior like this?
>>>
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message