hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Oskarson (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-27) MapRed tries to allocate tasks to nodes that have no available disk space
Date Mon, 19 Jun 2006 14:53:30 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-27?page=all ]

Johan Oskarson updated HADOOP-27:
---------------------------------

    Attachment: minspace_mapredtask.patch

To begin with, disregard my last comment, I have no idea what I was thinking :)

Created this patch to stop a job from losing the map tasks already completed when a task tracker
runs out of space.
Instead, the task tracker will stop accepting new tasks if the space runs below a certain
threshold.
However, if for some reason space clears up it will start accepting tasks again.

If for example a reduce operation previously have been assigned to the task tracker there's
a chance it will run out of space anyway. So if the tracker goes below a second threshold
it will completely stop accepting new tasks until the job is done and also kill the reduce
operation running, or if none is found a map task. It will try to take the one with the least
progress.

The solution might not be ideal, but it's at least better then having the job fail all the
time because the task trackers drop off one by one.

Suggestions are of course welcome.
I've tested this on our tiny cluster and it seems to work fine, just saved me a couple of
hours of redundant computation on a big job

/Johan

> MapRed tries to allocate tasks to nodes that have no available disk space
> -------------------------------------------------------------------------
>
>          Key: HADOOP-27
>          URL: http://issues.apache.org/jira/browse/HADOOP-27
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Reporter: Mike Cafarella
>  Attachments: minspace_mapredtask.patch
>
>   What it says above.  MapRed TaskTrackers should not offer task service if the local
disk
> space is too constrained.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message