hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area
Date Mon, 21 Jul 2008 22:55:31 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ari Rabkin updated HADOOP-9:

    Attachment: hadoop9.patch

Patch fixing this issue.

> mapred.local.dir  temp dir. space allocation limited by smallest area
> ---------------------------------------------------------------------
>                 Key: HADOOP-9
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: all
>            Reporter: Paul Baclace
>            Assignee: Ari Rabkin
>         Attachments: hadoop9.patch
> When mapred.local.dir is used to specify multiple  temp dir. areas, space allocation
limited by smallest area because the temp dir. selection algorithm is "round robin starting
from a randomish point".   When round robin is used with approximately constant sized chunks,
the smallest area runs out of space first, and this is a fatal error. 
> Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available
> I wrote a patch to JobConf (currenly being tested) which uses df to check available space
(once a minute or less often) and then uses an efficient roulette selection to do allocation
weighted by magnitude of available space. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message