hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area
Date Fri, 22 Aug 2008 12:34:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624768#action_12624768

Hudson commented on HADOOP-9:

Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])

> mapred.local.dir  temp dir. space allocation limited by smallest area
> ---------------------------------------------------------------------
>                 Key: HADOOP-9
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: all
>            Reporter: Paul Baclace
>            Assignee: Ari Rabkin
>            Priority: Minor
>             Fix For: 0.19.0
>         Attachments: hadoop9.patch
> When mapred.local.dir is used to specify multiple  temp dir. areas, space allocation
limited by smallest area because the temp dir. selection algorithm is "round robin starting
from a randomish point".   When round robin is used with approximately constant sized chunks,
the smallest area runs out of space first, and this is a fatal error. 
> Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available
> I wrote a patch to JobConf (currenly being tested) which uses df to check available space
(once a minute or less often) and then uses an efficient roulette selection to do allocation
weighted by magnitude of available space. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message