hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <j...@apache.org>
Subject [jira] Assigned: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area
Date Fri, 13 Jun 2008 18:55:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ari Rabkin reassigned HADOOP-9:

    Assignee: Ari Rabkin  (was: Owen O'Malley)

> mapred.local.dir  temp dir. space allocation limited by smallest area
> ---------------------------------------------------------------------
>                 Key: HADOOP-9
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: all
>            Reporter: Paul Baclace
>            Assignee: Ari Rabkin
> When mapred.local.dir is used to specify multiple  temp dir. areas, space allocation
limited by smallest area because the temp dir. selection algorithm is "round robin starting
from a randomish point".   When round robin is used with approximately constant sized chunks,
the smallest area runs out of space first, and this is a fatal error. 
> Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available
> I wrote a patch to JobConf (currenly being tested) which uses df to check available space
(once a minute or less often) and then uses an efficient roulette selection to do allocation
weighted by magnitude of available space. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message