hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Baclace (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area
Date Sat, 06 Nov 2010 05:28:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928934#action_12928934

Paul Baclace commented on HADOOP-9:


This issue was originally NUTCH-181 before Hadoop was split off.  I wrote a patch Dec. 29
2005 and used it at archive.org Jan-Feb 2006.  Looking at my old notes, I created this issue
on Jan. 11 2006, and prepared the patch on Feb. 28 2006, but it was either lost in a Jira
transition or the attachment somehow failed.  

When I looked at your patch yesterday, it was similar enough to what I remembered (5 years
ago) that I thought it must be a revision of the patch I did.  Today I found my source and
2005-2006 work notes and it is clear that you implemented the change without seeing mine.

Thanks for doing it in same "roulette-y" spirit of my lost patch! 

> mapred.local.dir  temp dir. space allocation limited by smallest area
> ---------------------------------------------------------------------
>                 Key: HADOOP-9
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: all
>            Reporter: Paul Baclace
>            Assignee: Ari Rabkin
>            Priority: Minor
>             Fix For: 0.19.0
>         Attachments: hadoop9.patch
> When mapred.local.dir is used to specify multiple  temp dir. areas, space allocation
limited by smallest area because the temp dir. selection algorithm is "round robin starting
from a randomish point".   When round robin is used with approximately constant sized chunks,
the smallest area runs out of space first, and this is a fatal error. 
> Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available
> I wrote a patch to JobConf (currenly being tested) which uses df to check available space
(once a minute or less often) and then uses an efficient roulette selection to do allocation
weighted by magnitude of available space. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message