hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2227) wrong usage of mapred.local.dir.minspacestart
Date Fri, 30 Nov 2007 12:49:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547119

Amareshwari Sri Ramadasu commented on HADOOP-2227:

bq. when one of the disks has enough space for a task, then TaskTracker.enoughFreeSpace should
return true.

With current behaviour , if we do this, localizeJob() and localizeTask() can fail because
they use getLocalPath() for localizing. In getLocalPath(), there is no space availability
check. And local file names are hashed to determine which device to store the file on.
One solution could be to change getLocalPath to use localDirAllocator as proposed in HADOOP-1991
and then do this change.  But there would be some cases where we do not know size of files
apriori, to check for disk space.


> wrong usage of mapred.local.dir.minspacestart 
> ----------------------------------------------
>                 Key: HADOOP-2227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.15.1
>            Reporter: Christian Kunz
>            Assignee: Amareshwari Sri Ramadasu
>            Priority: Critical
> As I understand it, mapred.local.dir.minspacestart should be used to specify that a map
or reduce task requires a minimum disk space to be executed. When several disks are available,
and mapred.local.dir lists all of them, then the free space should be added up. But TaskTracker.enoughFreeSpace
returns false whenever one of the disks listed in mapred.local.dir has less than the space
specified by mapred.local.dir.minspacestart. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message