hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2227) wrong usage of mapred.local.dir.minspacestart
Date Mon, 10 Dec 2007 12:50:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550044
] 

Amareshwari Sri Ramadasu commented on HADOOP-2227:
--------------------------------------------------

We have two options here. 
1.    The simple approach of just checking whether at least one of the disks has enough space.
This will cover all the intermediate outputs generation/copy cases (for maps and reduces respectively).
The bulk of the other operations that the tasktracker does to do with disks is copying the
job jar file, and, localizing the tasks' cache. Both these are done before the task launch
(process spawn) and the tasktracker won't even launch the process if it encounters a problem
in this part (it will throw an exception). The tasktracker then marks that task as having
"failed". This behavior should be okay since we haven't spent much time executing the task
yet.

2.   The other option is to change things like localizeJob, localizeTask and localizeCache,
to use the LocalDirAllocator but in order to do that, we have to know the sizes of the cache
files. This is doable, but involves an RPC.

Thoughts?




> wrong usage of mapred.local.dir.minspacestart 
> ----------------------------------------------
>
>                 Key: HADOOP-2227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.15.1
>            Reporter: Christian Kunz
>            Assignee: Amareshwari Sri Ramadasu
>            Priority: Critical
>
> As I understand it, mapred.local.dir.minspacestart should be used to specify that a map
or reduce task requires a minimum disk space to be executed. When several disks are available,
and mapred.local.dir lists all of them, then the free space should be added up. But TaskTracker.enoughFreeSpace
returns false whenever one of the disks listed in mapred.local.dir has less than the space
specified by mapred.local.dir.minspacestart. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message