hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guilin Sun (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-311) JobClient should use multiple volumes as hadoop.tmp.dir
Date Tue, 30 Mar 2010 12:18:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851369#action_12851369
] 

Guilin Sun commented on MAPREDUCE-311:
--------------------------------------

Maybe hadoop.log.dir should use multi-volumes too, and LocalDirAllocator could be used to
allocator log dirs.

> JobClient should use multiple volumes as hadoop.tmp.dir
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-311
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-311
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>         Environment: All
>            Reporter: Milind Bhandarkar
>
> Currently, hadoop.tmp.dir configuration variable allows specification of only a single
directory to be used as scratch space. In particular, on the job launcher nodes with multiple
volumes, this fails the entire job if the tmp.dir is somehow unusable. When the job launcher
nodes have multiple volumes, the tmp space availability can be improved by using multiple
volumes (either randomly or in round-robin.) The code for choosing a volume from a comma-separated
list of multiple volumes is already there for mapred.local.dir etc. That needs to be used
by job client as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message