hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2116) Job.local.dir to be exposed to tasks
Date Wed, 09 Jan 2008 06:25:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557149#action_12557149

Amareshwari Sri Ramadasu commented on HADOOP-2116:

I propose the following  the job cache directory structure to address the above needs:
                                  --------> job_jar_xml/
                                             ---------> job.jar
                                             ---------> job.xml
                                             ---------> unJarred directory
                                  --------> work/

And we can have the directories job_jar_xml, work and taskdir on different disks.
 mapred/local/tasktracker/jobcache/<jobid>/job_jar_xml/job.jar is available via mapred.jar
and mapred/local/tasktracker/jobcache/<jobid>/work is available via job.local.dir ,
which is an empty directory.


> Job.local.dir to be exposed to tasks
> ------------------------------------
>                 Key: HADOOP-2116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2116
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.3
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.16.0
> Currently, since all task cwds are created under a jobcache directory, users that need
a job-specific shared directory for use as scratch space, create ../work. This is hacky, and
will break when HADOOP-2115 is addressed. For such jobs, hadoop mapred should expose job.local.dir
via localized configuration.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message