hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2116) Job.local.dir to be exposed to tasks
Date Fri, 11 Jan 2008 06:41:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557918#action_12557918
] 

Owen O'Malley commented on HADOOP-2116:
---------------------------------------

*Ugh* is right.

I'd propose some better names:

$local/work/$jobid/
       cache/               -- file cache
       jars/                    -- expanded jar
       job.xml               -- the generic job conf
       $taskid/
             job.xml        -- task localized job conf
             output/         -- map outputs
             work/            -- cwd for task
 
with each of the leaf directories being placed independently on the partitions.

We should define localized attributes to point to where each of the leaf directories is.

I agree with Arun that we should re-work the -file option to use the file cache with symlinks.
    

> Job.local.dir to be exposed to tasks
> ------------------------------------
>
>                 Key: HADOOP-2116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2116
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.3
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.16.0
>
>         Attachments: patch-2116.txt, patch-2116.txt
>
>
> Currently, since all task cwds are created under a jobcache directory, users that need
a job-specific shared directory for use as scratch space, create ../work. This is hacky, and
will break when HADOOP-2115 is addressed. For such jobs, hadoop mapred should expose job.local.dir
via localized configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message