hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sri Ramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2116) Job.local.dir to be exposed to tasks
Date Wed, 20 Feb 2008 09:01:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570614#action_12570614
] 

Amareshwari Sri Ramadasu commented on HADOOP-2116:
--------------------------------------------------

I feel even with permissions, DistributedCache behavior should be the same. If the same user
wants to share files across jobs, he should be allowed. And if he wants to share with other
user who has permissions to access should be allowed. TaskTracker need not worry about the
user permissions for localizing cache, those should be taken care in DistributedCache itself.
Permissions aspect of DistributedCache has to be handled in a different JIRA. 
I propose the new layout would be the same as Owen suggested without filecache as part of
job cache.
So, it is
{noformat}
mapred/local/taskTracker/jobcache/$jobid/
                                       work/                  -- the scratch space
                                       jars/                  -- expanded jar
                                       job.xml                -- the generic job conf
                                      $taskid/
                                           job.xml          -- task localized job conf
                                           output/          -- map outputs
                                           work/            -- cwd for task
mapred/local/taskTracker/archive/   -- distributed cache
{noformat}

Thoughts?

> Job.local.dir to be exposed to tasks
> ------------------------------------
>
>                 Key: HADOOP-2116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2116
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.3
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.17.0
>
>         Attachments: patch-2116.txt, patch-2116.txt, patch-2116.txt, patch-2116.txt
>
>
> Currently, since all task cwds are created under a jobcache directory, users that need
a job-specific shared directory for use as scratch space, create ../work. This is hacky, and
will break when HADOOP-2115 is addressed. For such jobs, hadoop mapred should expose job.local.dir
via localized configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message