Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1753650.1200984514456.JavaMail.jira@brutus>
Date: Mon, 21 Jan 2008 22:48:34 -0800 (PST)
From: "Amareshwari Sri Ramadasu (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-2116) Job.local.dir to be exposed to
 tasks
In-Reply-To: <27538009.1193600390785.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561229#action_12561229 ] 

Amareshwari Sri Ramadasu commented on HADOOP-2116:
--------------------------------------------------

A clarification regarding distributed cache:
The current behavior of distributed cache is that the distributed cache is shared among the jobs. The cache is localized under mapred/local/tasktracker/archive. i.e If two jobs want to localize files with same name, they actually share them unless they have different file timestamps.  Whenever a task releases cache, it decrements the reference count for the cache-id. Cache is cleaned up only when the cache size exceeds the allowed lize (local.cache.size).  
Is it the intended behavior, or should the cache be job specific? With the directory structure that Owen has suggested, it seems like cache should be job specific.

> Job.local.dir to be exposed to tasks
> ------------------------------------
>
>                 Key: HADOOP-2116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2116
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.14.3
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.16.0
>
>         Attachments: patch-2116.txt, patch-2116.txt, patch-2116.txt, patch-2116.txt
>
>
> Currently, since all task cwds are created under a jobcache directory, users that need a job-specific shared directory for use as scratch space, create ../work. This is hacky, and will break when HADOOP-2115 is addressed. For such jobs, hadoop mapred should expose job.local.dir via localized configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.