Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1604026254.1204707100997.JavaMail.jira@brutus>
Date: Wed, 5 Mar 2008 00:51:40 -0800 (PST)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-2925) [HOD] Create mapred system directory
 using a naming convention that will avoid clashes in multi-user shared
 cluster scenario.
In-Reply-To: <982917235.1204544210456.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-2925:
-------------------------------------

    Attachment: 2925.1.patch

This new patch preserves the idea of the earlier patch, but parametrizes the root directory of the mapred system directory. This was earlier hardcoded to /mapredsystem. But it was decided it will be better to make it a config parameter, so administrators have a choice of setting it to the value they want according to security policies.

The current path will be hodring.mapred-system-dir-root/userid/mapredsystem/torque-jobid. Note that setting hodring.mapred-system-dir-root to /user will make these get written to the user's home directory in DFS and thus negate the need for any management in configuring permissions for the system directory.

Another change included in the patch is do a small change to the parameter of the hod logs directory. So, this will now be hodring.log-destination-uri/userid/hod-logs/torque-jobid/.. The change is for the same reason.


> [HOD] Create mapred system directory using a naming convention that will avoid clashes in multi-user shared cluster scenario.
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2925
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.16.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.16.1
>
>         Attachments: 2925.1.patch, 2925.patch
>
>
> Currently, HOD generates the name of the mapredsystem directory using the name /mapredsystem/hostname-of-jobtracker.
> In HADOOP-2899, we ran into a scenario where this naming convention could lead to problems in case dfs permissions are enabled. While the bug should ideally be addressed in Hadoop M/R, it will be better that HOD does not generate names that can potentially clash across runs. One way to solve the problem is to do what HOD already does for local log and temp directories - name it using username.torque-job-id, which is going to be pretty unique mostly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.