hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
Date Tue, 08 Dec 2009 09:41:18 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ravi Gummadi updated MAPREDUCE-896:

    Attachment: y896.v2.patch

> Was there a need to change CleanupQueue.addToQueue to take a FileSystem as argument instead
of Configuration ? It has caused more changes than required by this patch - like in JobTracker
and JobInProgress. Can we retain the original API and pass in a Configuration as before ?

With the new code you suggested, those changes are anyway needed in JobTracker and JobInProgress
and also it will be consistent with trunk's code. So sending FileSystem instead of Configuration.

> When adding a task directory to delete, we are adding paths from all the local directories
instead of just the one where files for the task are actually created. At a minimum, this
is more work being done than necessary. More importantly, I don't know if there are any side
effects this will cause. We can check which among the local directories the path belongs to
(by doing a startsWith on the path) and all only that I think.

As some of the files under workDir can be on different local directories, deletion of all
the paths is to be done.

Incorporated all other review comments. Also changed the permissions being set recursively
on the path to be deleted is changed to 777 as that is the default permission setting done
by TaskTracker when taskDir and workDir are created(just to be consistent with that).

A testcase of this patch depends on HADOOP-5771. So this patch needs to be applied on top
of HADOOP-5771.

Attaching new patch for Y! 20.
Please review and provide your comments.

> Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
> ------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-896
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-896
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>         Attachments: MR-896.patch, MR-896.v1.patch, y896.v1.patch, y896.v2.patch
> As of now, irrespective of the TaskController in use, TT itself does a full delete on
local files created by itself or job tasks. This step, depending upon TT's umask and the permissions
set by files by the user, for e.g in job-work/task-work or child.tmp directories, may or may
not go through successful completion fully. Thus is left an opportunity for abusing disk space
usage either accidentally or intentionally by TT/users.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message