hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
Date Wed, 23 Dec 2009 05:53:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793937#action_12793937

Hemanth Yamijala commented on MAPREDUCE-896:

Some more details on the test failure in TestServiceLevelAuthorization:

The problem is with TestMiniMRWithDFS.checkTaskDirectories. As explained above, this method
verifies an exact list of localized directories on a tasktracker. To do so, it seems to wait
until the TTs become idle. But idleness is only a function of the number of active tasks.
When tasks complete, the TT adds directories to be deleted to the CleanupQueue, which by default,
asynchronously deletes them. Clearly that means that waiting for the TTs to become idle is
not a sufficient pre-condition for the testing of localized directory contents.

Like I explained, this problem exists on trunk as well. But this patch can increase the chances
of hitting it more often, because it does more work in the asynchronous portion. Given that,
I am not comfortable committing it, particularly as there's no blazing hurry for this patch
right now. We do run a risk of the patch going stale, but I am relying on the holiday season
to slow people down (*smile*). More importantly, we'll not make the test scene worse than
it already is.

As regards the fix, a simple solution may be to configure the MiniMRCluster with an inline
cleanup queue whenever we want to check the deletion of localized files and directories -
essentially in all test cases that use TestMiniMRWithDFS.checkTaskDirectories. Fortunately,
there are only four of them right now. Another option could be to wait until the cleanup queue
is empty. The disadvantage with this approach is that it uses the much maligned "sleep(100)
until some condition is satisfied" pattern. I would use the former approach instead, which
has worked well elsewhere. It may slow the tests down a tad bit, but hopefully not too badly.

> Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
> ------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-896
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-896
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>         Attachments: MR-896.patch, MR-896.v1.patch, MR-896.v2.patch, MR-896.v3.patch,
MR-896.v4.patch, MR-896.v5.patch, MR-896.v6.patch, y896.v1.patch, y896.v2.1.fix.patch, y896.v2.1.fix.v1.patch,
y896.v2.1.fix.v2.patch, y896.v2.1.patch, y896.v2.patch
> As of now, irrespective of the TaskController in use, TT itself does a full delete on
local files created by itself or job tasks. This step, depending upon TT's umask and the permissions
set by files by the user, for e.g in job-work/task-work or child.tmp directories, may or may
not go through successful completion fully. Thus is left an opportunity for abusing disk space
usage either accidentally or intentionally by TT/users.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message