hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1422) Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in all mapred-local-directories succeeds always
Date Tue, 09 Mar 2010 03:07:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842892#action_12842892
] 

Amar Kamat commented on MAPREDUCE-1422:
---------------------------------------

bq. Please change the method name buildTaskControllerPathDeletionContexts() to buildTaskControllerTaskPathDeletionContexts()
I kept it that way to avoid unnecessary diffs in my patch. Let me try to change it in this
patch.

bq. AFAIK, task work directory is always cleaned up, but task directory is not cleaned up
sometimes until a job cleanup happens
Can you tell me a scenario where this can happen. But it better we add files to task dir too.

bq. Also, just so to avoid false positives concerning this kind of loop
Just to add more confidence to this patch, I added log statements and checked it. I will add
checks to make sure that atleast one file gets written. I see similar code in TestTaskTrackerLocalization#validateRemoveFiles().
I will add checks there too. Thoughts?

> Changing permissions of files/dirs under job-work-dir may be needed sothat cleaning up
of job-dir in all mapred-local-directories succeeds always
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1422
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>            Reporter: Ravi Gummadi
>            Assignee: Amar Kamat
>         Attachments: mapreduce-1422-test-v1.0.patch, mapreduce-1422-v1.4.2.patch, mapreduce-1422-v1.5.2.patch
>
>
> After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller binary
is launched for changing permissions of taskAttemptDir and taskWorkDir before cleaning up
of these directories sothat cleanup will be succeeded even if user had created files/dirs
under taskAttemptDir or taskWorkDir with non-writable permissions. Users can't create files/dirs
under job-dir directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions
and user can create files/dirs under job-work-dir with non-writable permissions, Changing
permissions of files/dirs under job-work-dir may be needed sothat cleaning up of job-dir in
all mapred-local-directories succeeds always.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message