hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
Date Thu, 05 Nov 2009 08:21:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773828#action_12773828
] 

Vinod K V commented on MAPREDUCE-1186:
--------------------------------------

Some attempts that I've verified which use DistributedCache got stuck doing chmod on the base-dir
for the distcache. When I myself tried to replicate this scenario, I found out that chmod
indeed takes a lot of time when number of files is large (For 35K files,2.7GB in archive directory,
it took 1.2 minutes.) This should be fixed, we shouldn't do a recursive chmod on the whole
archive directory for every task.

> While localizing a DistributedCache file, TT sets permissions recursively on the whole
base-dir
> -----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1186
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>             Fix For: 0.21.0
>
>
> This is a performance problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message