hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1568) TrackerDistributedCacheManager should do deleteLocalPath asynchronously
Date Mon, 26 Apr 2010 04:59:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860811#action_12860811
] 

Amareshwari Sriramadasu commented on MAPREDUCE-1568:
----------------------------------------------------

I agree that the current change is simple and safe, but I don't see any value getting added
by the patch. The file deletion is already asynchronous in a sense, after MAPREDUCE-1302,
which uses MRAsyncDiskService for deletion. Now, the new thread is added just for iterating
over the deleteSet. 
I think this jira should remove the deletion logic from getLocalCache so that Task will just
localize files it needs and the deletion is taken care by TrackerDIstributedCacheManager.
Also, with security coming in, deletion of the distributed cache should not be in Task's space.

> TrackerDistributedCacheManager should do deleteLocalPath asynchronously
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1568
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1568.txt
>
>
> TrackerDistributedCacheManager.deleteCache() has been improved:
> MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the main thread
and then delete them in the background 
> MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of directories
can also takes a long time)
> But the deleteLocalCache is still in the main thread of TaskRunner.run(). So it will
still slow down the task which triggers the deletion (originally this will blocks all tasks,
but it is fixed by MAPREDUCE-1098). Other tasks do not wait for the deletion. The task which
triggers the deletion should not wait for this either. TrackerDistributedCacheManager should
do deleteLocalPath() asynchronously.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message