hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
Date Tue, 12 May 2015 18:21:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540396#comment-14540396

Karthik Kambatla commented on MAPREDUCE-5951:

Sorry again for the delay here. Haven't been able to get a large enough chunk of time to review

Few comments based on the partial review I have been able to get to so far. Will post comments
for the remaining review as soon as I can:
# DistributedCache#getPathStringWithoutFragment: Is this to make sure we get the path to the
file and not its fields? Can we add comments to describe what the method is intended to do?
And, may be a different name? Can't think of a simpler one myself. 
# Job - have some javadoc suggestions, but may be it is simpler to post an updated patch once
the patch is ready.
# JobImpl - is the cleanup of upload-policies intended to be in init-transition? Is that because
we don't need the policies once we are done uploading the resources? 
# JobResourceUploader
## In isSharedCacheFilesEnabled and co., we should reverse the order of checks. isScmAvailable()
should come first as it is cheaper. Also, don't need the outer parentheses there. 
## Rename getFiles to mergeLocalAndCacheResources and make it static? 

> Add support for the YARN Shared Cache
> -------------------------------------
>                 Key: MAPREDUCE-5951
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch,
MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch
> Implement the necessary changes so that the MapReduce application can leverage the new
YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify which set
of resources they would like to cache (i.e. jobjar, libjars, archives, files).

This message was sent by Atlassian JIRA

View raw message