hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Trezzo (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5951) Add support for the YARN Shared Cache
Date Fri, 16 Dec 2016 18:37:58 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Trezzo updated MAPREDUCE-5951:
------------------------------------
    Attachment: MAPREDUCE-5951-trunk.018.patch

Attached is v18 to fix the one checkstyle issue. There are three outstanding checkstyle issues
that I am leaning towards not fixing as part of the patch. Please let me know your thoughts.
They are the following:
bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
 private static ContainerLaunchContext createCommonContainerLaunchContext(:3: Method length
is 172 lines (max allowed is 150).

This patch barely touches this method so it seems wrong to refactor the patch as part of this
jira. I can file a separate jira to fix this.

bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
 public ApplicationSubmissionContext createApplicationSubmissionContext(:3: Method length
is 249 lines (max allowed is 150).

The same reasoning applies to this warning as the previous issue. I can file a separate jira
to fix this.

bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:572:
 private static void parseDistributedCacheArtifacts(:23: More than 7 parameters (found 8).

This issue was caused by this patch adding an additional parameter to this method. I can fix
the number of parameter issues, but that forces me to touch three existing calls to the deprecated
DistributedCache api, which would fix 1 warning but create 3 new ones. It is a larger change
to not use the deprecated api because the existing code is not set up to use it, furthermore
use of the deprecated api is currently widespread in this code. My thoughts are that I will
leave this warning as is.

> Add support for the YARN Shared Cache
> -------------------------------------
>
>                 Key: MAPREDUCE-5951
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch,
MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch,
MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch,
MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch,
MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can leverage the new
YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify which set
of resources they would like to cache (i.e. jobjar, libjars, archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message