Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 16 Dec 2016 18:37:58 +0000 (UTC)
From: "Chris Trezzo (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12724549.1404162748000.541717.1481913478551@Atlassian.JIRA>
In-Reply-To: <JIRA.12724549.1404162748000@Atlassian.JIRA>
References: <JIRA.12724549.1404162748000@Atlassian.JIRA> <JIRA.12724549.1404162748627@arcas>
Subject: [jira] [Updated] (MAPREDUCE-5951) Add support for the YARN Shared
 Cache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 16 Dec 2016 18:38:00 -0000


     [ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Trezzo updated MAPREDUCE-5951:
------------------------------------
    Attachment: MAPREDUCE-5951-trunk.018.patch

Attached is v18 to fix the one checkstyle issue. There are three outstanding checkstyle issues that I am leaning towards not fixing as part of the patch. Please let me know your thoughts. They are the following:
bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:  private static ContainerLaunchContext createCommonContainerLaunchContext(:3: Method length is 172 lines (max allowed is 150).

This patch barely touches this method so it seems wrong to refactor the patch as part of this jira. I can file a separate jira to fix this.

bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:  public ApplicationSubmissionContext createApplicationSubmissionContext(:3: Method length is 249 lines (max allowed is 150).

The same reasoning applies to this warning as the previous issue. I can file a separate jira to fix this.

bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:572:  private static void parseDistributedCacheArtifacts(:23: More than 7 parameters (found 8).

This issue was caused by this patch adding an additional parameter to this method. I can fix the number of parameter issues, but that forces me to touch three existing calls to the deprecated DistributedCache api, which would fix 1 warning but create 3 new ones. It is a larger change to not use the deprecated api because the existing code is not set up to use it, furthermore use of the deprecated api is currently widespread in this code. My thoughts are that I will leave this warning as is.

> Add support for the YARN Shared Cache
> -------------------------------------
>
>                 Key: MAPREDUCE-5951
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify which set of resources they would like to cache (i.e. jobjar, libjars, archives, files).


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org