hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
Date Fri, 28 Apr 2017 15:45:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989035#comment-15989035

Erik Krogen commented on MAPREDUCE-5951:

Ah, excellent point, [~jlowe]... I actually would love to hear the reasoning behind the current
strategy of <client uploads resource to HDFS -> AM downloads resource -> AM uploads
resource to SCM> rather than the seemingly more obvious/simpler <client uploads resource
to SCM>. Is this so that the uploading to SCM can be done by the NM, which is a privileged
user, to have more secure control over it?

[~ctrezzo], first off thanks for getting back so quickly! And for the pointer to YARN-5727;
that's an interesting issue. The public visibility solution is certainly simpler from the
YARN side and seems pretty reasonable from a point of expectation of burden on an application
("you want a publicly shared resource? put it somewhere public"). It  doesn't add _too_ much
complexity on the MR side, though having a separate staging directory just for public resources
is a bit cumbersome. It also means that other application developers will have to build the
same type of logic - in general I would lean towards more logic pushed into the YARN level
so that it is easy for application devs to support. I don't have good insight into how difficult
your initially proposed solution in YARN-5727 would be to implement, though.

> Add support for the YARN Shared Cache
> -------------------------------------
>                 Key: MAPREDUCE-5951
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5951-Overview.001.pdf, MAPREDUCE-5951-trunk.016.patch,
MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch,
MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch,
MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch,
MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch,
MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch
> Implement the necessary changes so that the MapReduce application can leverage the new
YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify which set
of resources they would like to cache (i.e. jobjar, libjars, archives, files).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message