hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
Date Fri, 28 Apr 2017 15:45:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989035#comment-15989035
] 

Erik Krogen commented on MAPREDUCE-5951:
----------------------------------------

Ah, excellent point, [~jlowe]... I actually would love to hear the reasoning behind the current
strategy of <client uploads resource to HDFS -> AM downloads resource -> AM uploads
resource to SCM> rather than the seemingly more obvious/simpler <client uploads resource
to SCM>. Is this so that the uploading to SCM can be done by the NM, which is a privileged
user, to have more secure control over it?

[~ctrezzo], first off thanks for getting back so quickly! And for the pointer to YARN-5727;
that's an interesting issue. The public visibility solution is certainly simpler from the
YARN side and seems pretty reasonable from a point of expectation of burden on an application
("you want a publicly shared resource? put it somewhere public"). It  doesn't add _too_ much
complexity on the MR side, though having a separate staging directory just for public resources
is a bit cumbersome. It also means that other application developers will have to build the
same type of logic - in general I would lean towards more logic pushed into the YARN level
so that it is easy for application devs to support. I don't have good insight into how difficult
your initially proposed solution in YARN-5727 would be to implement, though.

> Add support for the YARN Shared Cache
> -------------------------------------
>
>                 Key: MAPREDUCE-5951
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5951-Overview.001.pdf, MAPREDUCE-5951-trunk.016.patch,
MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch,
MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch,
MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch,
MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch,
MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch,
MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can leverage the new
YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify which set
of resources they would like to cache (i.e. jobjar, libjars, archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message