hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1503) Support making additional 'LocalResources' available to running containers
Date Thu, 25 Aug 2016 00:05:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435976#comment-15435976

Junping Du commented on YARN-1503:

Thanks [~jianhe] for the document. 
The use cases and proposal make sense to me. However, I haven't found our solution details
for some failed over cases, like: AM or NM restart:
- In case of AM get failed and restart, does these localization effort need to redo again?
Do we have some way to track re-localization effort on running containers by communication
with NMs or shared some context across different attempts? 
- For NM, it could get just restart when receiving/replying additional localization request
from AM. We should consider to persistent it to NM state store before replying back to AM.
May be we should add these scenarios into detail design?

> Support making additional 'LocalResources' available to running containers
> --------------------------------------------------------------------------
>                 Key: YARN-1503
>                 URL: https://issues.apache.org/jira/browse/YARN-1503
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Jian He
>         Attachments: Continuous-resource-localization.pdf
> We have a use case, where additional resources (jars, libraries etc) need to be made
available to an already running container. Ideally, we'd like this to be done via YARN (instead
of having potentially multiple containers per node download resources on their own).
> Proposal:
>   NM to support an additional API where a list of resources can be specified. Something
like "localiceResource(ContainerId, Map<String, LocalResource>)
>   NM would also require an additional API to get state for these resources - "getLocalizationState(ContainerId)"
- which returns the current state of all local resources for the specified container(s).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message