hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1503) Continuous resource-localization for YARN containers
Date Tue, 23 Aug 2016 14:40:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432920#comment-15432920

Jason Lowe commented on YARN-1503:

Thanks for driving this, Jian!  Seems reasonable overall with just one concern about placing
the localization status in the container status.

The container status is currently sent from the NM to the RM during the NM heartbeat.  I know
some cluster setups use container launch contexts that are quite large due to tons of resources
to localize.  That means this design could cause a very significant increase in the NM heartbeat
request size when the NM is localizing containers.  If a job ran very wide then every node
in the cluster could be doing a very large localization.  In addition each node could have
dozens of tasks that all need the same large set of resources.  The localization status would
then be replicated for all those containers and the NM heartbeat would grow even more.

This makes me wonder if we should do one of the following:
- Omit the localization info for NM heartbeats?  The RM currently doesn't care, so sending
it is a waste.
- Provide a separate API to get the gory details of localization?  That way it's "cheap" for
someone to ask about container state when they don't care about localization details.  Others
that care whether the container is currently localizing and how far along can call the localization
status API.  Instead of a separate API we could extend GetContainerStatusesRequest with a
flag on whether localization details should be returned in the response.

> Continuous resource-localization for YARN containers
> ----------------------------------------------------
>                 Key: YARN-1503
>                 URL: https://issues.apache.org/jira/browse/YARN-1503
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Jian He
>         Attachments: Continuous-resource-localization.pdf
> We have a use case, where additional resources (jars, libraries etc) need to be made
available to an already running container. Ideally, we'd like this to be done via YARN (instead
of having potentially multiple containers per node download resources on their own).
> The goal is to enable NodeManagers to localize resources while container is running.
Today, resource-localization is always the first step before starting a container. It will
be useful if YARN can localize the resources continuously even while container is running.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message