hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1197) Support changing resources of an allocated container
Date Thu, 12 Dec 2013 05:26:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846075#comment-13846075

Bikas Saha commented on YARN-1197:

I am afraid we seem to mixing things over here. This jira deals with the issue of increasing
and decreasing the resources of an allocated container. There are clear use cases for it like
mentioned in previous comments on this jira. e.g. having a long running worker daemon increase
and decrease its resources depending on load. We have already discussed at length on this
jira on how increasing a container resource is internally no different than requesting for
an additional container and merging it with an existing container. However the new container
followed by merge is way more complicated for the user and adds additional complexity to the
system (eg how to deal with the new container that was merged into the old one). This complexity
is in addition to the common work with simply increasing resources on a given container. wrt
the user, asking for a container and being able to increase its resources will give the same
effect as asking for many containers and merging them.
The scenario for YARN-1488 is logically different. That covers the case when an app wants
to use a shared service and purchases that service by transferring its own container resource
to that shared service that itself is running inside YARN. The consumer app may never need
to increase its own container resource. Secondly, the shared service is not requesting an
increase in its own container resources. So this jira does not come into the picture at all.

I believe we have a clear and cleanly separated piece of useful functionality being implemented
in this jira. We should go ahead and bring this work to completion and facilitate the creation
of long running services in YARN.

wrt doing this in a branch. There are new API's being added here for which functionality does
not exist or is not supported yet. And none of that code will get executed until clients actually
support doing it or someone writes code against it. So I dont think that any of this is going
to destabilize the code base. I agree that the scheduler changes are going to be complicated.
We can do them in the end when all the plumbing is in place and they could be separate jiras
for each scheduler. Of course, schedulers would want their own flags to turn this on/off.
So its not clear to me what benefits a branch would bring here but it would entail the overhead
of maintenance and lack of test automation. Does this mean that every feature addition to
YARN needs to be done in a branch? I propose we do this work in trunk and later merge it into
branch-2 when we are satisfied with its stability.

> Support changing resources of an allocated container
> ----------------------------------------------------
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: mapreduce-project.patch.ver.1, tools-project.patch.ver.1, yarn-1197-v2.pdf,
yarn-1197-v3.pdf, yarn-1197-v4.pdf, yarn-1197-v5.pdf, yarn-1197.pdf, yarn-api-protocol.patch.ver.1,
yarn-pb-impl.patch.ver.1, yarn-server-common.patch.ver.1, yarn-server-nodemanager.patch.ver.1,
> The current YARN resource management logic assumes resource allocated to a container
is fixed during the lifetime of it. When users want to change a resource 
> of an allocated container the only way is releasing it and allocating a new container
with expected size.
> Allowing run-time changing resources of an allocated container will give us better control
of resource usage in application side

This message was sent by Atlassian JIRA

View raw message