hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-914) (Umbrella) Support graceful decommission of nodemanager
Date Mon, 21 Dec 2015 14:59:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066518#comment-15066518
] 

Junping Du commented on YARN-914:
---------------------------------

Hi [~danzhi], thanks for sharing the information above and welcome to join the contribution
to Apache Hadoop.

bq. Our implementation is much in sync with the architecture and idea in the JIRA design document.
Good to hear that we are on the same page. One thing we need to pay attention is: we already
have many patches committed into trunk/branch-2.8. As an continuous developing effort on YARN,
we need to remove the code (current internal to yourself) for similar functionality or APIs
before contributing or it would take reviewer/committer more effort to differentiate which
functionalities/APIs are duplicated and which are not - that usually take much longer time.

bq. On the other hand, there are additional details and component level designs that the JIRA
design document not necessarily discuss or touch. These details naturally surfaced up during
the development iterations and the corresponding design became matured and stabilized.
I agree that the design document could miss some details of implementation in general. However,
we can find more background/details in JIRA discussion or patch implementation. Let me explain
below.

bq. One example is the DecommissioningNodeWatcher, which embedded in ResourceTrackingService,
tracks DECOMMISSIONING nodes status automatically and asynchronously after client/admin made
the graceful decommission request. Another example is per node decommission timeout support,
which is useful to decommission node that will be terminated soon.
Actually, our current design and committed patches already support timeout feature. There
are basically two ways to handle timeout: RM side or CLI side, both have pros and cons.
Per disussions above (https://issues.apache.org/jira/browse/YARN-914?focusedCommentId=14312677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14312677
and https://issues.apache.org/jira/browse/YARN-914?focusedCommentId=14312677&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14312677),
we (Jason, Vinod and I) all agreed to go with CLI way first and we already implement it in
sub JIRA (YARN-3225) and get committed. Of course, we are open for the other way of implementation,
but we do want it can be based on a switch on/off configuration that doesn't affect current
preferred option that we already implemented.

bq. Are you able to share these details in an "augmented" design doc? Agreeing on the design
would greatly help with review/commits later.
I would prefer the effort to abstract the different implementation for tracking/handling timeout.
This doesn't sounds like a overall "augmented" design as prevous saying it "much in sync"
with current architecture and design. Also it is more proper to create a sub jira to discuss
your ideas and put your document there given we already have a very long discussion here on
overall design.

bq. As far as implementation goes, it is recommended to create subtasks as you see fit. Note
that it is easier to review smaller chunks of code. Also, since you guys have implemented
it already, can you comment on how much of the code changes are in frequently updated parts?
If not much, it might make sense to develop on a branch and merge it to trunk.
I would say most parts of YARN-914 are already get committed or patch available already. It
doesn't sounds massive of work for enhancing the timeout tracking/handling here, so a dedicated
develop branch sounds unnecessary to me. However, I would prefer to create a sub jira to discuss
the idea/scope and take a look at your demo code (with removing the duplicated code/feature
that already committed or patch available public) before making any judegement/decision.

[~danzhi], the concrete steps I would suggest for now is:
1. Review all JIRA discussions/design doc/implementations under this umbrella JIRA so far,
and understand the scope and gap with your current internal implementation.
2. Raise a sub jira to put your ideas/design to highlight different options for discussion.
If possible, put a demo patch with removing any similar code or feature on existing patches
for better understanding. We can discuss later on how to bring in your patch contribution.
Make sense?

> (Umbrella) Support graceful decommission of nodemanager
> -------------------------------------------------------
>
>                 Key: YARN-914
>                 URL: https://issues.apache.org/jira/browse/YARN-914
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: graceful
>    Affects Versions: 2.0.4-alpha
>            Reporter: Luke Lu
>            Assignee: Junping Du
>         Attachments: Gracefully Decommission of NodeManager (v1).pdf, Gracefully Decommission
of NodeManager (v2).pdf, GracefullyDecommissionofNodeManagerv3.pdf
>
>
> When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable
to minimize the impact to running applications.
> Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled
on other NMs. Further more, for finished map tasks, if their map output are not fetched by
the reducers of the job, these map tasks will need to be rerun as well.
> We propose to introduce a mechanism to optionally gracefully decommission a node manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message