hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-4879) Enhance Allocate Protocol to Identify Requests Explicitly
Date Thu, 09 Jun 2016 20:12:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323233#comment-15323233
] 

Subru Krishnan edited comment on YARN-4879 at 6/9/16 8:11 PM:
--------------------------------------------------------------

There was a discussion regarding the implementation details with [~leftnoteasy], [~vinodkv],
[~kasha] and [~asuresh]. The crux of the discussion was:
  * The request ID can be used for either identifying a single RR or a group of RRs. The latter
is required if we want to encode the notion of relaxLocality - i.e. allocate a container in
node N1, if not fallback to rack R1 and finally ANY. I will fix the RR API Javdocs in YARN-4887
accordingly.
  * The existing *AMRMClientImpl* is convoluted and has some vestigial behaviour that results
in auto expansion of client RRs due to rack inference. Quite a bit of the complexity in *AMRMClientImpl*
results from the matching logic which can be greatly simplified by using the request ID. So
the consensus was to leave the existing  *AMRMClientImpl* as-is for backward compatibility
and start with a fresh version of  *AMRMClientImpl (v2)* that will have a clean implementation
using the request ID. I created YARN-5225 to track the first version of *AMRMClientImpl (v2)*


was (Author: subru):
There was a discussion regarding the implementation details with [~leftnoteasy], [~vinodkv],
[~kasha] and [~asuresh]. The crux of the discussion was:
  * The request ID can be used for either identifying a single RR or a group of RRs. The latter
is required if we want to encode the notion of relaxLocality - i.e. allocate a container in
node N1, if not fallback to rack R1 and finally ANY. I will fix the RR API Javdocs in YARN-4887
accordingly.
  * The existing *AMRMClientImpl* is convoluted and has some vestigial behaviour that results
in auto expansion of client RRs due to rack inference. Quite a bit of the complexity in *AMRMClientImpl*
results from the matching logic which can be greatly simplified by using the request ID. So
the consensus was to leave the existing  *AMRMClientImpl* as-is for backward compatibility
and start with a fresh version of  *AMRMClientImpl (v2)* that will have a clean implementation
using the request ID.

> Enhance Allocate Protocol to Identify Requests Explicitly
> ---------------------------------------------------------
>
>                 Key: YARN-4879
>                 URL: https://issues.apache.org/jira/browse/YARN-4879
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications, resourcemanager
>            Reporter: Subru Krishnan
>            Assignee: Subru Krishnan
>         Attachments: SimpleAllocateProtocolProposal-v1.pdf, SimpleAllocateProtocolProposal-v2.pdf
>
>
> For legacy reasons, the current allocate protocol expects expanded requests which represent
the cumulative request for any change in resource constraints. This is not only very difficult
to comprehend but makes it impossible for the scheduler to associate container allocations
to the original requests. This problem is amplified by the fact that the expansion is managed
by the AMRMClient which makes it cumbersome for non-Java clients as they all have to replicate
the non-trivial logic. In this JIRA, we are proposing enhancement to the Allocate Protocol
to allow AMs to identify requests explicitly.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message