Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 9 Jun 2016 20:12:21 +0000 (UTC)
From: "Subru Krishnan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12953708.1458956090000.44084.1465503141435@Atlassian.JIRA>
In-Reply-To: <JIRA.12953708.1458956090000@Atlassian.JIRA>
References: <JIRA.12953708.1458956090000@Atlassian.JIRA> <JIRA.12953708.1458956090168@arcas>
Subject: [jira] [Comment Edited] (YARN-4879) Enhance Allocate Protocol to
 Identify Requests Explicitly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 09 Jun 2016 20:12:23 -0000


    [ https://issues.apache.org/jira/browse/YARN-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323233#comment-15323233 ] 

Subru Krishnan edited comment on YARN-4879 at 6/9/16 8:11 PM:
--------------------------------------------------------------

There was a discussion regarding the implementation details with [~leftnoteasy], [~vinodkv], [~kasha] and [~asuresh]. The crux of the discussion was:
  * The request ID can be used for either identifying a single RR or a group of RRs. The latter is required if we want to encode the notion of relaxLocality - i.e. allocate a container in node N1, if not fallback to rack R1 and finally ANY. I will fix the RR API Javdocs in YARN-4887 accordingly.
  * The existing *AMRMClientImpl* is convoluted and has some vestigial behaviour that results in auto expansion of client RRs due to rack inference. Quite a bit of the complexity in *AMRMClientImpl* results from the matching logic which can be greatly simplified by using the request ID. So the consensus was to leave the existing  *AMRMClientImpl* as-is for backward compatibility and start with a fresh version of  *AMRMClientImpl (v2)* that will have a clean implementation using the request ID. I created YARN-5225 to track the first version of *AMRMClientImpl (v2)*


was (Author: subru):
There was a discussion regarding the implementation details with [~leftnoteasy], [~vinodkv], [~kasha] and [~asuresh]. The crux of the discussion was:
  * The request ID can be used for either identifying a single RR or a group of RRs. The latter is required if we want to encode the notion of relaxLocality - i.e. allocate a container in node N1, if not fallback to rack R1 and finally ANY. I will fix the RR API Javdocs in YARN-4887 accordingly.
  * The existing *AMRMClientImpl* is convoluted and has some vestigial behaviour that results in auto expansion of client RRs due to rack inference. Quite a bit of the complexity in *AMRMClientImpl* results from the matching logic which can be greatly simplified by using the request ID. So the consensus was to leave the existing  *AMRMClientImpl* as-is for backward compatibility and start with a fresh version of  *AMRMClientImpl (v2)* that will have a clean implementation using the request ID.

> Enhance Allocate Protocol to Identify Requests Explicitly
> ---------------------------------------------------------
>
>                 Key: YARN-4879
>                 URL: https://issues.apache.org/jira/browse/YARN-4879
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications, resourcemanager
>            Reporter: Subru Krishnan
>            Assignee: Subru Krishnan
>         Attachments: SimpleAllocateProtocolProposal-v1.pdf, SimpleAllocateProtocolProposal-v2.pdf
>
>
> For legacy reasons, the current allocate protocol expects expanded requests which represent the cumulative request for any change in resource constraints. This is not only very difficult to comprehend but makes it impossible for the scheduler to associate container allocations to the original requests. This problem is amplified by the fact that the expansion is managed by the AMRMClient which makes it cumbersome for non-Java clients as they all have to replicate the non-trivial logic. In this JIRA, we are proposing enhancement to the Allocate Protocol to allow AMs to identify requests explicitly.  


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org