hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4879) Enhance Allocate Protocol to Identify Requests Explicitly
Date Thu, 14 Apr 2016 01:38:25 GMT

     [ https://issues.apache.org/jira/browse/YARN-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Subru Krishnan updated YARN-4879:
    Attachment: SimpleAllocateProtocolProposal-v2.pdf

Thanks [~kasha], [~leftnoteasy], [~vinodkv] and [~stevel@apache.org] for taking a look at
our proposal. PFA updated doc (v2) that incorporates your feedback.

A few additional clarifications (have addressed rest of comments directly in the updated doc):

bq. While making these changes, would it possible to address YARN-314 too? 
bq. I'm okay if we can get two in a shot, but I'd caution against risking this effort by blowing
up the size.

We will address YARN-314 as long as applications specify the Request-ID as they can request
for multiple containers at same priority through independent requests.

bq. Why are we putting priority semantics onto the ID? We should just follow the existing
priority ordering.

We will continue to follow the existing priority ordering. But as explained above, with the
proposed enhancement user can potentially make multiple requests at same priority (YARN-314).
In such a scenario, we will simply allocate containers in FIFO order.

bq. BTW, for the federation related issue, does the client-library need to always generate
these IDs? How does that interact with application generated IDs?

In Federation also, we expect applications to generate the IDs. For e.g.: we will work with
the REEF team (and the long running service AM proposed as part of YARN-4692) to start specifying
IDs for their allocation requests.

bq. I would also like to see if the allocated containers could support a role ID field too...nothing
much, but enough that on an AM restart their role can be determined. That one, I'd keep separate
from the request ID; they serve slightly different purposes. (I could have 5 requests outstanding
for containers of role 4; I'd want to track those requests)

I agree that having an explicit role ID is useful but feel its outside the scope of this JIRA
which IIUC is what you are also observing. I think adding a role ID should be part of YARN-4692/YARN-4902.

> Enhance Allocate Protocol to Identify Requests Explicitly
> ---------------------------------------------------------
>                 Key: YARN-4879
>                 URL: https://issues.apache.org/jira/browse/YARN-4879
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications, resourcemanager
>            Reporter: Subru Krishnan
>            Assignee: Subru Krishnan
>         Attachments: SimpleAllocateProtocolProposal-v1.pdf, SimpleAllocateProtocolProposal-v2.pdf
> For legacy reasons, the current allocate protocol expects expanded requests which represent
the cumulative request for any change in resource constraints. This is not only very difficult
to comprehend but makes it impossible for the scheduler to associate container allocations
to the original requests. This problem is amplified by the fact that the expansion is managed
by the AMRMClient which makes it cumbersome for non-Java clients as they all have to replicate
the non-trivial logic. In this JIRA, we are proposing enhancement to the Allocate Protocol
to allow AMs to identify requests explicitly.  

This message was sent by Atlassian JIRA

View raw message