hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
Date Sat, 11 May 2013 00:07:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655049#comment-13655049
] 

Bikas Saha commented on YARN-660:
---------------------------------

Vinod, I have iterated through different combinations of generics including the one you suggest.
I found that the current version is the least ugly wrt user code. Both the test code for the
AMRMClient as well as my use of it in TEZ gave me that impression. If you write user code
that has to always cast/convert back from ContainerRequest to user-defined types you will
see that it looks ugly and I dont want users to keep having to cast these value when they
use specific types. The only user overhead when they use ContainerRequest is to add a <ContainerRequest>
when declaring the member variable and during new. Thereafter everything works. And ContainerRequest
is something that they anyways have to include in the code. Look at the diff of TestAMRMClient
to see the minimal changes needed to make the existing code compile. Ideally if there was
a typedef in Java then even that could be hidden but there isnt.
IMO, ContainerRequest is useful mostly in cases when the user wants a bunch of containers
at *. For most of the other cases where requests are made with more specificity StoredContainerRequest
is more useful. Its provided by the library because it will be commonly needed and also to
support returning matching requests from within the AMRMClient for reasons outlined earlier.
Storing and retrieving matching requests in a meaningful manner cannot be done until we limit
the number of containers in an individual request to 1. StoredContainerRequest provides a
clear type inside the AMRMClient to say what to store and also to limit the container count
per request to 1. e.g. if AMRMClient will save simple ContainerRequest then lets say it saves
the container request in addContainerRequest(ContainerRequest(P1,R1,Count=4)) and then if
user calls removeContainerRequest(ContainerRequest(P1,R1,Count=3)) its hard for AMRMClient
to tell if the stored container request should be removed or not.
Cookies are not required but very useful in reducing user burden. If we think about using
AMRMClient inside the MR app master (or see the impl of TEZ) then we will find that for every
request we need to save a cookie somewhere (eg. Scheduler*LaunchEvent) that will be used when
the request is matched to a container. Either the client can write a map to maintain and store
this relationship or the library provides a helper cookie to keep the info in one place.

Let me know if you have any other comments.
                
> Improve AMRMClient with cookies and matching requests
> -----------------------------------------------------
>
>                 Key: YARN-660
>                 URL: https://issues.apache.org/jira/browse/YARN-660
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.0.5-beta
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>             Fix For: 2.0.5-beta
>
>         Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, YARN-660.4.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message