hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-103) Add a yarn AM - RM client module
Date Tue, 09 Oct 2012 23:34:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472863#comment-13472863
] 

Siddharth Seth commented on YARN-103:
-------------------------------------

bq. This client is the barebones implementation and hence has no smarts. The checks can be
implemented/may not be necessary in smarter versions of the client. Additionally, like you
say, some information may be better exposed at different API's.
Why have an additional smarter client on top of this one in YARN ? Individual apps may choose
to have their own - MR for RPC specific error handling, etc. We could even considering getting
rid of getNumClusterNodes / getAvailableResources since they're available via AllocateResponse.
Would like others to take a look at the entire API specification before committing this.
bq. I actually want to test against the MiniMRCluster instead of an expected behavior via
a mockRM. I want the client test to fail if the RM behavior changes. So yes this is more of
a functional test than a unit test.
RM behaviour depends upon the underlying scheduler. For a simple case like this, the behaviour
is likely to be the same across schedulers - but the test runs against the default - which
is the CS right now. We don't differentiate between functional / unit tests at the moment
- which makes the test runtimes reasonably high. (MiniClusetr tests take several seconds versus
fractions of a second for mocks).
In the current functional test - there's a sleep-poll cycle which could get stuck.
bq. Per your comments in MAPREDUCE-4671, I have attached another version of the 4th patch
that uses a Wrapper class for ResourceRequest instead of not deleting objects. You can compare
both and see which one seems more palatable. Based on that MAPREDUCE-4671 can be updated if
needed.
A custom comparator may be sufficient - o.a.h.y.s.r.Application already uses this. Alternately,
using the wrapper only for the 'ask' table may be simpler ?

Looks like the patches accidentally include some hdfs changes.
                
> Add a yarn AM - RM client module
> --------------------------------
>
>                 Key: YARN-103
>                 URL: https://issues.apache.org/jira/browse/YARN-103
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch, YARN-103.4.patch,
YARN-103.4.wrapper.patch
>
>
> Add a basic client wrapper library to the AM RM protocol in order to prevent proliferation
of code being duplicated everywhere. Provide helper functions to perform reverse mapping of
container requests to RM allocation resource request table format.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message