hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5325) Stateless ARMRMProxy policies implementation
Date Mon, 10 Oct 2016 19:03:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563165#comment-15563165
] 

Carlo Curino commented on YARN-5325:
------------------------------------

Hi [~subru], the uploaded .09 version of the patch address most of your asks above plus the
discussion we had on having a fallback to "homesubcluster" in case the resolver cannot locate
the subcluster owning a node/rack. We also log whether this happens. In case the homeSubCluster
is not active, we do similarly to what yarn would do and silently ignore the ResourceRequest
(please confirm this is ok for you).

A few minor things were not changed:
 I fixed on AtomicLong that was redundandt, but the one in the HashMap are useful to avoid
recreating objects at every update (or lengthy code). 
TreeMap is simply to have sorted output when printed (useful for debugging and with <20
sub-clusters and no data-structure churn this should never be a per issue). 
The system.outs in tests are useful for debugging the tests. I would leave them there if you
are ok with it. 
Exceptions in tests were not ignored, they had to be handled by the code later on.
Regarding multi invocation, the only case for which it makes sense is for "headroom" (as this
is the only somewhat stateful side of the policy). I added a second invocation in the corresponding
test, to validate.
Regarding {{FederationPoliciesTestUtil::createResourceRequest}} there are few objects to create,
and this would make the code in tests quite redundant I think.

Re thread safeness, the switch to re-using AllocationBookkeeper I think made the code more
fragile, so we should definitely double-check whether the intended use 
is single-threaded or protect the splitResourceRequest() with a lock.

> Stateless ARMRMProxy policies implementation
> --------------------------------------------
>
>                 Key: YARN-5325
>                 URL: https://issues.apache.org/jira/browse/YARN-5325
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>    Affects Versions: YARN-2915
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5325-YARN-2915.05.patch, YARN-5325-YARN-2915.06.patch, YARN-5325-YARN-2915.07.patch,
YARN-5325-YARN-2915.08.patch, YARN-5325-YARN-2915.09.patch, YARN-5325.01.patch, YARN-5325.02.patch,
YARN-5325.03.patch, YARN-5325.04.patch
>
>
> This JIRA tracks policies in the AMRMProxy that decide how to forward ResourceRequests,
without maintaining substantial state across decissions (e.g., broadcast).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message