hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8002) Support NOT_SELF and ALL namespace types for allocation tag
Date Sat, 10 Mar 2018 02:26:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393974#comment-16393974

Wangda Tan commented on YARN-8002:

Thanks [~cheersyang] for the contexts, suggestions from a quick glance:

1) AllocationTags: 
 The naming is a bit confusing, from the name it looks like List<AllocationTag>. However
it is actually for cardinality query. So how about calling it like "AllocationTagsFilter"
or "AllocationTagsQuery"?

2) About ALL evaluator: 
 Is it really necessary (since we already have special handling for cluster level tags). And
some related classes like {{AllocationTagsManager#getAllApplicationIds}}. 
 I think we should be able to eliminate, correct?

3) As I mentioned offline, I'm not sure if NOT_SELF is necessary, unless it is must-to-have,
I don't suggest to add it.

4) About existing aggregation approach: 
 Currently it takes list of application id from the eval result. Which is a O(N) operation
(N = #apps from the definition). And it uses addtional O(N) space as well. (Since we need
to copy map in some cases).

Alternatively, if we can assume:
 - Each app has only 1 app-label.
 - And no NOT_SELF namespace.

We can actually finish all get Cardinality / update Cardinality in O(1) and space space complexity
is also O(1).

(Explanation, for each application update, it will update 6 maps:
 - App_Id node/rack.
 - App_label node/rack.
 - Global node/rack.

Basically, it will be 3 * (#node-scope), node-scope includes node/rack/failure-zone, etc.
which will still be better than O(#apps) * (#node-scope) for every retrive operation.)

And retrieve cardinality for ALL/App_id/App_label will be just O(1).

  public long getNodeCardinalityByOp(NodeId nodeId, ApplicationId applicationId,
      Set<String> tags, LongBinaryOperator op)
  public long getRackCardinalityByOp(String rack, ApplicationId applicationId,
      Set<String> tags, LongBinaryOperator op)
Can be removed, we should keep as less public interfaces as possible for ATM

> Support NOT_SELF and ALL namespace types for allocation tag
> -----------------------------------------------------------
>                 Key: YARN-8002
>                 URL: https://issues.apache.org/jira/browse/YARN-8002
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Major
>         Attachments: YARN-8002.001.patch, YARN-8002.002.patch
> This is a continua task after YARN-7972, YARN-7972 adds support to specify tags with
namespace SELF and APP_ID, like following
>  * self/<tag>
>  * app-id/<appid>/<tag>
> this task is to track the work to support 2 of remaining namespace types *NOT_SELF* &
*ALL* (we'll support app-label later),
>  * not-self/<tag>
>  * all/<tag>
> this will require a bit refactoring in {{AllocationTagsManager}} as it needs to do some
proper aggregation on tags for multiple apps.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message