hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6612) Update fair scheduler policies to be aware of resource types
Date Thu, 27 Jul 2017 20:34:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103856#comment-16103856
] 

Yufei Gu commented on YARN-6612:
--------------------------------

Some major thoughts:
# I prefer to put functionality of ResourceWeights sort(e.g., {{getResourcesSortedByWeight}})
in class {{DominantResourceFairnessComparator}}, that case we could exam the performance of
{{compare()}} more easily. That is due to when I was thinking about which data structure is
better for {{ResourceWeights#resources}}, ArrayList or TreeSet. They seems no difference in
{{compare()}}.
# Another performance improvement would be calculating fair share only if necessary, which
is similar to what YARN-6307 does. I guess we can do similar refactor in this jira.
# I'm not sure we handle minShare usage well, e.g. what if multiple resource usages are under
their minShare, but we probably think more about it and somehow solve it in followup JIRAs.

Minor ones:
# Why remove the {{FileNotFoundException fe}} in LOG.info() in Class {{ResourceUtils}}?
# {float weight = 0.0f;} in {{ResourceWeights}} initialization is not necessary.
# We can't use ResourceType here, {{for (int i = 0; i < ResourceType.values().length; i++)}}
in {{ResourceWeights#toString}}. Moreover, the enum {{ResourceType}} and related code should
be removed.
# Several unused imports
# I like the refactor of {{ResourceNotFoundException}}. Do we need this method {{ResourceNotFoundException(String
resource, Throwable cause)}} since it is never used?
# Method {{getResourceInformation()}} throw an unchecked exception, which isn't necessary
to go into the method signature. This applies to {{getResourceValue()}} and several other
methods in this class. Some of them you didn't touch. Maybe you follow the convention, but
why do we do that at the first place? Besides that, {{ResourcePBImpl#getResourceInformation}}
doesn't throw {{ResourceNotFoundException}} in method signature, but {{ResourcePBImpl#setResourceValue}}
does. I think we need to solve this inconsistency.
# I like new message in class {{ResourceNotFoundException}}, can we also provide possible
reasons of RM inconsistent state?

> Update fair scheduler policies to be aware of resource types
> ------------------------------------------------------------
>
>                 Key: YARN-6612
>                 URL: https://issues.apache.org/jira/browse/YARN-6612
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>    Affects Versions: YARN-3926
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>         Attachments: YARN-6612.YARN-3926.001.patch, YARN-6612.YARN-3926.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message