hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7757) Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers
Date Fri, 02 Feb 2018 08:51:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349983#comment-16349983
] 

Weiwei Yang edited comment on YARN-7757 at 2/2/18 8:50 AM:
-----------------------------------------------------------

Per offline discussion with [~Naganarasimha], uploaded v6 patch majorly introduced another
abstract class layer {{NodeLabelsProvider}} and {{NodeAttributesProvider}}, to avoid a potential
typing mis-match while initializing by reflection, hierarchy see  [^nodeLabelsProvider_refactor_v3.pdf]
 . Other improvements [~Naganarasimha] mentioned we agree to postpone to individual jiras
so we can get this blocker done first. Some details for reference:

bq. It looks like it can create only one of the provider, either for labels or for attributes.
I think we need to explicitly support for both.

This will be done in YARN-7871

bq. multi scripts for different types of attributes

Our configuration doesn't allow to configure multiple scripts now, it will fail on script
verification. Right now we do not see a need to support this, but we can revisit if necessary.
We will also make sure this is documented properly in YARN-7865.

bq. Comments over NodeManager and NodeStatusUpdate

Addressed in v6 patch.

bq. verifyConfiguredScript seems to be out of place

Right now the verifyConfiguredScript is only used by scripted based providers, lets keep it
for now. If further we see it can be reused in some place else, we can pull it out.

bq. serviceStart needs to capture that taskInterval needs to be set before the service is
started

It is initiated with -1 value, and gets override by particular provider. 

bq.  Lets use scheduledexecutorservice instead of timer task ...

We have agreed on this, but since this is not a work of refactoring, we agreed to open another
lower priority JIRA to track.

bq. output format of ScriptBasedNodeAttributesProvider

This will need to be taken care of by YARN-7871 once we decided the finalized format of the
attributes and conventions. This also depends on YARN-7856.

Hope this addresses everything so far.
Thanks.


was (Author: cheersyang):
Per offline discussion with [~Naganarasimha], uploaded v6 patch majorly introduced another
abstract class layer {{NodeLabelsProvider}} and {{NodeAttributesProvider}}, to avoid a potential
typing mis-match while initializing by reflection. Other improvements [~Naganarasimha] mentioned
we agree to postpone to individual jiras so we can get this blocker done first. Some details
for reference:

bq. It looks like it can create only one of the provider, either for labels or for attributes.
I think we need to explicitly support for both.

This will be done in YARN-7871

bq. multi scripts for different types of attributes

Our configuration doesn't allow to configure multiple scripts now, it will fail on script
verification. Right now we do not see a need to support this, but we can revisit if necessary.
We will also make sure this is documented properly in YARN-7865.

bq. Comments over NodeManager and NodeStatusUpdate

Addressed in v6 patch.

bq. verifyConfiguredScript seems to be out of place

Right now the verifyConfiguredScript is only used by scripted based providers, lets keep it
for now. If further we see it can be reused in some place else, we can pull it out.

bq. serviceStart needs to capture that taskInterval needs to be set before the service is
started

It is initiated with -1 value, and gets override by particular provider. 

bq.  Lets use scheduledexecutorservice instead of timer task ...

We have agreed on this, but since this is not a work of refactoring, we agreed to open another
lower priority JIRA to track.

bq. output format of ScriptBasedNodeAttributesProvider

This will need to be taken care of by YARN-7871 once we decided the finalized format of the
attributes and conventions. This also depends on YARN-7856.

Hope this addresses everything so far.
Thanks.

> Refactor NodeLabelsProvider to be more generic and reusable for node attributes providers
> -----------------------------------------------------------------------------------------
>
>                 Key: YARN-7757
>                 URL: https://issues.apache.org/jira/browse/YARN-7757
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Blocker
>         Attachments: YARN-7757-YARN-3409.001.patch, YARN-7757-YARN-3409.002.patch, YARN-7757-YARN-3409.003.patch,
YARN-7757-YARN-3409.004.patch, YARN-7757-YARN-3409.005.patch, YARN-7757-YARN-3409.006.patch,
nodeLabelsProvider_refactor_class_hierarchy.pdf, nodeLabelsProvider_refactor_v2.pdf, nodeLabelsProvider_refactor_v3.pdf
>
>
> Propose to do refactor on {{NodeLabelsProvider}}, {{AbstractNodeLabelsProvider}} to be
more generic, so node attributes providers can reuse these interface/abstract classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message