hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7202) End-to-end UT for api-server
Date Mon, 09 Oct 2017 06:30:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196544#comment-16196544

Eric Yang commented on YARN-7202:

[~jianhe] ServiceClient code is partially completed in YARN-7202.  "updateService" code is
not quite testable because service operation and config change sequence are inverse of what
they are supposed to be.  

Prior to this patch, the PUT method is:

If Service object should be in STARTED state, then return.
If Service object should be in STOPPED state, then return.
If Service object should increase or decrease containers, perform the operation.

The problem arises, if someone retrieve Service status object via GET /ws/v1/services/{service_name},
and change number of container for the service.  It will not perform the container changes.
 The code got to this point because there is no separation between configuration, and status.
 If status object is reused to perform configuration changes, then the code doesn't perform
what it was designed to do.  For clarity, flex number of container operation should be considered
as configuration changes because even existing containers may need to have information of
newly spawning containers.  Hence, it is best that we treat flexing container numbers as configuration
change, and restart all containers with in the service.  

Giving the above reasons, the proper design should be:

If Service config changed, save configuration.
If Service should increase/decrease containers, perform flex operation.
If Service should be in STARTED state, then start containers.
If Service should be in STOPPED state, then stop containers.

At this point, ServiceClient is not written to support the proper sequence of operation to
perform.  I only made changes to ApiServer updateService method to contain the scope of required
changes.  Therefore, I use mock object here to allow TestApiService to pass without making
a failure test case to block other development.  ServiceClient will not pass the test code
at this point, but ApiServer code can pass the test with improved logic.  This is the reason
that mock object is used here.  In YARN-7215, after spec and status are separated two calls.
 I added additional logic in ServiceClient code to have ability to make service config changes.
 In YARN-7215, you can see that TestApiService is switched to the real ServiceClient, and
both positive and negative test cases passed.

The dependency are required to import MiniCluster.  For some reason, I think the code refactoring
has some issues.  Hadoop-yarn-service-api is not a sibling project to hadoop-yarn-services,
or hadoop-yarn-services-core.  Therefore, it doesn't inherit MiniCluster imports.  This is
the reason that test jar files are imported again.

I think YARN-7202, YARN-7216, and YARN-7215 should be one transaction.  Giving degree of complexity
in the design bug, it is good that we discuss the reasons for these changes.  The issues are
raised out of order due to late discovery of the bugs in the inherited code.  Hence, the course
correction is only made in later JIRAs.

Hope this clarifies that the end goal of this JIRA is met, but the proper code will only happen
after YARN-7216 and YARN-7215 are committed.

> End-to-end UT for api-server
> ----------------------------
>                 Key: YARN-7202
>                 URL: https://issues.apache.org/jira/browse/YARN-7202
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Eric Yang
>         Attachments: YARN-7202.yarn-native-services.001.patch, YARN-7202.yarn-native-services.002.patch,
YARN-7202.yarn-native-services.003.patch, YARN-7202.yarn-native-services.004.patch, YARN-7202.yarn-native-services.005.patch,
YARN-7202.yarn-native-services.006.patch, YARN-7202.yarn-native-services.007.patch, YARN-7202.yarn-native-services.008.patch

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message