hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billie Rinaldi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6335) Port slider's groovy unit tests to yarn native services
Date Tue, 18 Apr 2017 14:55:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972851#comment-15972851

Billie Rinaldi commented on YARN-6335:

bq. Looks like the "yarn.resource.normalization.enabled" will only control whether slider
AM will normalize the request at client side or not. But, regardless of this setting, the
normalization will always be done in YARN scheduler to round up the resource size.

This normalization is specifically for capping the resource request at the maximum value allowed
by YARN. Slider used to automatically lower the resource request when it was too high, but
some users ran into issues because their app could not run with lower resources. They preferred
the app to fail due to requesting resources that were too high, which is why this parameter
was introduced.

bq. What's the difference between these two paths: the path defined in deleteZookeeperNode
and the path defined by registryPathForInstance ?

The first one is the ZK node created for the app's use, separate from the registry nodes that
are used by Slider. In Slider, the DEFAULT_ZK_PATH variable was set for the app here: https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/client/SliderClient.java#L1751-L1775
and here: https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java#L491
but it looks like this code has been removed in YARN native services. We should reintroduce
this functionality.

bq. Even if we check the existence of app Dir at last line of the method, if the create happens
to be done right after this check and before the method returns, it is still the same problem.
To user this just looks as if the create immediately happens after destroy. It is a micro
optimization, but the semantics is still not deterministic.

Okay, I will remove the last check. I am more concerned about the case where something went
wrong on the Slider side, rather than a create/destroy race condition on user side. It is
very frustrating as a user for destroy to succeed and the HDFS directory still to exist, preventing
creation of the app again. But since we are checking the return value of fs.delete, this shouldn't

> Port slider's groovy unit tests to yarn native services
> -------------------------------------------------------
>                 Key: YARN-6335
>                 URL: https://issues.apache.org/jira/browse/YARN-6335
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Billie Rinaldi
>            Assignee: Billie Rinaldi
>             Fix For: yarn-native-services
>         Attachments: YARN-6335-yarn-native-services.001.patch, YARN-6335-yarn-native-services.002.patch,
YARN-6335-yarn-native-services.003.patch, YARN-6335-yarn-native-services.004.patch, YARN-6335-yarn-native-services.005.patch
> Slider has a lot of useful unit tests implemented in groovy. We could convert these to
Java for YARN native services. This scope of this ticket will include unit / minicluster tests
only and will not include Slider's funtests which require a running cluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message