Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 18 Apr 2017 14:55:41 +0000 (UTC)
From: "Billie Rinaldi (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13056046.1489519040000.306675.1492527341647@Atlassian.JIRA>
In-Reply-To: <JIRA.13056046.1489519040000@Atlassian.JIRA>
References: <JIRA.13056046.1489519040000@Atlassian.JIRA> <JIRA.13056046.1489519040934@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6335) Port slider's groovy unit tests to
 yarn native services
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 18 Apr 2017 14:55:46 -0000


    [ https://issues.apache.org/jira/browse/YARN-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972851#comment-15972851 ] 

Billie Rinaldi commented on YARN-6335:
--------------------------------------

bq. Looks like the "yarn.resource.normalization.enabled" will only control whether slider AM will normalize the request at client side or not. But, regardless of this setting, the normalization will always be done in YARN scheduler to round up the resource size.

This normalization is specifically for capping the resource request at the maximum value allowed by YARN. Slider used to automatically lower the resource request when it was too high, but some users ran into issues because their app could not run with lower resources. They preferred the app to fail due to requesting resources that were too high, which is why this parameter was introduced.

bq. What's the difference between these two paths: the path defined in deleteZookeeperNode and the path defined by registryPathForInstance ?

The first one is the ZK node created for the app's use, separate from the registry nodes that are used by Slider. In Slider, the DEFAULT_ZK_PATH variable was set for the app here: https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/client/SliderClient.java#L1751-L1775
and here: https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/core/build/InstanceBuilder.java#L491
but it looks like this code has been removed in YARN native services. We should reintroduce this functionality.

bq. Even if we check the existence of app Dir at last line of the method, if the create happens to be done right after this check and before the method returns, it is still the same problem. To user this just looks as if the create immediately happens after destroy. It is a micro optimization, but the semantics is still not deterministic.

Okay, I will remove the last check. I am more concerned about the case where something went wrong on the Slider side, rather than a create/destroy race condition on user side. It is very frustrating as a user for destroy to succeed and the HDFS directory still to exist, preventing creation of the app again. But since we are checking the return value of fs.delete, this shouldn't happen.

> Port slider's groovy unit tests to yarn native services
> -------------------------------------------------------
>
>                 Key: YARN-6335
>                 URL: https://issues.apache.org/jira/browse/YARN-6335
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Billie Rinaldi
>            Assignee: Billie Rinaldi
>             Fix For: yarn-native-services
>
>         Attachments: YARN-6335-yarn-native-services.001.patch, YARN-6335-yarn-native-services.002.patch, YARN-6335-yarn-native-services.003.patch, YARN-6335-yarn-native-services.004.patch, YARN-6335-yarn-native-services.005.patch
>
>
> Slider has a lot of useful unit tests implemented in groovy. We could convert these to Java for YARN native services. This scope of this ticket will include unit / minicluster tests only and will not include Slider's funtests which require a running cluster.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org