hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billie Rinaldi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5701) Fix issues in yarn native services apps-of-apps
Date Fri, 14 Oct 2016 18:04:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576040#comment-15576040
] 

Billie Rinaldi commented on YARN-5701:
--------------------------------------

Here are details on the bugs I am attempting to fix in this patch.

The changes in SliderClient are to use System.console() instead of a BufferedReader to prompt
the user for passwords when the app requests passwords in the appConfig. When using the old
slider python script that launched the SliderClient, System.console did not work, so we used
BufferedReader and masked the passwords in the slider script. When running with the yarn script
instead of slider, we need to use System.console() (as is done in CredentialShell) because
the yarn script is not masking the passwords.

Another issue that came up when testing an assembly app that requests passwords is that the
assembly is not inheriting the credentials request of the child / external apps. This is fixed
in the InstanceBuilder.java changes.

In ProviderUtils.filterSiteOptions, the return map is being used as substitution tokens, so
the keys should be in the format $\{key\}. This is fixing a problem where tokens are not being
substituted properly.

In ProviderUtils.createConfigFile and ProviderUtils.localizeConfigFile, there is a concurrency
problem in the creation of config files that occurs when multiple containers of the same type
are being launched at once. This part of the patch fixes those.

The new methods in ProviderUtils, getHostNamesList and getIPsList, are used to populate substitution
tokens COMPONENTNAME_HOSTNAME and COMPONENTNAME_IP. These are needed in addition to COMPONENTNAME_HOST
token which provides the name of the host where a container is running. In the docker container
case, the HOSTNAME substitution will be the docker hostname of the container (we may want
to change this later to be the DNS hostname in the case when RegistryDNS is enabled). This
patch also adds handling for the nested assembly case when role.prefix is populated.

DockerProviderService line 374 fixes a typo; it should have been using replaceAll instead
of replace.

DockerProviderService.buildMonitorDetails and DockerProviderService.buildRoleHostDetails are
needed to populate information on the AM UI page. These methods are copied from the original
AgentProviderService.

The .keep file needs to be added to make the Slider AM web service work. It is hard to add
this file. You have to run "git add -f hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/resources/webapps/slideram/.keep"
before doing the git commit.

Lastly, clusterName has been added as a parameter to a couple of methods where it was missing.
CLUSTER_NAME was not being substituted properly in exports because the parameter was not being
passed.

> Fix issues in yarn native services apps-of-apps
> -----------------------------------------------
>
>                 Key: YARN-5701
>                 URL: https://issues.apache.org/jira/browse/YARN-5701
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Billie Rinaldi
>            Assignee: Billie Rinaldi
>         Attachments: YARN-5701-yarn-native-services-001.patch, YARN-5701-yarn-native-services.001.patch
>
>
> In testing out complex apps-of-apps with the docker provider, I found a few issues primarily
related to exports, as well as a couple of issues in config generation and CredentialProvider
handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message