apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay M Pujare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-711) Support custom SSL keystore for the Stram REST API web service
Date Mon, 08 May 2017 22:15:04 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001657#comment-16001657

Sanjay M Pujare commented on APEXCORE-711:

Recapping my earlier analysis about eliminating dependencies on Hadoop/Yarn:

- In org.apache.hadoop.yarn.webapp.WebApps.Builder<T> the start(WebApp) method does
bulk of the work in creating the WebApp container object. The problematic line in this method
currently is


which does not pass (honor) the config object passed to Builder.with(Configuration). The above
line should be changed to

         WebAppUtils.loadSslConfiguration(builder, conf);

which passes the required config object. Since a class static method (WebAppUtils.loadSslConfiguration)
is being called there is no way to get the new behavior by subclassing a class and overriding
a simple method. All of the code in org.apache.hadoop.yarn.webapp.WebApps.Builder.start(WebApp)
will need to be copied in a new class on the Apex core side. BTW, precisely because of this,
https://issues.apache.org/jira/browse/YARN-4562 was created and fixed (albeit in 2.9.x and
not in any earlier version)

- with this we still have an unresolved issue in the current Yarn. The current code in org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(Builder,
Configuration) is as follows (trimmed):

  public static HttpServer2.Builder loadSslConfiguration(
      HttpServer2.Builder builder, Configuration sslConf) {
    if (sslConf == null) {
      sslConf = new Configuration(false);
    boolean needsClientAuth = YarnConfiguration.YARN_SSL_CLIENT_HTTPS_NEED_AUTH_DEFAULT;

    return ...

which has 2 major issues:
  a) The caller can set SSL properties in the passed sslConf object but they are overridden
by the default ssl-server.xml file because of the sslConf.addResource(...) call. There is
no way to point to a different XML file (because it uses the hardcoded YARN_SSL_SERVER_RESOURCE_DEFAULT
value and not the value of a property) which defeats the very purpose.

  b) the only way to solve (a) above is to mark the value final in the passed sslConf object
in which case the default ssl-server.xml values won't be applied even if they are final there.

Both of these are addressed by the changes in this JIRA.

> Support custom SSL keystore for the Stram REST API web service
> --------------------------------------------------------------
>                 Key: APEXCORE-711
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-711
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Sanjay M Pujare
>            Assignee: Sanjay M Pujare
>   Original Estimate: 72h
>  Remaining Estimate: 72h
> Currently StrAM supports only the default Hadoop SSL configuration for the web-service
because it uses org.apache.hadoop.yarn.webapp.WebApps helper class which has the limitation
of only using the default Hadoop SSL config that is read from Hadoop's ssl-server.xml resource
file. Some users have run into a situation where Hadoops' SSL keystore is not available on
most cluster nodes or the Stram process doesn't have read access to the keystore even when
present. So there is a need for the Stram to use a custom SSL keystore and configuration that
does not suffer from these limitations.
> There is already a PR https://github.com/apache/hadoop/pull/213 to Hadoop to support
this in Hadoop and it is in the process of getting merged soon.
> After that Stram needs to be enhanced (this JIRA) to accept the location of a custom
ssl-server.xml file (supplied by the client via a DAG attribute) and use the values from that
file to set up the config object to be passed to WebApps which will end up using the custom
SSL configuration. This approach has already been verified in a prototype.

This message was sent by Atlassian JIRA

View raw message