ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Speidel" <jspei...@hortonworks.com>
Subject Re: Review Request 35607: Filter out Kerberos hostname properties from exported Blueprint
Date Thu, 18 Jun 2015 14:55:32 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35607/#review88379
-----------------------------------------------------------

Ship it!


Ship It!

- John Speidel


On June 18, 2015, 2:49 p.m., Robert Nettleton wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35607/
> -----------------------------------------------------------
> 
> (Updated June 18, 2015, 2:49 p.m.)
> 
> 
> Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas.
> 
> 
> Bugs: AMBARI-12000
>     https://issues.apache.org/jira/browse/AMBARI-12000
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch resolves AMBARI-12000. 
> 
> An export of a Blueprint from a running, Kerberized cluster would include the following
properties with hostnames:
> 
> "admin_server_host" in "kerberos-env"
> "kdc_host" in "kerberos-env"
> "hadoop.proxyuser.yarn.hosts" in "core-site"
> 
> Exported Blueprints should not contain hostnames in general, since this makes the Blueprint
less portable over different cluster configurations. 
> 
> This patch implements the following changes:
> 
> 1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties from an
exported Blueprint.  These hostname properties refer to the KDC server of a Kerberized cluster,
and so are not useful to an exported Blueprint at this time.  Since these properties generally
refer to an external KDC server, it is likely that a new cluster would point to a new KDC
install, which is typically an external server. 
> 2. Adds a property updaters to mask the hostname information in the "hadoop.proxyuser.yarn.hosts"
property in "core-site".  This property includes a comma-separated list of hostnames that
include Yarn ResourceManager instances. While not immediately useful outside of Kerberos,
having this property include the %HOSTGROUP% syntax for hostname substitution is in keeping
with the handling of other "*proxyuser*" properties, and could eventually be helpful in creating
Kerberized clusters in future versions of Ambari.
> 3. Adds a unit test to verify the changes to the handling for these three properties.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
af3b331 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
abf5675 
> 
> Diff: https://reviews.apache.org/r/35607/diff/
> 
> 
> Testing
> -------
> 
> 1. Ran the ambari-server unit test suite.  Results:
> 
> Java Tests:
> 
> "Results :
> 
> Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28"
> 
> Python Tests:
> 
> "Failed tests:
> FAIL: test_recommendTezConfigurations (test_stack_advisor.TestHDP23StackAdvisor)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py",
line 1199, in patched
>     return func(*args, **keywargs)
>   File "/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py",
line 661, in test_recommendTezConfigurations
>     self.assertEquals(configurations, expected)
> AssertionError: {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues':
'q [truncated]... != {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues':
'q [truncated]...
>   {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'queue1,queue2'}},
>    'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc
-XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
>                                'tez.am.resource.memory.mb': '4000',
>                                'tez.runtime.io.sort.mb': '307',
>                                'tez.runtime.unordered.output.buffer.size-mb': '57',
>                                'tez.session.am.dag.submit.timeout.secs': '600',
>                                'tez.task.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc
-XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
>                                'tez.task.resource.memory.mb': '768',
> -                              'tez.tez-ui.history-url.base': 'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ?                                                                     ^^ ^ -- ^ ^^^
> 
> +                              'tez.tez-ui.history-url.base': 'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ?                                                                     ^^ ^^^  ^ ^^
> 
>    'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': '8192',
>                                 'yarn.scheduler.minimum-allocation-mb': '256'}}}
> 
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:1"
> 
> 
> The failure seen here also occurs in trunk without my patch applied, and so is an un-related
failure.  
> 
> 2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari UI, with
my patch applied.  Exported a Blueprint from this running cluster, and verified that these
properties are no longer include in the exported Blueprint, or in the case of the proxyuser
property are properly masked with %HOSTGROUP% tokens.  Also verified that no hostname information
is included in the exported Blueprint for this cluster.
> 
> 
> Thanks,
> 
> Robert Nettleton
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message