ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sid Wagle" <swa...@hortonworks.com>
Subject Re: Review Request 41660: AMBARI-14445 Switch to cluster Zookeeper for AMS in distributed mode deployment
Date Tue, 22 Dec 2015 22:30:47 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41660/#review111672
-----------------------------------------------------------



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line
59)
<https://reviews.apache.org/r/41660/#comment171886>

    We should use the zookeeper quorum property asis from ams-hbase-site.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line
205)
<https://reviews.apache.org/r/41660/#comment171887>

    This should not be required.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java (line
241)
<https://reviews.apache.org/r/41660/#comment171891>

    Lets check existing value and if it is defualt "/hbase", change it unconditionally to
/ams-hbase-(security-classfier).
    
    However, this needs check of Security State.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
(line 530)
<https://reviews.apache.org/r/41660/#comment171892>

    Delete.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py
(line 61)
<https://reviews.apache.org/r/41660/#comment171894>

    We should do the same for hdfs-site.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
(line 150)
<https://reviews.apache.org/r/41660/#comment171895>

    Change.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 459)
<https://reviews.apache.org/r/41660/#comment171896>

    Two calls registered for "ams-site"



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 476)
<https://reviews.apache.org/r/41660/#comment171898>

    Change.


- Sid Wagle


On Dec. 22, 2015, 9:59 p.m., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41660/
> -----------------------------------------------------------
> 
> (Updated Dec. 22, 2015, 9:59 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-14445
>     https://issues.apache.org/jira/browse/AMBARI-14445
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch involves changing of AMS in distributed mode to use cluster zookeeper instead
of local zookeeper.
> 
> Summary of Changes in different areas.
> 
> AMS service side
> 1. Added new config amshbase.zookeeper.quorum in ams-site
> 2. Moved the zookeeper.znode.parent config item from ams-hbase-security-site to ams-hbase-site
> 3. Changed the default value of zookeeper.znode.parent in configs and Java code to /ams-hbase-unsecure
to make sure there is no conflict with HBase service.
> 4. Remove -Dzookeeper.sasl.client.username={{zk_servicename}} in ams-env and ams-hbase-env
content.
> 
> Stack Advisor
> 1. Recommend cluster zookeeper quorum to amshbase.zookeeper.quorum in ams-site.
> 2. Recommend and validate correct ports for hbase.zookeeper.property.clientPort (cluster
zookeeper port for distributed and default 61181 for embedded)
> 3. Minor refactoring of util method in stack_advisor to make sure we have 1 method each
for getZookeeperHost and getZookeeperHostsWithPort.
> 
> AMS start up scripts
> 1. Do not start hbase zookeeper in distributed mode.
> 2. hbase_zookeeper_quorum should resolve to cluster zookeeper quorum and localhost in
distributed and embedded modes respectively.
> 3. If embedded mode, delete core-site.xml from collector and ams-hbase conf directories
to make sure embedded mode works in secure cluster.
> 
> UI
> 1. Add amshbase.zookeeper.quorum in ams-site to the list of "watchers" in the config
initialization to make sure cluster zookeeper quorum is captured in the config.
> 2. Add the same config in the UI warning messages when a Zookeeper server is being added
/ deleted.
> 
> Upgrade
> 1. Move zookeeper.znode.parent from ams-hbase-security-site to ams-hbase-site.
> 2. Add amshbase.zookeeper.quorum in ams-site.
> 3. Update ams-env and ams-hbase-env to remove -Dzookeeper.sasl.client.username={{zk_servicename}}
> 4. Fix minor bug in UpgradeCatalog220.
> 
> Security
> 1. Change in kerberos.json to move zookeeper.znode.parent config item from ams-hbase-security-site
to ams-hbase-site.
> 2. Similar changes in ranger upgrade JSON files.
> 
> Others
> 1. Remove AMS alert that checks zookeeper state.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultPhoenixDataSource.java
562049b 
>   ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java
e1d256d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java
85b142d 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog221.java
a27a2b2 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/alerts.json 4015590

>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml
96e2bb3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml
a061006 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-security-site.xml
5e7bc518 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml
e97cfee 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
beb0fa9 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
dac60f3 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams_service.py
0726802 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py
556380e 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
87fd476 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py a723f75

>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog220Test.java
896011a 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog221Test.java
7cf386e 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py
ab4d006 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 1ee3e7a 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 61a7cdc

>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json bba7542

>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_1_3.json 3b4dff4

>   ambari-web/app/assets/data/stacks/HDP-2.1/service_components.json ca9ac3c 
>   ambari-web/app/messages.js c0e1e16 
>   ambari-web/app/utils/configs/add_component_config_initializer.js 6fc505c 
> 
> Diff: https://reviews.apache.org/r/41660/diff/
> 
> 
> Testing
> -------
> 
> Manual Testing on embedded & distributed mode in secure & unsecure environments.
> 
> Ambari server and agent python unit tests pass.
> 
> Java Unit tests passed before last set of changes. Currently being run again.
> 
> Patch submitted through apache.
> 
> Added unit tests for config changes to UpgradeCatalog221.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message