ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumit Mohanty" <smoha...@hortonworks.com>
Subject Re: Review Request 38788: RU - HDFS_Client restart and hdp-select causes dfs_data_dir_mount.hist to be lost
Date Mon, 28 Sep 2015 06:11:00 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/#review100792
-----------------------------------------------------------

Ship it!


Ship It!

- Sumit Mohanty


On Sept. 28, 2015, 5:29 a.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38788/
> -----------------------------------------------------------
> 
> (Updated Sept. 28, 2015, 5:29 a.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-13251
>     https://issues.apache.org/jira/browse/AMBARI-13251
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/
from the old version to the new version.
> 
> Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist
and
> /etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf 
> it means that the file must be copied to the new location.
> 
> Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end
up calling
> {code}
> hdp_select.select("hadoop-client", params.version)
> {code}
> that changes the symlink, so the first one to be ran during RU must copy the file. If
DataNode is present, it means that HDFS Client is guaranteed to be there, and from all of
those clients it is the first one in the Upgrade Pack.
> 
> During a Rolling Downgrade, the file should already exist in the previous version, and
the file is only written to when DataNode is restarted, so we are ok.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 8746172 
>   ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java
37a87ab 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml
4cb2274 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_datanode_unmounted_data_dir.py
2912406 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py
34ec8cd 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
3160af2 
>   ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml
c6dd202 
>   ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py
fb0a4db 
>   ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml
b5451d8 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py
4406231 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0 
>   ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4 
>   ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411

>   ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085

>   ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e

>   ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b

>   ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728

>   ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json
b2fd6e8 
>   ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9 
>   ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259 
>   ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df

>   ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b

>   ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2 
>   ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6 
>   ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6 
>   ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918

>   ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9 
>   ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9 
>   ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json
e06882a 
>   ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11 
>   ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec 
>   ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8 
>   ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58 
>   ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a

>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17

>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247

>   ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614

>   ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5

>   ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487 
>   ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621 
>   ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b 
>   ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json
d42d01c 
>   ambari-web/app/assets/data/configurations/config_versions.json 603d33e 
>   ambari-web/app/assets/data/configurations/service_versions.json c5eb46c 
>   ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656 
>   ambari-web/app/data/BIGTOP/site_properties.js e71cab4 
>   ambari-web/app/data/HDP2/site_properties.js 03ffd31 
> 
> Diff: https://reviews.apache.org/r/38788/diff/
> 
> 
> Testing
> -------
> 
> Tested the alert during RU, and the mount history file was copied to the new location,
so no warnings were generated.
> 
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:805
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message