ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 38788: RU - HDFS_Client restart and hdp-select causes dfs_data_dir_mount.hist to be lost
Date Sun, 27 Sep 2015 03:05:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38788/
-----------------------------------------------------------

(Updated Sept. 27, 2015, 3:05 a.m.)


Review request for Ambari, Dmytro Sen, Jayush Luniya, Mahadev Konar, and Sumit Mohanty.


Changes
-------

Used mv command, and updated defaults.


Bugs: AMBARI-13251
    https://issues.apache.org/jira/browse/AMBARI-13251


Repository: ambari


Description
-------

During the Clients group of RU, restarting hdfs_client changes changes the symlink /usr/hdp/current/hadoop-client/
from the old version to the new version.

Because the Data Dir Mount History file is stored at /etc/hadoop/conf/dfs_data_dir_mount.hist
and
/etc/hadoop/conf symlinks to /usr/hdp/current/hadoop-client/conf 
it means that the file must be copied to the new location.

Multiple clients (HDFS, HBase, Hive, YARN, MapReduce, Pig, Slider, Tez) actually end up calling
{code}
hdp_select.select("hadoop-client", params.version)
{code}
that changes the symlink, so the first one to be ran during RU must copy the file. If DataNode
is present, it means that HDFS Client is guaranteed to be there, and from all of those clients
it is the first one in the Upgrade Pack.

During a Rolling Downgrade, the file should already exist in the previous version, and the
file is only written to when DataNode is restarted, so we are ok.


Diffs (updated)
-----

  ambari-agent/src/main/python/ambari_agent/Controller.py 8746172 
  ambari-agent/src/test/python/resource_management/TestDatanodeHelper.py 70539ac 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog212.java 37a87ab

  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml
4cb2274 
  ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml
c6dd202 
  ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml
b5451d8 
  ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_datanode_unmounted_data_dir.py
4406231 
  ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py d5a42f0 
  ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json f6c9bb4 
  ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json c13e5c9 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.json a0da7f0 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json b644411 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 770e085 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 988f38e 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 4cf647b 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 286a728 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json
b2fd6e8 
  ambari-server/src/test/python/stacks/2.0.6/configs/flume_22.json 4f30caa 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json e7a516f 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 83120eb 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 349a1cf 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json b71c4c8 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json a7187f9 
  ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json a026259 
  ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 01856df 
  ambari-server/src/test/python/stacks/2.0.6/configs/ranger-namenode-start.json 84c798b 
  ambari-server/src/test/python/stacks/2.0.6/configs/secured.json c9e20e2 
  ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json ca0b1a6 
  ambari-server/src/test/python/stacks/2.2/configs/default.json dcd38d6 
  ambari-server/src/test/python/stacks/2.2/configs/default_custom_path_config.json 2c9c918

  ambari-server/src/test/python/stacks/2.2/configs/falcon-upgrade.json a4f9dc9 
  ambari-server/src/test/python/stacks/2.2/configs/hive-upgrade.json 95456f9 
  ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json e06882a

  ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 1f23f11 
  ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json c77b3ec 
  ambari-server/src/test/python/stacks/2.2/configs/oozie-downgrade.json e9018d8 
  ambari-server/src/test/python/stacks/2.2/configs/oozie-upgrade.json fc4df58 
  ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json ad7a75a 
  ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-default.json f012a17 
  ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json f47a247 
  ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json fa4d614 
  ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json 185d4d5 
  ambari-server/src/test/python/stacks/2.3/configs/hbase_default.json f5a4487 
  ambari-server/src/test/python/stacks/2.3/configs/hbase_secure.json d31c621 
  ambari-server/src/test/python/stacks/2.3/configs/spark_default.json 730a81b 
  ambari-server/src/test/resources/custom_actions/ru_execute_tasks_namenode_prepare.json d42d01c

  ambari-web/app/assets/data/configurations/config_versions.json 603d33e 
  ambari-web/app/assets/data/configurations/service_versions.json c5eb46c 
  ambari-web/app/assets/data/stacks/HDP-2.2/configurations.json 0399656 
  ambari-web/app/data/BIGTOP/site_properties.js e71cab4 
  ambari-web/app/data/HDP2/site_properties.js 03ffd31 

Diff: https://reviews.apache.org/r/38788/diff/


Testing
-------

Tested the alert during RU, and the mount history file was copied to the new location, so
no warnings were generated.

Python unit tests passed,
----------------------------------------------------------------------
Total run:805
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message