ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Fernandez" <afernan...@hortonworks.com>
Subject Re: Review Request 26858: Rolling Upgrades - hdfs:///apps/tez/tez.tar.gz needs to be versioned
Date Sat, 18 Oct 2014 01:17:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------

(Updated Oct. 18, 2014, 1:17 a.m.)


Review request for Ambari, Sumit Mohanty and Sid Wagle.


Bugs: AMBARI-7825
    https://issues.apache.org/jira/browse/AMBARI-7825


Repository: ambari


Description
-------

HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz
file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because they are not needed,
and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still needs to reference
the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz
=> tez.tar.gz

Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the value of the tez.lib.uris
parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz

It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files

This will break the Download Client Configs, which calculates the dynamic properties on the
agent-side.


Diffs
-----

  ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
PRE-CREATION 
  ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
453c546 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
34f2d96 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
08a0a50 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
5834b89 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
b034cbc 
  ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py 78d2ea2

  ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py fe4324c

  ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
6a793bf 

Diff: https://reviews.apache.org/r/26858/diff/


Testing
-------

Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,


1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source
            "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version
}}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris
                 "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{
hdp_stack_version }}.tar.gz"

2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site

3. Copy changed files
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
                  /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
 /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
           /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
                /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
                      /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
                   /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py

4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/

5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/

6. Before running Tez script, check that it still has the dynamic property without the actual
values,
less /etc/tez/conf/tez-site.xml	
/ tez.lib.uris

7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr  /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py
CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS
/var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp

Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>

8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;

*Unit tests currently fail*


Thanks,

Alejandro Fernandez


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message