ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sid Wagle" <swa...@hortonworks.com>
Subject Re: Review Request 26858: Rolling Upgrades - hdfs:///apps/tez/tez.tar.gz needs to be versioned
Date Mon, 20 Oct 2014 23:08:10 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/#review57460
-----------------------------------------------------------

Ship it!


Ship It!

- Sid Wagle


On Oct. 20, 2014, 10:26 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26858/
> -----------------------------------------------------------
> 
> (Updated Oct. 20, 2014, 10:26 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate Cole, Sumit
Mohanty, Srimanth Gunturi, and Sid Wagle.
> 
> 
> Bugs: AMBARI-7825
>     https://issues.apache.org/jira/browse/AMBARI-7825
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> HDP 2.2 using versioned RPMs
> When Ambari installs Hive, it copies any file that matches /usr/hdp/current/tez-client/lib/tez*.tar.gz
file to hdfs:///apps/tez/tez.tar.gz
> (Aside: it also ends up copying other tez-* files that it shouldn't because they are
not needed, and already zipped in the tar.gz file)
> The problem is that during a rolling upgrade, currently running jobs still needs to reference
the same jar files as when they were created.
> Today, Hive's install_jars.py and params.py route /usr/hdp/current/tez-client/lib/tez*.tar.gz
=> tez.tar.gz
> 
> Instead, it should preserve the version number when copying to HDFS.
> Further, when tez.py generates the tez-site.xml file, it should replace the value of
the tez.lib.uris parameter to include the rpm version number in the value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
> 
> It does 2 things:
> 1. Copies files from the local file system to HDFS using a predefined directory structure
> 2. Provides a way of doing dynamic variable interpretation in *-site.xml files
> 
> This will break the Download Client Configs, which calculates the dynamic properties
on the agent-side.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
PRE-CREATION 
>   ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
453c546 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
34f2d96 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
08a0a50 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
5834b89 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
b034cbc 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
78d2ea2 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
fe4324c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 7370dd2

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
6a793bf 
>   ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac 
>   ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 746f346 
>   ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9 
> 
> Diff: https://reviews.apache.org/r/26858/diff/
> 
> 
> Testing
> -------
> 
> Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
> 
> 
> 1. Set properties
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_source
            "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ hdp_stack_version
}}.tar.gz"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev cluster-env tez_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
> /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site tez.lib.uris
                 "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/tez-{{ component_version }}.{{
hdp_stack_version }}.tar.gz"
> 
> 2. Verified properties were saved
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
> http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
> 
> 3. Copy changed files
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
                  /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
> yes | cp /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
 /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
           /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
                /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
                      /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
> yes | cp /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
                   /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
> rm /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
> 
> 4. Check that tarballs are not already in HDFS. If they are, delete them.
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r /hdp/apps/2.2.0.0-913/tez/
> 
> 5. Restart Hive service and verify file is copied to HDFS,
> [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-913/tez/
> 
> 6. Before running Tez script, check that it still has the dynamic property without the
actual values,
> less /etc/tez/conf/tez-site.xml	
> / tez.lib.uris
> 
> 7. Run tez script manually, which generates tez-site.xml during its install
> Need to first find the latest command file.
> ls -ltr  /var/lib/ambari-agent/data/command-*.json
> Get the digits from the last command-*.json file
> python /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py
CONFIGURE /var/lib/ambari-agent/data/command-69.json /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS
/var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
> 
> Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
> <name>tez.lib.uris</name>
> <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
> 
> 8. Ensure Hive runs with tez-execution engine
> [root@c6404 ~]# su - hive
> [hive@c6404 ~]$ hive
> hive> select name FROM school order by name asc limit 10;
> 
> *Unit tests passed*
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message