ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Review Request 41542: Get rid of wildcards in package names.
Date Fri, 18 Dec 2015 10:32:56 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41542/
-----------------------------------------------------------

Review request for Ambari and Myroslav Papirkovskyy.


Bugs: AMBARI-14430
    https://issues.apache.org/jira/browse/AMBARI-14430


Repository: ambari


Description
-------

**Problem**:  
Wildcards in packages bring and bunch of problems connected with RU and normal
installs.  
a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get
installed, but hadoop_2_2_0_1234 fails, we can no longer install master
package.  
b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get
installed before hive_2_2_*, we get problems with inability to install. I've
bumped into a couple of different situations like this. (storm_2_2* and
storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3
are real bugs which we had to hack out  
c) We install a lot of packages which are not needed by default like hadoop-
2_2_...fuse, just by asking hadoop_2_2_*.  
d) Situtations like 47931 can happen in RU.  
e) The bug in RU where we add a service after install_packages was triggered.

**Proposal:**  
1\. Replace * by ${hdp_version} in metainfo.xml.  
E.g.:

    
    
            <osSpecific>
              <osFamily>redhat7,redhat6,suse11</osFamily>
              <packages>
                <package>
                  <name>hbase_2_3_*</name>
                </package>
                <package>
                  <name>phoenix_2_3_*</name>
                </package>
              </packages>
            </osSpecific>
    

->
    
    
            <osSpecific>
              <osFamily>redhat7,redhat6,suse11</osFamily>
              <packages>
                <package>
                  <name>hbase_${hdp_version}</name>
                </package>
                <package>
                  <name>hbase_${hdp_version}-regionserver</name>
                </package>
                <package>
                  <name>phoenix_${hdp_version}</name>
                </package>
              </packages>
            </osSpecific>
    

Note: this step will also let us not override packages in metainfo.xml with
every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*.

2\. During INSTALL action (script.py) or during install packages
(install_packages.py) get the version we are currently using from hdp-select.

e.g. for centos:

    
    
    [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}'
    2.3.4.0
    [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}'
    3204 # a bit slow for every INSTALL task
    

Another better aproach is to install hdp-select and than use rpm/dpkg, which
is faster than yum/zypper/apt-get, considering we will do this on every
INSTALL task this is important.

    
    
    [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g'
# faster approach
    2.3.4.0-3204
    
    
    
    u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' 
    2.3.4.0-3475
    

Vs other approaches:  
I think it's the only possible approach to get the version here, sending
version which is specified by user during creating of a version. Seems to be a
bad idea as the user can specify wrong version. Also we will need to use this
during fresh install, where we don't have it specified at all.

3\. Replace ${hdp_version} with the version we got in step 2.

I also discussed this with dlysnichenko this seems to be feasible in terms of
RU/EU.  
The overall idea was discussed in 47931  
cc smohanty


Diffs
-----

  ambari-common/src/main/python/resource_management/core/providers/package/apt.py 70a553d

  ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py 87938ad

  ambari-common/src/main/python/resource_management/core/providers/package/zypper.py 1fbc9bf

  ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py
6dbcef5 
  ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
7f34cc1 
  ambari-common/src/main/python/resource_management/libraries/script/script.py c954a15 
  ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py
0d9687b 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py d39ccbf 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml fe47276 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 4fa4ff7 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 71297b1 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 0366758 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml d59b89e 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml aa164e3 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 481fca9 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml db7e025 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml 5472f43 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml 0c457f7 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 47bb207 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml 5cb598e 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml e7ff122 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml ccf765b 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml b0ccddd 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml 189defc 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml d0b37cf

  ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml b19aac1 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml c3a3c81 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml 6fdad8b 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml 5031f85 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml a37378c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml d103caf 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml b6b6bb8 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml 330b542 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml d1172b1 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 2fe0ed0 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 8ba2426 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml bc52e85 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml b1a5584

  ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml 8287b50 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 2cf290b 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml 1290cde 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml 00af620 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml 7720a3d 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml 088bb16 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml c2fe742

  ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml 9e82a82 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml 158ef8c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml 183d55c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml b7cd3d1 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml a1c2bd0 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml ce3fb95 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml 5ad52b6 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml 90bc472 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml e465d42 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml 2f09e88 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml 9ae1f42 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml d76ad5c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml a27bf31

  ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml ff998f0 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml f4880e2 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml 721cc33 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml 36b01e4 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml 356225e 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml c61a270 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml 4bf4198

  ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 
  ambari-server/src/test/python/custom_actions/configs/install_packages_config.json 50e327a

  ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2 

Diff: https://reviews.apache.org/r/41542/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message