ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Re: Review Request 41542: Get rid of wildcards in package names.
Date Fri, 18 Dec 2015 16:38:54 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41542/
-----------------------------------------------------------

(Updated Dec. 18, 2015, 4:38 p.m.)


Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav Papirkovskyy, Nate Cole,
and Sumit Mohanty.


Bugs: AMBARI-14430
    https://issues.apache.org/jira/browse/AMBARI-14430


Repository: ambari


Description
-------

**Problem**:  
Wildcards in packages bring and bunch of problems connected with RU and normal
installs.  
a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get
installed, but hadoop_2_2_0_1234 fails, we can no longer install master
package.  
b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get
installed before hive_2_2_*, we get problems with inability to install. I've
bumped into a couple of different situations like this. (storm_2_2* and
storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3
are real bugs which we had to hack out  
c) We install a lot of packages which are not needed by default like hadoop-
2_2_...fuse, just by asking hadoop_2_2_*.  
d) Situtations like 47931 can happen in RU.  
e) The bug in RU where we add a service after install_packages was triggered.

**Proposal:**  
1\. Replace * by ${hdp_version} in metainfo.xml.  
E.g.:

    
    
            <osSpecific>
              <osFamily>redhat7,redhat6,suse11</osFamily>
              <packages>
                <package>
                  <name>hbase_2_3_*</name>
                </package>
                <package>
                  <name>phoenix_2_3_*</name>
                </package>
              </packages>
            </osSpecific>
    

->
    
    
            <osSpecific>
              <osFamily>redhat7,redhat6,suse11</osFamily>
              <packages>
                <package>
                  <name>hbase_${hdp_version}</name>
                </package>
                <package>
                  <name>hbase_${hdp_version}-regionserver</name>
                </package>
                <package>
                  <name>phoenix_${hdp_version}</name>
                </package>
              </packages>
            </osSpecific>
    

Note: this step will also let us not override packages in metainfo.xml with
every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*.

2\. During INSTALL action (script.py) or during install packages
(install_packages.py) get the version we are currently using from hdp-select.

e.g. for centos:

    
    
    [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}'
    2.3.4.0
    [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}'
    3204 # a bit slow for every INSTALL task
    

Another better aproach is to install hdp-select and than use rpm/dpkg, which
is faster than yum/zypper/apt-get, considering we will do this on every
INSTALL task this is important.

    
    
    [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g'
# faster approach
    2.3.4.0-3204
    
    
    
    u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' 
    2.3.4.0-3475
    

Vs other approaches:  
I think it's the only possible approach to get the version here, sending
version which is specified by user during creating of a version. Seems to be a
bad idea as the user can specify wrong version. Also we will need to use this
during fresh install, where we don't have it specified at all.

3\. Replace ${hdp_version} with the version we got in step 2.

I also discussed this with dlysnichenko this seems to be feasible in terms of
RU/EU.  
The overall idea was discussed in 47931  
cc smohanty


Also after talking to dlysnichenko he said we were excluding repos because of issues with
wildcard on suse when it is not excluded and mutptiple packages is found for a wildcard. That's
why when we stop using wildcards we can remove those exclude repos, which as well caused a
bunch of bugs.


Diffs
-----

  ambari-common/src/main/python/resource_management/core/providers/package/apt.py 70a553d

  ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py 87938ad

  ambari-common/src/main/python/resource_management/core/providers/package/zypper.py 1fbc9bf

  ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py
6dbcef5 
  ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
7f34cc1 
  ambari-common/src/main/python/resource_management/libraries/script/script.py c954a15 
  ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py
0d9687b 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py d39ccbf 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml fe47276 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 4fa4ff7 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 71297b1 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 0366758 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml d59b89e 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml aa164e3 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 481fca9 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml db7e025 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml 5472f43 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml 0c457f7 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 47bb207 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml 5cb598e 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml e7ff122 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml ccf765b 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml b0ccddd 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml 189defc 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml d0b37cf

  ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml b19aac1 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml c3a3c81 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml 6fdad8b 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml 5031f85 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml a37378c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml d103caf 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml b6b6bb8 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml 330b542 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml d1172b1 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 2fe0ed0 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 8ba2426 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml bc52e85 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml b1a5584

  ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml 8287b50 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 2cf290b 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml 1290cde 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml 00af620 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml 7720a3d 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml 088bb16 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml c2fe742

  ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml 9e82a82 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml 158ef8c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml 183d55c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml b7cd3d1 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml a1c2bd0 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml ce3fb95 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml 5ad52b6 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml 90bc472 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml e465d42 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml 2f09e88 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml 9ae1f42 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml d76ad5c 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml a27bf31

  ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml ff998f0 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml f4880e2 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml 721cc33 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml 36b01e4 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml 356225e 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml c61a270 
  ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml 4bf4198

  ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 
  ambari-server/src/test/python/custom_actions/configs/install_packages_config.json 50e327a

  ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2 

Diff: https://reviews.apache.org/r/41542/diff/


Testing (updated)
-------

mvn clean test
- stack 2.3 and 2.4. - full stacks.
- Centos 6 and Ubuntu 12 - full stacks.
- EU full stack
- RU full stack
- downgrade EU


Thanks,

Andrew Onischuk


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message