ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Re: Review Request 41542: Get rid of wildcards in package names.
Date Tue, 22 Dec 2015 20:36:17 GMT


> On Dec. 22, 2015, 8:31 p.m., Jayush Luniya wrote:
> > @Andrew Onischuk
> > Did you verify your changes for pluggable stacks? Can you verify there are no regressions
by these changes

Should I use something like BIGTOP?


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41542/#review111649
-----------------------------------------------------------


On Dec. 18, 2015, 4:38 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41542/
> -----------------------------------------------------------
> 
> (Updated Dec. 18, 2015, 4:38 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav Papirkovskyy, Nate
Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-14430
>     https://issues.apache.org/jira/browse/AMBARI-14430
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> **Problem**:  
> Wildcards in packages bring and bunch of problems connected with RU and normal
> installs.  
> a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get
> installed, but hadoop_2_2_0_1234 fails, we can no longer install master
> package.  
> b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get
> installed before hive_2_2_*, we get problems with inability to install. I've
> bumped into a couple of different situations like this. (storm_2_2* and
> storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3
> are real bugs which we had to hack out  
> c) We install a lot of packages which are not needed by default like hadoop-
> 2_2_...fuse, just by asking hadoop_2_2_*.  
> d) Situtations like 47931 can happen in RU.  
> e) The bug in RU where we add a service after install_packages was triggered.
> 
> **Proposal:**  
> 1\. Replace * by ${hdp_version} in metainfo.xml.  
> E.g.:
> 
>     
>     
>             <osSpecific>
>               <osFamily>redhat7,redhat6,suse11</osFamily>
>               <packages>
>                 <package>
>                   <name>hbase_2_3_*</name>
>                 </package>
>                 <package>
>                   <name>phoenix_2_3_*</name>
>                 </package>
>               </packages>
>             </osSpecific>
>     
> 
> ->
>     
>     
>             <osSpecific>
>               <osFamily>redhat7,redhat6,suse11</osFamily>
>               <packages>
>                 <package>
>                   <name>hbase_${hdp_version}</name>
>                 </package>
>                 <package>
>                   <name>hbase_${hdp_version}-regionserver</name>
>                 </package>
>                 <package>
>                   <name>phoenix_${hdp_version}</name>
>                 </package>
>               </packages>
>             </osSpecific>
>     
> 
> Note: this step will also let us not override packages in metainfo.xml with
> every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*.
> 
> 2\. During INSTALL action (script.py) or during install packages
> (install_packages.py) get the version we are currently using from hdp-select.
> 
> e.g. for centos:
> 
>     
>     
>     [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}'
>     2.3.4.0
>     [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}'
>     3204 # a bit slow for every INSTALL task
>     
> 
> Another better aproach is to install hdp-select and than use rpm/dpkg, which
> is faster than yum/zypper/apt-get, considering we will do this on every
> INSTALL task this is important.
> 
>     
>     
>     [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select | sed
-e 's/\.el[0-9]//g' # faster approach
>     2.3.4.0-3204
>     
>     
>     
>     u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' 
>     2.3.4.0-3475
>     
> 
> Vs other approaches:  
> I think it's the only possible approach to get the version here, sending
> version which is specified by user during creating of a version. Seems to be a
> bad idea as the user can specify wrong version. Also we will need to use this
> during fresh install, where we don't have it specified at all.
> 
> 3\. Replace ${hdp_version} with the version we got in step 2.
> 
> I also discussed this with dlysnichenko this seems to be feasible in terms of
> RU/EU.  
> The overall idea was discussed in 47931  
> cc smohanty
> 
> 
> Also after talking to dlysnichenko he said we were excluding repos because of issues
with wildcard on suse when it is not excluded and mutptiple packages is found for a wildcard.
That's why when we stop using wildcards we can remove those exclude repos, which as well caused
a bunch of bugs.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/core/providers/package/apt.py 70a553d

>   ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py
87938ad 
>   ambari-common/src/main/python/resource_management/core/providers/package/zypper.py
1fbc9bf 
>   ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py
6dbcef5 
>   ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
7f34cc1 
>   ambari-common/src/main/python/resource_management/libraries/script/script.py c954a15

>   ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py
0d9687b 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py d39ccbf

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml fe47276

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 4fa4ff7

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 71297b1

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 0366758

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml d59b89e

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml aa164e3

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 481fca9

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml db7e025

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml 5472f43 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml 0c457f7

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 47bb207

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml 5cb598e

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml e7ff122

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml ccf765b

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml b0ccddd 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml 189defc

>   ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml d0b37cf

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml b19aac1

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml c3a3c81

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml 6fdad8b

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml 5031f85

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml a37378c

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml d103caf

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml b6b6bb8

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml 330b542

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml d1172b1

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 2fe0ed0

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 8ba2426 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml bc52e85

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml b1a5584

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml 8287b50

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 2cf290b

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml 1290cde

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml 00af620

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml 7720a3d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml 088bb16

>   ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml c2fe742

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml 9e82a82

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml 158ef8c

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml 183d55c

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml b7cd3d1

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml a1c2bd0

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml ce3fb95

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml 5ad52b6

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml 90bc472

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml e465d42

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml 2f09e88

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml 9ae1f42 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml d76ad5c

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml a27bf31

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml ff998f0

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml f4880e2

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml 721cc33

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml 36b01e4

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml 356225e 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml c61a270

>   ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml 4bf4198

>   ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 
>   ambari-server/src/test/python/custom_actions/configs/install_packages_config.json 50e327a

>   ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2 
> 
> Diff: https://reviews.apache.org/r/41542/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> - stack 2.3 and 2.4. - full stacks.
> - Centos 6 and Ubuntu 12 - full stacks.
> - EU full stack
> - RU full stack
> - downgrade EU
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message