Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F3553183E2 for ; Tue, 29 Dec 2015 09:51:39 +0000 (UTC) Received: (qmail 65889 invoked by uid 500); 29 Dec 2015 09:51:39 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 65853 invoked by uid 500); 29 Dec 2015 09:51:39 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 65835 invoked by uid 99); 29 Dec 2015 09:51:39 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Dec 2015 09:51:39 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id B0289295CD2; Tue, 29 Dec 2015 09:51:38 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5276528270751753070==" MIME-Version: 1.0 Subject: Re: Review Request 41542: Get rid of wildcards in package names. From: "Andrew Onischuk" To: "Sumit Mohanty" , "Jayush Luniya" , "Myroslav Papirkovskyy" , "Nate Cole" , "Jonathan Hurley" Cc: "Andrew Onischuk" , "Ambari" , "Alejandro Fernandez" Date: Tue, 29 Dec 2015 09:51:38 -0000 Message-ID: <20151229095138.4181.17560@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Andrew Onischuk" X-ReviewGroup: Ambari X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/41542/ X-Sender: "Andrew Onischuk" References: <20151229070745.4181.72093@reviews.apache.org> In-Reply-To: <20151229070745.4181.72093@reviews.apache.org> Reply-To: "Andrew Onischuk" X-ReviewRequest-Repository: ambari --===============5276528270751753070== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Dec. 29, 2015, 7:07 a.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml, line 127 > > > > > > These OS'es are typically disjoint per osSpecific group. Is it ok that debian7,ubuntu12,ubuntu14 are also repeated above? Yes sure it's ok to repeat and put them in any combinations, ambari-server understand all of those. Also just fyi that code is copied is in ambari for a long time :) - Andrew ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41542/#review112129 ----------------------------------------------------------- On Dec. 18, 2015, 4:38 p.m., Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41542/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2015, 4:38 p.m.) > > > Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav Papirkovskyy, Nate Cole, and Sumit Mohanty. > > > Bugs: AMBARI-14430 > https://issues.apache.org/jira/browse/AMBARI-14430 > > > Repository: ambari > > > Description > ------- > > **Problem**: > Wildcards in packages bring and bunch of problems connected with RU and normal > installs. > a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get > installed, but hadoop_2_2_0_1234 fails, we can no longer install master > package. > b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get > installed before hive_2_2_*, we get problems with inability to install. I've > bumped into a couple of different situations like this. (storm_2_2* and > storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3 > are real bugs which we had to hack out > c) We install a lot of packages which are not needed by default like hadoop- > 2_2_...fuse, just by asking hadoop_2_2_*. > d) Situtations like 47931 can happen in RU. > e) The bug in RU where we add a service after install_packages was triggered. > > **Proposal:** > 1\. Replace * by ${hdp_version} in metainfo.xml. > E.g.: > > > > > redhat7,redhat6,suse11 > > > hbase_2_3_* > > > phoenix_2_3_* > > > > > > -> > > > > redhat7,redhat6,suse11 > > > hbase_${hdp_version} > > > hbase_${hdp_version}-regionserver > > > phoenix_${hdp_version} > > > > > > Note: this step will also let us not override packages in metainfo.xml with > every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*. > > 2\. During INSTALL action (script.py) or during install packages > (install_packages.py) get the version we are currently using from hdp-select. > > e.g. for centos: > > > > [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}' > 2.3.4.0 > [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}' > 3204 # a bit slow for every INSTALL task > > > Another better aproach is to install hdp-select and than use rpm/dpkg, which > is faster than yum/zypper/apt-get, considering we will do this on every > INSTALL task this is important. > > > > [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g' # faster approach > 2.3.4.0-3204 > > > > u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' > 2.3.4.0-3475 > > > Vs other approaches: > I think it's the only possible approach to get the version here, sending > version which is specified by user during creating of a version. Seems to be a > bad idea as the user can specify wrong version. Also we will need to use this > during fresh install, where we don't have it specified at all. > > 3\. Replace ${hdp_version} with the version we got in step 2. > > I also discussed this with dlysnichenko this seems to be feasible in terms of > RU/EU. > The overall idea was discussed in 47931 > cc smohanty > > > Also after talking to dlysnichenko he said we were excluding repos because of issues with wildcard on suse when it is not excluded and mutptiple packages is found for a wildcard. That's why when we stop using wildcards we can remove those exclude repos, which as well caused a bunch of bugs. > > > Diffs > ----- > > ambari-common/src/main/python/resource_management/core/providers/package/apt.py 70a553d > ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py 87938ad > ambari-common/src/main/python/resource_management/core/providers/package/zypper.py 1fbc9bf > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py 6dbcef5 > ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py 7f34cc1 > ambari-common/src/main/python/resource_management/libraries/script/script.py c954a15 > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py 0d9687b > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py d39ccbf > ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml fe47276 > ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 4fa4ff7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 71297b1 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 0366758 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml d59b89e > ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml aa164e3 > ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 481fca9 > ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml db7e025 > ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml 5472f43 > ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml 0c457f7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 47bb207 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml 5cb598e > ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml e7ff122 > ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml ccf765b > ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml b0ccddd > ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml 189defc > ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml d0b37cf > ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml b19aac1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml c3a3c81 > ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml 6fdad8b > ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml 5031f85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml a37378c > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml d103caf > ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml b6b6bb8 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml 330b542 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml d1172b1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 2fe0ed0 > ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 8ba2426 > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml bc52e85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml b1a5584 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml 8287b50 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 2cf290b > ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml 1290cde > ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml 00af620 > ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml 7720a3d > ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml 088bb16 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml c2fe742 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml 9e82a82 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml 158ef8c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml 183d55c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml b7cd3d1 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml a1c2bd0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml ce3fb95 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml 5ad52b6 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml 90bc472 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml e465d42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml 2f09e88 > ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml 9ae1f42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml d76ad5c > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml a27bf31 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml ff998f0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml f4880e2 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml 721cc33 > ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml 36b01e4 > ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml 356225e > ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml c61a270 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml 4bf4198 > ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 > ambari-server/src/test/python/custom_actions/configs/install_packages_config.json 50e327a > ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2 > > Diff: https://reviews.apache.org/r/41542/diff/ > > > Testing > ------- > > mvn clean test > - stack 2.3 and 2.4. - full stacks. > - Centos 6 and Ubuntu 12 - full stacks. > - EU full stack > - RU full stack > - downgrade EU > > > Thanks, > > Andrew Onischuk > > --===============5276528270751753070==--