Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A21C18E09 for ; Sat, 19 Dec 2015 21:45:54 +0000 (UTC) Received: (qmail 82320 invoked by uid 500); 19 Dec 2015 21:45:54 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 82278 invoked by uid 500); 19 Dec 2015 21:45:54 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 82258 invoked by uid 99); 19 Dec 2015 21:45:53 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Dec 2015 21:45:53 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 4F59F27EE5A; Sat, 19 Dec 2015 21:45:53 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1693431718387289950==" MIME-Version: 1.0 Subject: Re: Review Request 41542: Get rid of wildcards in package names. From: "Nate Cole" To: "Sumit Mohanty" , "Jayush Luniya" , "Myroslav Papirkovskyy" , "Nate Cole" , "Jonathan Hurley" Cc: "Andrew Onischuk" , "Ambari" Date: Sat, 19 Dec 2015 21:45:53 -0000 Message-ID: <20151219214553.1646.88862@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Nate Cole" X-ReviewGroup: Ambari X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/41542/ X-Sender: "Nate Cole" References: <20151218163854.1673.22696@reviews.apache.org> In-Reply-To: <20151218163854.1673.22696@reviews.apache.org> Reply-To: "Nate Cole" X-ReviewRequest-Repository: ambari --===============1693431718387289950== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41542/#review111386 ----------------------------------------------------------- Overall I'm ok with string-substituting the version, but should we query the repo right after we make the .repo file to check if it's an "HDP, versionable repo" instead of trying to see if hdp-select is in it? Can this be controlled (somehow) by the server by having something in cluster-env that would specify what to run to make this decision for us? As I look over this code, and issues come up, it will be extremely difficult to troubleshoot and track what's really happening. - Nate Cole On Dec. 18, 2015, 11:38 a.m., Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41542/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2015, 11:38 a.m.) > > > Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav Papirkovskyy, Nate Cole, and Sumit Mohanty. > > > Bugs: AMBARI-14430 > https://issues.apache.org/jira/browse/AMBARI-14430 > > > Repository: ambari > > > Description > ------- > > **Problem**: > Wildcards in packages bring and bunch of problems connected with RU and normal > installs. > a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get > installed, but hadoop_2_2_0_1234 fails, we can no longer install master > package. > b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get > installed before hive_2_2_*, we get problems with inability to install. I've > bumped into a couple of different situations like this. (storm_2_2* and > storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3 > are real bugs which we had to hack out > c) We install a lot of packages which are not needed by default like hadoop- > 2_2_...fuse, just by asking hadoop_2_2_*. > d) Situtations like 47931 can happen in RU. > e) The bug in RU where we add a service after install_packages was triggered. > > **Proposal:** > 1\. Replace * by ${hdp_version} in metainfo.xml. > E.g.: > > > > > redhat7,redhat6,suse11 > > > hbase_2_3_* > > > phoenix_2_3_* > > > > > > -> > > > > redhat7,redhat6,suse11 > > > hbase_${hdp_version} > > > hbase_${hdp_version}-regionserver > > > phoenix_${hdp_version} > > > > > > Note: this step will also let us not override packages in metainfo.xml with > every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*. > > 2\. During INSTALL action (script.py) or during install packages > (install_packages.py) get the version we are currently using from hdp-select. > > e.g. for centos: > > > > [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}' > 2.3.4.0 > [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}' > 3204 # a bit slow for every INSTALL task > > > Another better aproach is to install hdp-select and than use rpm/dpkg, which > is faster than yum/zypper/apt-get, considering we will do this on every > INSTALL task this is important. > > > > [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g' # faster approach > 2.3.4.0-3204 > > > > u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' > 2.3.4.0-3475 > > > Vs other approaches: > I think it's the only possible approach to get the version here, sending > version which is specified by user during creating of a version. Seems to be a > bad idea as the user can specify wrong version. Also we will need to use this > during fresh install, where we don't have it specified at all. > > 3\. Replace ${hdp_version} with the version we got in step 2. > > I also discussed this with dlysnichenko this seems to be feasible in terms of > RU/EU. > The overall idea was discussed in 47931 > cc smohanty > > > Also after talking to dlysnichenko he said we were excluding repos because of issues with wildcard on suse when it is not excluded and mutptiple packages is found for a wildcard. That's why when we stop using wildcards we can remove those exclude repos, which as well caused a bunch of bugs. > > > Diffs > ----- > > ambari-common/src/main/python/resource_management/core/providers/package/apt.py 70a553d > ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py 87938ad > ambari-common/src/main/python/resource_management/core/providers/package/zypper.py 1fbc9bf > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py 6dbcef5 > ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py 7f34cc1 > ambari-common/src/main/python/resource_management/libraries/script/script.py c954a15 > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py 0d9687b > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py d39ccbf > ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml fe47276 > ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 4fa4ff7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 71297b1 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 0366758 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml d59b89e > ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml aa164e3 > ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 481fca9 > ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml db7e025 > ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml 5472f43 > ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml 0c457f7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 47bb207 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml 5cb598e > ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml e7ff122 > ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml ccf765b > ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml b0ccddd > ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml 189defc > ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml d0b37cf > ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml b19aac1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml c3a3c81 > ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml 6fdad8b > ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml 5031f85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml a37378c > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml d103caf > ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml b6b6bb8 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml 330b542 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml d1172b1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 2fe0ed0 > ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 8ba2426 > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml bc52e85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml b1a5584 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml 8287b50 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml 2cf290b > ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml 1290cde > ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml 00af620 > ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml 7720a3d > ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml 088bb16 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml c2fe742 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml 9e82a82 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml 158ef8c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml 183d55c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml b7cd3d1 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml a1c2bd0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml ce3fb95 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml 5ad52b6 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml 90bc472 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml e465d42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml 2f09e88 > ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml 9ae1f42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml d76ad5c > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml a27bf31 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml ff998f0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml f4880e2 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml 721cc33 > ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml 36b01e4 > ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml 356225e > ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml c61a270 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml 4bf4198 > ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 > ambari-server/src/test/python/custom_actions/configs/install_packages_config.json 50e327a > ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2 > > Diff: https://reviews.apache.org/r/41542/diff/ > > > Testing > ------- > > mvn clean test > - stack 2.3 and 2.4. - full stacks. > - Centos 6 and Ubuntu 12 - full stacks. > - EU full stack > - RU full stack > - downgrade EU > > > Thanks, > > Andrew Onischuk > > --===============1693431718387289950==--