Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 03F9B19000 for ; Fri, 16 Oct 2015 04:06:05 +0000 (UTC) Received: (qmail 85236 invoked by uid 500); 16 Oct 2015 04:06:04 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 85195 invoked by uid 500); 16 Oct 2015 04:06:04 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 85179 invoked by uid 99); 16 Oct 2015 04:06:04 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2015 04:06:04 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 863B32A1F33; Fri, 16 Oct 2015 04:06:02 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5236005836466682589==" MIME-Version: 1.0 Subject: Re: Review Request 39111: [PREVIEW] Stop-and-Start Upgrade: Handle Core Services From: "Alejandro Fernandez" To: "Nate Cole" , "Dmitro Lisnichenko" , "Dmytro Grinenko" , "Jonathan Hurley" Cc: "Ambari" , "Alejandro Fernandez" Date: Fri, 16 Oct 2015 04:06:02 -0000 Message-ID: <20151016040602.1509.52166@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Alejandro Fernandez" X-ReviewGroup: Ambari X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/39111/ X-Sender: "Alejandro Fernandez" References: <20151008160811.16483.1964@reviews.apache.org> In-Reply-To: <20151008160811.16483.1964@reviews.apache.org> Reply-To: "Alejandro Fernandez" X-ReviewRequest-Repository: ambari --===============5236005836466682589== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Oct. 8, 2015, 4:08 p.m., Jonathan Hurley wrote: > > ambari-common/src/main/python/resource_management/libraries/script/script.py, lines 439-442 > > > > > > Why do we we need to convert from foo_upgrade to foo here; we should be using consistent naming conventions across Java and python. I'd say just use "rolling" or "nonrolling" for the restart_type. I have a TODO comment for this on the Java side. Will fix it after I get the basics working. > On Oct. 8, 2015, 4:08 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java, line 936 > > > > > > Can you specify what extra parameters are and provide an example; to another engineer looking at this, it's hard to understand. Will do > On Oct. 8, 2015, 4:08 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java, lines 936-945 > > > > > > Can this code be abstracted into a method? It's cut/pasted a few times. Will do > On Oct. 8, 2015, 4:08 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py, lines 71-79 > > > > > > Any reason you didn't make this generic; something like `get_binary_path` in case it needs to be re-used for other components? Only HDFS needs it for now. If we see other components need to do the same thing, maybe hive, then I'll abstract it. - Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39111/#review101925 ----------------------------------------------------------- On Oct. 8, 2015, midnight, Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39111/ > ----------------------------------------------------------- > > (Updated Oct. 8, 2015, midnight) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole. > > > Bugs: AMBARI-12701 > https://issues.apache.org/jira/browse/AMBARI-12701 > > > Repository: ambari > > > Description > ------- > > Stop-and-Start Upgrade of the Stack to handle core services: HDFS, YARN, MR, ZK > > * Renamed the functions pre_rolling_upgrade and post_rolling_upgrade to pre_upgrade_restart and post_upgrade_restart that accept the upgrade_type (rolling|nonrolling) as a param. > > * When NameNode HA is present, the command used to start NameNode is different between the active and the standby, so changed MasterHostResolver.java to pick one of the NameNodes to become the active and the other the standby, and pass it in a parameter called "desired_namenode_role" > > * Wrote a function to take a backup of HDFS > > * Changed the "hdfs" command to actually search which one to use depending on the stack and version to account for Express Upgrade from HDP 2.1 -> 2.3 > > * Tested Express Upgrade with and without NameNode HA > > > Diffs > ----- > > ambari-common/src/main/python/resource_management/libraries/script/script.py 26fd158 > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java a2f642c > ambari-server/src/main/java/org/apache/ambari/server/stack/MasterHostResolver.java ef75d38 > ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java ed9b3a7 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java 648e7a4 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ColocatedGrouping.java a8e9c43 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java 1164335 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 4116101 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java 68fec03 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java 92df3b5 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java 6c0f3c7 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/TaskWrapper.java f7cc930 > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/TaskWrapperBuilder.java a5813e3 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py fa68435 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode_upgrade.py 2e5ac19 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py dd0dca4 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py b11d7ea > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py 46c7272 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode_upgrade.py e2ebbcb > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 07be496 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_ha_state.py e8c142c > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py fb39878 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py be6f0d5 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 5f6b8fe > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py f5df86f > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/snamenode.py 108ef01 > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 7bc0b6a > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py 14de094 > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py f3b3d11 > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py 16e34d4 > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py 5263d9f > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py fd25651 > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f3914f3 > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py e67f1ce > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py e58ea3c > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 973fa0f > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 02c1006 > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py b7fb578 > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 685eb6d > > Diff: https://reviews.apache.org/r/39111/diff/ > > > Testing > ------- > > * Tested Express Upgrade with and without NameNode HA > > There are still a handful of bugs. > * YARN history server start fails because NameNode is still in safemode and waiting for block reports. > * YARN Service Check fails due to some bad properties in the config pack from 2.1->2.3 > * May need to remove HDP 2.1 > * Taking HDFS backup and report to /tmp folder > > > Thanks, > > Alejandro Fernandez > > --===============5236005836466682589==--