Return-Path: X-Original-To: apmail-ambari-dev-archive@www.apache.org Delivered-To: apmail-ambari-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 623111761F for ; Mon, 30 Mar 2015 23:43:56 +0000 (UTC) Received: (qmail 53391 invoked by uid 500); 30 Mar 2015 23:43:56 -0000 Delivered-To: apmail-ambari-dev-archive@ambari.apache.org Received: (qmail 53363 invoked by uid 500); 30 Mar 2015 23:43:56 -0000 Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ambari.apache.org Delivered-To: mailing list dev@ambari.apache.org Received: (qmail 53348 invoked by uid 99); 30 Mar 2015 23:43:55 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Mar 2015 23:43:55 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 9BA6D1D5C7A; Mon, 30 Mar 2015 23:43:54 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2237456227975581612==" MIME-Version: 1.0 Subject: Re: Review Request 32557: Full Delete of Host : Switch host-related tables to use host_id instead of host_name column for hoststate, hostcomponentstate, hostcomponentdesiredstate From: "Alejandro Fernandez" To: "Nate Cole" , "Sid Wagle" , "Jonathan Hurley" Cc: "Alejandro Fernandez" , "Ambari" Date: Mon, 30 Mar 2015 23:43:54 -0000 Message-ID: <20150330234354.16791.18162@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Alejandro Fernandez" X-ReviewGroup: Ambari X-ReviewRequest-URL: https://reviews.apache.org/r/32557/ X-Sender: "Alejandro Fernandez" References: <20150328004543.1505.87014@reviews.apache.org> In-Reply-To: <20150328004543.1505.87014@reviews.apache.org> Reply-To: "Alejandro Fernandez" X-ReviewRequest-Repository: ambari --===============2237456227975581612== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32557/ ----------------------------------------------------------- (Updated March 30, 2015, 11:43 p.m.) Review request for Ambari, Jonathan Hurley, Nate Cole, and Sid Wagle. Changes ------- Addressed incompatible UpgradeCatalogs, so Ambari 2.1.0 cannot be used to perform UpgradeCatalogs on versions before 2.1.0 Fixed double quote issue. UQ constraint on the host_name was already added in patch revision #1. Bugs: AMBARI-10168 https://issues.apache.org/jira/browse/AMBARI-10168 Repository: ambari Description ------- See p;arent Jira AMBARI-10167 Deleting a host from the UI only removes it from the mapping to a cluster. The host still exists at the hosts endpoint, and all of the historic data maintains intact; config overrides, config groups, host state, host version, host role command logs, etc. This causes two problems. 1. If a host is removed from a cluster and its underlying environment changes (OS, IP, network card, etc), and is re-added to the cluster, then it still maintains the old information that is incorrect. 2. If host x dies, and is removed from the cluster, there is no way for a new host to take over its hostname. Expectation in Ambari 2.1.0 Deleting a host should be clean, i.e., it should delete all configuration and historic records. Because renaming a host requires a more complicated workflow, so it will have to be done manually by the user (see instructions in A1). Lastly, Ambari will require ambari-agent to be stopped before the host can be deleted. Tables: 10 tables have foreign keys to the host_name in the hosts table: 1. hostcomponentstate (fixed in this patch) 2. hostcomponentdesiredstate (fixed in this patch) 3. hoststate (fixed in this patch) 4. host_version (future patch) 5. host_role_command (future patch) 6. request_operation_level (future patch) 7. hostconfigmapping (future patch) 8. configgrouphostmapping (future patch) 9. kerberos_principal_host (future patch) 10. clusterhostmapping (already fixed) Diffs (updated) ----- ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostComponentDesiredStateDAO.java 8e92877 ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java 0fb9c59 ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostComponentDesiredStateEntity.java e0f1e9e ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostComponentDesiredStateEntityPK.java fc92858 ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostComponentStateEntity.java 0a31b90 ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostComponentStateEntityPK.java bc103a1 ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostEntity.java 4df5f39 ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostStateEntity.java 5d5a1e8 ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java 1bd2814 ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java 7975f4e ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java 1f3c09a ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java 3691af2 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog.java fcf4f51 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog150.java dd8e33e ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog151.java afe27b5 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog160.java a1f11d8 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog161.java 4ab0cb6 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java 98ac89f ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog200.java d3b062a ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java ba4dd2d ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 26ff9bb ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 13e8939 ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 622fca3 ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 9fe9cd7 ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql a722d2a ambari-server/src/test/java/org/apache/ambari/server/orm/dao/HostComponentDesiredStateDAOTest.java 7b34827 ambari-server/src/test/java/org/apache/ambari/server/state/ServiceComponentTest.java c27e600 ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClustersTest.java ebf742e ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java 8ebf2ce ambari-server/src/test/java/org/apache/ambari/server/upgrade/SectionDDL.java PRE-CREATION ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog150Test.java af6721c ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java 6fa3e0a ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog200Test.java 9c39749 ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog210Test.java 07b8410 ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalogHelper.java b5db360 ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalogTest.java b0c8fd9 Diff: https://reviews.apache.org/r/32557/diff/ Testing ------- Unit tests passed, [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ....................................... SUCCESS [4.737s] [INFO] Apache Ambari Project POM ......................... SUCCESS [0.031s] [INFO] Ambari Web ........................................ SUCCESS [38.862s] [INFO] Ambari Views ...................................... SUCCESS [2.078s] [INFO] Ambari Admin View ................................. SUCCESS [9.289s] [INFO] Ambari Metrics Common ............................. SUCCESS [1.487s] [INFO] Ambari Server ..................................... SUCCESS [34:09.719s] [INFO] Ambari Agent ...................................... SUCCESS [21.818s] [INFO] Ambari Client ..................................... SUCCESS [0.033s] [INFO] Ambari Python Client .............................. SUCCESS [0.254s] [INFO] Ambari Groovy Client .............................. SUCCESS [9.834s] [INFO] Ambari Shell ...................................... SUCCESS [0.032s] [INFO] Ambari Python Shell ............................... SUCCESS [0.044s] [INFO] Ambari Groovy Shell ............................... SUCCESS [6.661s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 35:45.473s [INFO] Finished at: Thu Mar 26 17:39:24 PDT 2015 [INFO] Final Memory: 66M/500M [INFO] ------------------------------------------------------------------------ I tested this by making the DB changes, copying my jar, then hitting the dashboard, API, and restarting services on a cluster. -- For Postgres --- Fix hoststate SELECT COUNT(*) FROM hoststate; ALTER TABLE hoststate ADD COLUMN host_id BIGINT NULL; UPDATE hoststate t SET host_id = (SELECT host_id FROM hosts h WHERE h.host_name = t.host_name) WHERE t.host_id IS NULL AND t.host_name IS NOT NULL; ALTER TABLE hoststate ALTER COLUMN host_id SET NOT NULL; ALTER TABLE hoststate ADD CONSTRAINT FK_hoststate_host_id FOREIGN KEY (host_id) REFERENCES hosts (host_id); ALTER TABLE hoststate DROP COLUMN host_name; ALTER TABLE hoststate ADD CONSTRAINT hoststate_pkey PRIMARY KEY (host_id); -- Fix hostcomponentstate SELECT COUNT(*) FROM hostcomponentstate; ALTER TABLE hostcomponentstate ADD COLUMN host_id BIGINT NULL; UPDATE hostcomponentstate t SET host_id = (SELECT host_id FROM hosts h WHERE h.host_name = t.host_name) WHERE t.host_id IS NULL AND t.host_name IS NOT NULL; ALTER TABLE hostcomponentstate ALTER COLUMN host_id SET NOT NULL; ALTER TABLE hostcomponentstate ADD CONSTRAINT FK_hostcomponentstate_host_id FOREIGN KEY (host_id) REFERENCES hosts (host_id); ALTER TABLE hostcomponentstate DROP COLUMN host_name; ALTER TABLE hostcomponentstate ADD CONSTRAINT hostcomponentstate_pkey PRIMARY KEY (cluster_id, component_name, host_id, service_name); -- Fix hostcomponentdesiredstate SELECT COUNT(*) FROM hostcomponentdesiredstate; ALTER TABLE hostcomponentdesiredstate ADD COLUMN host_id BIGINT NULL; UPDATE hostcomponentdesiredstate t SET host_id = (SELECT host_id FROM hosts h WHERE h.host_name = t.host_name) WHERE t.host_id IS NULL AND t.host_name IS NOT NULL; ALTER TABLE hostcomponentdesiredstate ALTER COLUMN host_id SET NOT NULL; ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT FK_hostcomponentdesiredstate_host_id FOREIGN KEY (host_id) REFERENCES hosts (host_id); ALTER TABLE hostcomponentdesiredstate DROP COLUMN host_name; ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT hostcomponentdesiredstate_pkey PRIMARY KEY (cluster_id, component_name, host_id, service_name); Thanks, Alejandro Fernandez --===============2237456227975581612==--