From issues-return-143284-archive-asf-public=cust-asf.ponee.io@hive.apache.org Tue Nov 20 09:30:34 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2308618064E for ; Tue, 20 Nov 2018 09:29:55 +0100 (CET) Received: (qmail 22460 invoked by uid 500); 20 Nov 2018 08:28:45 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 22451 invoked by uid 99); 20 Nov 2018 08:28:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Nov 2018 08:28:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5258A1816B4 for ; Tue, 20 Nov 2018 08:28:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id XqZP8nLiuX8k for ; Tue, 20 Nov 2018 08:28:44 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 144485F645 for ; Tue, 20 Nov 2018 08:19:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 84CABE0E5D for ; Tue, 20 Nov 2018 08:19:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2516321E0A for ; Tue, 20 Nov 2018 08:19:00 +0000 (UTC) Date: Tue, 20 Nov 2018 08:19:00 +0000 (UTC) From: "Ashutosh Bapat (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-20794) Use Zookeeper for metastore service discovery MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20794: ---------------------------------- Status: In Progress (was: Patch Available) > Use Zookeeper for metastore service discovery > --------------------------------------------- > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement > Reporter: Ashutosh Bapat > Assignee: Ashutosh Bapat > Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02 > > > Right now, multiple metastore services can be specified in hive.metastore.uris configuration, but that list is static and can not be modified dynamically. Use Zookeeper for dynamic service discovery of metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper parameters directly from HiveConf. The class is changed so that it could be used for both HiveServer2 and Metastore server and works with both the configurations. Following methods from HiveServer2 are now moved into ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper quorum. When THRIFT_SERVICE_DISCOVERY_MODE (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are used as ZooKeeper quorum. When it's set to be empty, the URIs are used to locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf counterparts. It looks odd that the Metastore related configurations do not have their macros start with METASTORE, but start with THRIFT. I have just followed naming convention used for other parameters. > #* HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE (hive.metastore.zookeeper.namespace) > #* HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT (hive.metastore.zookeeper.client.port) > #* HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - (hive.metastore.zookeeper.connection.timeout) > #* HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES (hive.metastore.zookeeper.connection.max.retries) > #* HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME (hive.metastore.zookeeper.connection.basesleeptime) > Following Hive ZK configurations seem to be related to managing locks and seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when configured. > # When shutting a metastore server down it should deregister itself from Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from the specified ZooKeeper and treat those as if they were specified in THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)