hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
Date Fri, 16 Nov 2018 19:34:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689923#comment-16689923
] 

Hive QA commented on HIVE-20794:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 36s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 15s{color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 30s{color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 47s{color}
| {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 33s{color} | {color:blue}
common in master has 65 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 15s{color} | {color:blue}
standalone-metastore/metastore-common in master has 29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 36s{color} | {color:blue}
itests/hive-unit in master has 2 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 40s{color} | {color:blue}
itests/util in master has 48 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 45s{color} | {color:blue}
ql in master has 2316 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 37s{color} | {color:blue}
service in master has 48 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  3s{color} | {color:blue}
standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 12s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  9s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m  3s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 35s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 35s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 15s{color}
| {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m  7s{color}
| {color:green} The patch standalone-metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m  7s{color}
| {color:green} The patch metastore-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 15s{color}
| {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 11s{color}
| {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 38s{color}
| {color:green} ql: The patch generated 0 new + 17 unchanged - 4 fixed = 17 total (was 21)
{color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 11s{color} | {color:red}
service: The patch generated 3 new + 35 unchanged - 0 fixed = 38 total (was 35) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m  7s{color}
| {color:green} The patch metastore-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  3s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 45s{color} | {color:red}
service generated 1 new + 48 unchanged - 0 fixed = 49 total (was 48) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 13s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 12s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 54s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:service |
|  |  Inconsistent synchronization of org.apache.hive.service.server.HiveServer2.zooKeeperHelper;
locked 55% of time  Unsynchronized access at HiveServer2.java:55% of time  Unsynchronized
access at HiveServer2.java:[line 573] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14965/dev-support/hive-personality.sh
|
| git revision | master / c737dbd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14965/yetus/diff-checkstyle-service.txt
|
| findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14965/yetus/new-findbugs-service.html
|
| modules | C: common standalone-metastore standalone-metastore/metastore-common itests/hive-unit
itests/util ql service standalone-metastore/metastore-server U: . |
| Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14965/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> Use Zookeeper for metastore service discovery
> ---------------------------------------------
>
>                 Key: HIVE-20794
>                 URL: https://issues.apache.org/jira/browse/HIVE-20794
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ashutosh Bapat
>            Assignee: Ashutosh Bapat
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-20794.01, HIVE-20794.02
>
>
> Right now, multiple metastore services can be specified in hive.metastore.uris configuration,
but that list is static and can not be modified dynamically. Use Zookeeper for dynamic service
discovery of metastore.
> h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome)
> The Zookeeper related code (for service discovery) accesses Zookeeper parameters directly
from HiveConf. The class is changed so that it could be used for both HiveServer2 and Metastore
server and works with both the configurations. Following methods from HiveServer2 are now
moved into ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # removeServerInstanceFromZooKeeper
> h3. HiveMetaStore conf changes
>  # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper quorum. When
THRIFT_SERVICE_DISCOVERY_MODE (hive.metastore.service.discovery.mode) is set to "zookeeper"
the URIs are used as ZooKeeper quorum. When it's set to be empty, the URIs are used to locate
the metastore directly.
>  # Here's list of Hiveserver2's parameters and their proposed metastore conf counterparts.
It looks odd that the Metastore related configurations do not have their macros start with
METASTORE, but start with THRIFT. I have just followed naming convention used for other parameters.
> #* HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE (hive.metastore.zookeeper.namespace)
> #* HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT (hive.metastore.zookeeper.client.port)
> #* HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - (hive.metastore.zookeeper.connection.timeout)
> #* HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES (hive.metastore.zookeeper.connection.max.retries)
> #* HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME
(hive.metastore.zookeeper.connection.basesleeptime)
> Following Hive ZK configurations seem to be related to managing locks and seem irrelevant
for MS ZK.
> # HIVE_ZOOKEEPER_SESSION_TIMEOUT
> # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES
> Since there is no configuration to be published, HIVE_ZOOKEEPER_PUBLISH_CONFIGS does
not have a THRIFT counterpart.
> h3. HiveMetaStore class changes
> # startMetaStore should also register the instance with Zookeeper, when configured.
> # When shutting a metastore server down it should deregister itself from Zookeeper, when
configured.
> # These changes use the refactored code described above.
> h3. HiveMetaStoreClient class changes
> When service discovery mode is zookeeper, we fetch the metatstore URIs from the specified
ZooKeeper and treat those as if they were specified in THRIFT_URIS i.e. use the existing mechanisms
to choose a metastore server to connect to and establish a connection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message