hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Na Yang (JIRA)" <>
Subject [jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
Date Mon, 22 Dec 2014 06:35:13 GMT


Na Yang commented on HIVE-9119:

According to a ZooKeeper test that I did, increasing the number of ZooKeeper client instances
does not help improving the ZooKeeper operation requests performance in a multi-thread environment.
The major load is on the ZooKeeper server side. Therefore,  a singleton ZooKeeper client is
a good choice for the ZookeeperHiveLockManager implementation. 

A zookeeper client/connection pool is not needed in this case. In addition, if a zookeeper
client/connection pool is used and the pool side is set too big by the user, that can cause
overhead on zookeeper server and even make the ZooKeeperHiveLockManager not workable if the
zookeeper client number exceeds the number that the zookeeper server could handle.    

> ZooKeeperHiveLockManager does not use zookeeper in the proper way
> -----------------------------------------------------------------
>                 Key: HIVE-9119
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Locking
>    Affects Versions: 0.13.0, 0.14.0, 0.13.1
>            Reporter: Na Yang
>            Assignee: Na Yang
>         Attachments: HIVE-9119.1.patch
> ZooKeeperHiveLockManager does not use zookeeper in the proper way. 
> Currently a new zookeeper client instance is created for each getlock/releaselock query
which sometimes causes the number of open connections between
> HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows.

> To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance
for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers.

This message was sent by Atlassian JIRA

View raw message