hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chaoyu Tang <ctang...@gmail.com>
Subject Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper
Date Thu, 06 Apr 2017 14:44:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171207
-----------------------------------------------------------




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1774 (patched)
<https://reviews.apache.org/r/58086/#comment244087>

    The data limit of a znode, which stores the zk lock infomration including the query string,
is 1M. Will it be better to set it (or some smaller value) as the hive.lock.query.string.max.length
default value instead of Integer.MAX_VALUE?


- Chaoyu Tang


On April 6, 2017, 6:04 a.m., Peter Vary wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> -----------------------------------------------------------
> 
> (Updated April 6, 2017, 6:04 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, and Barna
Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
>     https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query string in
the lock object
> - When constructing the HiveLockObject gets the configuration as a new parameter
> - When constructing the HiveLockObject truncates the length of the query string if it
is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 0afbc1c

>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/3/
> 
> 
> Testing
> -------
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message