hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harel Ben Attia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5853) Hive Lock Manager leaks zookeeper connections
Date Fri, 29 Nov 2013 15:49:37 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835429#comment-13835429
] 

Harel Ben Attia commented on HIVE-5853:
---------------------------------------


Thanks for the prompt answers.

I understand. I will try to understand the client-side implications of this kind of change,
since we have a consumer-web production system running more than a 100 services, and around
30 instances of HS1. Also, we're using a lot of INSERT OVERWRITE LOCAL DIRECTORY calls, which
cause the HS to actually write data to the local hard disk, and I'd need to check if there
is any change of behavior there.


> Hive Lock Manager leaks zookeeper connections
> ---------------------------------------------
>
>                 Key: HIVE-5853
>                 URL: https://issues.apache.org/jira/browse/HIVE-5853
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>            Reporter: Harel Ben Attia
>
> Hive 0.10 leaks zookeeper connections from ZooKeeperHiveLockManager. HIVE-3723 describes
a similar issue for cases of semantic errors and failures, but we're experiencing a consistent
connection leak per query (even simple successful queries like "select * from dual").
> Workaround: When turning off hive.support.concurrency, everything works fine - no leak
(obviously, since the lock manager is not used).
> Details:
> OS: CentOS 5.9
> Hive version: hive-server-0.10.0+67-1.cdh4.2.0.p0.10.el5 and hive-0.10.0+198-1.cdh4.4.0.p0.15.el5
> Hadoop version: CDH4.2
> Namenode uses HA. Hive's zookeeper configuration uses the NN zookeeper.
> The problem occurs both when using the python thrift API, and the java thrift API. 
> The leak happens even when we're running repeated "select * from dual" queries. We've
checked the zookeeper connections using "netstat -n | grep 2181 | grep ESTAB | wc -l".
> Eventually, the connection from the client reach the max connections per client limit
in ZK, causing new queries to get stuck and never return.
> We'll gladly provide more information if needed.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message