hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6268) Network resource leak with HiveClientCache when using HCatInputFormat
Date Fri, 24 Jan 2014 09:13:38 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880842#comment-13880842
] 

Lefty Leverenz commented on HIVE-6268:
--------------------------------------

Where should *hcatalog.hive.client.cache.disabled* be documented, besides the release notes?

AFAIK, none of the configuration properties in HCatConstants.java are mentioned in the wiki.
 HCatConstants itself is only mentioned a couple of times in "Notification for a New Partition":
 https://cwiki.apache.org/confluence/display/Hive/HCatalog+Notification#HCatalogNotification-NotificationforaNewPartition.

> Network resource leak with HiveClientCache when using HCatInputFormat
> ---------------------------------------------------------------------
>
>                 Key: HIVE-6268
>                 URL: https://issues.apache.org/jira/browse/HIVE-6268
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 0.12.0
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: HIVE-6268.patch
>
>
> HCatInputFormat has a cache feature that allows HCat to cache hive client connections
to the metastore, so as to not keep reinstantiating a new hive server every single time. This
uses a guava cache of hive clients, which only evicts entries from cache on the next write,
or by manually managing the cache.
> So, in a single threaded case, where we reuse the hive client, the cache works well,
but in a massively multithreaded case, where each thread might perform one action, and then
is never used, there are no more writes to the cache, and all the clients stay alive, thus
keeping ports open.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message