hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16979) Cache UGI for metastore
Date Wed, 28 Jun 2017 23:34:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067453#comment-16067453
] 

Tao Li commented on HIVE-16979:
-------------------------------

[~gopalv] So the fix is to remove the cache size limit and also switch to last access time
for expiration. The purpose is to make sure when we are evicting a cache entry, it's not being
used for the last 10 min for example. So there will be no read/write conflicts for the UGI.


> Cache UGI for metastore
> -----------------------
>
>                 Key: HIVE-16979
>                 URL: https://issues.apache.org/jira/browse/HIVE-16979
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Tao Li
>            Assignee: Tao Li
>         Attachments: HIVE-16979.1.patch, HIVE-16979.2.patch
>
>
> FileSystem.closeAllForUGI is called per request against metastore to dispose UGI, which
involves talking to HDFS name node and is time consuming. So the perf improvement would be
caching and reusing the UGI.
> Per FileSystem.closeAllForUG call could take up to 20 ms as E2E latency against HDFS.
Usually a Hive query could result in several calls against metastore, so we can save up to
50-100 ms per hive query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message