drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Challapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4161) Make Hive Metastore client caching user configurable.
Date Thu, 18 Feb 2016 18:54:18 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152844#comment-15152844

Rahul Challapalli commented on DRILL-4161:

Verified! (Not automated as the test framework does not support this use case without hacking

Created 2 hive plugins with the below details and verified that the metastore cache is properly
updated based on the below settings. 
hive1 :
"hive.metastore.cache-ttl-seconds": "600",
 "hive.metastore.cache-expire-after": "access"

"hive.metastore.cache-ttl-seconds": "30",
 "hive.metastore.cache-expire-after": "write"

> Make Hive Metastore client caching user configurable.
> -----------------------------------------------------
>                 Key: DRILL-4161
>                 URL: https://issues.apache.org/jira/browse/DRILL-4161
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Jinfeng Ni
>              Labels: documentation
>             Fix For: 1.5.0
> Drill leverages LoadingCache in hive metastore client, in order to avoid the long access
time to hive metastore server. However, there is a tradeoff between caching stale data and
the possibility of cache hit. 
> For instance, DRILL-3893 changes cache invalidation policy to "1 minute after last write",
to avoid the chances of hitting stale data. However, it also implies that the cached data
would be only valid for 1 minute after loading/write.
> It's desirable to allow user to configure the caching policy, per their individual use
case requirement. In particular, we probably should allow user to specify:
> 1) caching invalidation policy : expire after last access, or expire after last write.
> 2) cache TTL.

This message was sent by Atlassian JIRA

View raw message