drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dechang Gu (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (DRILL-4126) Adding HiveMetaStore caching when impersonation is enabled.
Date Mon, 25 Jul 2016 23:06:20 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Dechang Gu closed DRILL-4126.

Verified with Apache Drill 1.5.0 (git id 3f228d3) against the commit (git id 539cbba) prior
to the patch, querying INFORTION_SCHEMA. Significant reduction in the function calls to HIVE
Before the patch (git id 539cbba):  
    -- get_all_databases was called 340 times
    -- get_all_tables was called 336 times.

with the patch (AD 1.5.0 git id 3f228d3), for the same query and same databases:
   -- get_all_databases was only called 2 times, and
   -- get_all_tables was called 38 times.

So the fixed LGTM, and the jira is closed.

> Adding HiveMetaStore caching when impersonation is enabled. 
> ------------------------------------------------------------
>                 Key: DRILL-4126
>                 URL: https://issues.apache.org/jira/browse/DRILL-4126
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Jinfeng Ni
>            Assignee: Jinfeng Ni
>             Fix For: 1.5.0
> Currently, HiveMetastore caching is used only when impersonation is disabled, such that
all the hivemetastore call goes through NonCloseableHiveClientWithCaching [1]. However, if
impersonation is enabled, caching is not used for HiveMetastore access.
> This could significantly increase the planning time when hive storage plugin is enabled,
or when running a query against INFORMATION_SCHEMA. Depending on the # of databases/tables
in Hive storage plugin, the planning time or INFORMATION_SCHEMA query could become unacceptable.
This becomes even worse if the hive metastore is running on a different node from drillbit,
making the access of hivemetastore even slower.
> We are seeing that it could takes 30~60 seconds for planning time, or execution time
for INFORMATION_SCHEMA query.  The long planning or execution time for INFORMATION_SCHEMA
query prevents Drill from acting "interactively" for such queries. 
> We should enable caching when impersonation is used. As long as the authorizer verifies
the user has the access to databases/tables, we should get the data from caching. By doing
that, we should see reduced number of api call to HiveMetaStore.
> [1] https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java#L299

This message was sent by Atlassian JIRA

View raw message