drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kevin zou (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
Date Tue, 29 Aug 2017 20:57:00 GMT

     [ https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

kevin zou updated DRILL-5738:
-----------------------------
    Attachment: log.tar.gz

Logs in debug mode

> Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
> --------------------------------------------------------------------------------------
>
>                 Key: DRILL-5738
>                 URL: https://issues.apache.org/jira/browse/DRILL-5738
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 1.6.0, 1.10.0
>         Environment: mapr 5.2
>            Reporter: kevin zou
>            Priority: Critical
>         Attachments: log.tar.gz
>
>
> I ve a Drill query on 14 tables in Hive. The query took a few seconds to execute. However,
the query would stay in "Starting" state for 10+ minutes before execution. 
>  
> I set up the log to "Debug" mode to figure out what Drill had been doing during the 10+
minutes, only to find out Drill generated excessive number of meta data queries to hive meta
store.  
>  
> Although each query took a few micro seconds (meta data cached in memory), the number
of queries was 3438793.
> drillbit.log:2017-06-05 18:50:57,201 [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG
o.a.d.e.s.hive.HiveMetadataProvider
>  - Took 4 ┬Ás to get stats from idm_intel_1x.lu_jde_emp_directory
> drillbit.log:2017-06-05 18:50:57,201 [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG
o.a.drill.exec.store.hive.HiveScan
> - HiveStats: numRows: 15, sizeInBytes: 15
> ........



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message