drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dechang Gu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4827) Checking modification time of directories takes too long, needs to be improved
Date Thu, 04 Aug 2016 21:10:21 GMT
Dechang Gu created DRILL-4827:
---------------------------------

             Summary: Checking modification time of directories takes too long, needs to be
improved
                 Key: DRILL-4827
                 URL: https://issues.apache.org/jira/browse/DRILL-4827
             Project: Apache Drill
          Issue Type: Improvement
          Components: Functions - Drill
    Affects Versions: 1.8.0
         Environment: RHEL 6
            Reporter: Dechang Gu


This is tracking bug for metadata cache performance for directory checking.

When evaluating the fix for Drill-4530, we run the following two queries on 50K parquet files
in a 3-layer directory hierarchy:
Query1: explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem`
where dir0=2006 and dir1=12 and dir2=15;

Query2:  explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem/2006/12/15`;

Query1 takes 3.254 secs. Query2 0.505 secs.

Drillbit.log shows that for Query1, 2.5 secs spent after metadata cache was read and before
partition pruning:

2016-08-02 15:43:43,051 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman]
INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query id 285edddf-b1f3-cd74-e826-84cb91ebc6e1:
explain plan for select * from dfs.`/tpchMetaParquet/tpch100_dir_partitioned_50000files/lineitem`
where dir0=2006 and dir1=12 and dir2=15
2016-08-02 15:43:43,193 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman]
INFO  o.a.d.exec.store.parquet.Metadata - Took 6 ms to read directories from directory cache
file
2016-08-02 15:43:45,745 ucs-node7.perf.lab [285edddf-b1f3-cd74-e826-84cb91ebc6e1:foreman]
INFO  o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule


Further investigation shows that the 2.5 secs was for checking modification time of directories,
which is proportional to the number of directories to be checked.  
Looks like this can be improved by only checking the top level directory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message