drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5394) Optimize query planning for MapR-DB tables by caching row counts
Date Wed, 29 Mar 2017 03:15:42 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946459#comment-15946459
] 

ASF GitHub Bot commented on DRILL-5394:
---------------------------------------

GitHub user ppadma opened a pull request:

    https://github.com/apache/drill/pull/802

    DRILL-5394: Optimize query planning for MapR-DB tables by caching row…

    … counts
    
    During the planning phase, we fetch MapR DB table row count multiple times. This is an
expensive operation and fetching it multiple times increases planning time considerably. 
Implemented per query cache. Fetch table rowCount only once, cache it in hbaseScanSpec and
reuse it. Copy the rowCount from old to new hbaseScanSpec when a new groupScan is created
i.e. when the filter is pushed down into scan.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ppadma/drill DRILL-5394

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #802
    
----
commit 8ff826c58f074ba58d1ee1b888b5da41eeb2d894
Author: Padma Penumarthy <ppenumar97@yahoo.com>
Date:   2017-03-16T18:40:13Z

    DRILL-5394: Optimize query planning for MapR-DB tables by caching row counts

----


> Optimize query planning for MapR-DB tables by caching row counts
> ----------------------------------------------------------------
>
>                 Key: DRILL-5394
>                 URL: https://issues.apache.org/jira/browse/DRILL-5394
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Query Planning & Optimization, Storage - MapRDB
>    Affects Versions: 1.9.0, 1.10.0
>            Reporter: Abhishek Girish
>            Assignee: Padma Penumarthy
>              Labels: MapR-DB-Binary
>             Fix For: 1.11.0
>
>
> On large MapR-DB tables, it was observed that the query planning time was longer than
expected. With DEBUG logs, it was understood that there were multiple calls being made to
get MapR-DB region locations and to fetch total row count for tables.
> {code}
> 2017-02-23 13:59:55,246 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan
- Getting region locations
> 2017-02-23 14:00:05,006 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq
- Function
> ...
> 2017-02-23 14:00:05,031 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan
- Getting region locations
> 2017-02-23 14:00:16,438 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq
- Special
> ...
> 2017-02-23 14:00:16,439 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan
- Getting region locations
> 2017-02-23 14:00:28,479 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq
- Special
> ...
> 2017-02-23 14:00:28,480 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan
- Getting region locations
> 2017-02-23 14:00:42,396 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq
- Special
> ...
> 2017-02-23 14:00:42,397 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan
- Getting region locations
> 2017-02-23 14:00:54,609 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.p.s.h.DefaultSqlHandler
- VOLCANO:Physical Planning (49588ms):
> {code}
> We should cache these stats and reuse them where all required during query planning.
This should help reduce query planning time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message