drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mehant Baid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3901) Performance regression with doing Explain of COUNT(*) over 100K files
Date Tue, 06 Oct 2015 22:57:26 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945960#comment-14945960
] 

Mehant Baid commented on DRILL-3901:
------------------------------------

Wanted to quickly update the status on this, I have a patch for avoiding listing files in
a directory twice (will post patch soon). I am waiting for some performance feedback, will
post the findings once I have them.  [~sphillips] can you please file a separate JIRA for
the issue you mentioned.

> Performance regression with doing Explain of COUNT(*) over 100K files
> ---------------------------------------------------------------------
>
>                 Key: DRILL-3901
>                 URL: https://issues.apache.org/jira/browse/DRILL-3901
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.1.0
>            Reporter: Aman Sinha
>            Assignee: Mehant Baid
>
> We are seeing a performance regression when doing an Explain of SELECT COUNT(*) over
100K files in a flat directory (no subdirectories) on latest master branch compared to a run
that was done on Sept 26.   Some initial details (I will have more later): 
> {code}
> master branch on Sept 26
>    No metadata cache: 71.452 secs
>    With metadata cache: 15.804 secs
> Latest master branch 
>    No metadata cache: 110 secs
>    With metadata cache: 32 secs
> {code}
> So, both cases show regression.  
> [~mehant] and I took an initial look at this and it appears we might be doing the directory
expansion twice.  
>    



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message