drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Sinha (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-4365) Performance with lots of small parquet files
Date Sun, 07 Feb 2016 17:54:40 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aman Sinha updated DRILL-4365:
------------------------------
    Attachment: ss1test.tar.gz

gzipped tar file for sample data set

> Performance with lots of small parquet files
> --------------------------------------------
>
>                 Key: DRILL-4365
>                 URL: https://issues.apache.org/jira/browse/DRILL-4365
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Parquet
>    Affects Versions: 1.5.0
>            Reporter: Aman Sinha
>         Attachments: ss1test.tar.gz
>
>
> I am seeing a performance degradation on 1.5.0 compared to 1.4.0 with a query over 968
small parquet files where the total # rows is only 1000, so just about 1 row per file.  The
profile shows parquet scan is slower.  With bigger  tables, I haven't seen the same issue
yet (although need confirmation from the full performance run).  
> Note: this is with default slice_target of 100K so only 1 scan fragment was used.   I
will attach the dataset to this JIRA if anyone wants to repro. 
> On 1.4.0: (with multiple runs):
> {noformat}
> 0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
> +---------+
> | EXPR$0  |
> +---------+
> | 39      |
> +---------+
> 1 row selected (2.544 seconds)
> 0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
> +---------+
> | EXPR$0  |
> +---------+
> | 39      |
> +---------+
> 1 row selected (2.434 seconds)
> {noformat}
> On 1.5.0:  (multiple runs):
> {noformat}
> 0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
> +---------+
> | EXPR$0  |
> +---------+
> | 39      |
> +---------+
> 1 row selected (3.851 seconds)
> 0: jdbc:drill:zk=local> select min(ss_item_sk) from  dfs.tmp.ss1test ;
> +---------+
> | EXPR$0  |
> +---------+
> | 39      |
> +---------+
> 1 row selected (3.61 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message