impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taras Bobrovytsky (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (IMPALA-5036) Improve COUNT(*) performance of Parquet scans.
Date Thu, 06 Jul 2017 18:37:00 GMT

     [ https://issues.apache.org/jira/browse/IMPALA-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Taras Bobrovytsky resolved IMPALA-5036.
---------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

> Improve COUNT(*) performance of Parquet scans.
> ----------------------------------------------
>
>                 Key: IMPALA-5036
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5036
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
>            Reporter: Alexander Behm
>            Assignee: Taras Bobrovytsky
>              Labels: parquet, performance, ramp-up
>             Fix For: Impala 2.10.0
>
>
> {code}
> select count(*) from parquet_table;
> select count(*) from parquet_table group by partition_col;
> {code}
> Impala already has a special code path for fast Parquet scans when no columns are scanned
and materialized, but the performance can be significantly improved with a plan+execution
change, as follows:
> *Execution change*
> Instead of returning empty batches until num_rows have been returned, the Parquet scanner
can populate a single slot with the num_rows from the Parquet row groups
> *Plan change*
> The count(*) local aggregation needs to be changed to a sum(num_rows_slot) aggregation.
> The final distributed plan will be:
> scan -> local agg with sum(num_rows_slot) -> merge agg sum(sum(num_rows_slot))
> This optimization is applicable where is only a count(*) and there are no scan predicates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message