Mailing-List: contact issues-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Date: Fri, 17 Mar 2017 21:28:41 +0000 (UTC)
From: "Alexander Behm (JIRA)" <jira@apache.org>
To: issues@impala.incubator.apache.org
Message-ID: <JIRA.13055655.1488844338000.54021.1489786121624@Atlassian.JIRA>
In-Reply-To: <JIRA.13055655.1488844338000@Atlassian.JIRA>
References: <JIRA.13055655.1488844338000@Atlassian.JIRA> <JIRA.13055655.1488844338000@jira-lw-us.apache.org>
Subject: [jira] [Updated] (IMPALA-5036) Improve COUNT(*) performance of
 Parquet scans.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 17 Mar 2017 21:28:46 -0000


     [ https://issues.apache.org/jira/browse/IMPALA-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Behm updated IMPALA-5036:
-----------------------------------
    Description: 
{code}
select count(*) from parquet_table;
select count(*) from parquet_table group by partition_col;
{code}

Impala already has a special code path for fast Parquet scans when no columns are scanned and materialized, but the performance can be significantly improved with a plan+execution change, as follows:

*Execution change*
Instead of returning empty batches until num_rows have been returned, the Parquet scanner can populate a single slot with the num_rows from the Parquet row groups

*Plan change*
The count(*) local aggregation needs to be changed to a sum(num_rows_slot) aggregation.
The final distributed plan will be:
scan -> local agg with sum(num_rows_slot) -> merge agg sum(sum(num_rows_slot))

This optimization is applicable where is only a count(*) and there are no scan predicates.

  was:
{code}
select count(*) from parquet_table;
select count(*) from parquet_table group by partition_col;
{code}

Impala already has a special code path for fast Parquet scans when no columns are scanned and materialized, but the performance can be significantly improved with a plan+execution change, as follows:

Execution change:
Instead of returning empty batches until num_rows have been returned, the Parquet scanner can populate a single slot with the num_rows from the Parquet row groups

Plan change:
The count(*) local aggregation needs to be changed to a sum(num_rows_slot) aggregation.
The final distributed plan will be:
scan -> local agg with sum(num_rows_slot) -> merge agg sum(sum(num_rows_slot))

This optimization is applicable where there is only count(*) and no scan predicates.


> Improve COUNT(*) performance of Parquet scans.
> ----------------------------------------------
>
>                 Key: IMPALA-5036
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5036
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
>            Reporter: Alexander Behm
>              Labels: parquet, performance
>
> {code}
> select count(*) from parquet_table;
> select count(*) from parquet_table group by partition_col;
> {code}
> Impala already has a special code path for fast Parquet scans when no columns are scanned and materialized, but the performance can be significantly improved with a plan+execution change, as follows:
> *Execution change*
> Instead of returning empty batches until num_rows have been returned, the Parquet scanner can populate a single slot with the num_rows from the Parquet row groups
> *Plan change*
> The count(*) local aggregation needs to be changed to a sum(num_rows_slot) aggregation.
> The final distributed plan will be:
> scan -> local agg with sum(num_rows_slot) -> merge agg sum(sum(num_rows_slot))
> This optimization is applicable where is only a count(*) and there are no scan predicates.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)