Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Tue, 16 Feb 2016 17:07:18 +0000 (UTC)
From: "Jinfeng Ni (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12939550.1455641798000.62195.1455642438233@Atlassian.JIRA>
In-Reply-To: <JIRA.12939550.1455641798000@Atlassian.JIRA>
References: <JIRA.12939550.1455641798000@Atlassian.JIRA>
 <JIRA.12939550.1455641798790@arcas>
Subject: [jira] [Updated] (DRILL-4387) Improve execution side when it
 handles skipAll query
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/DRILL-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jinfeng Ni updated DRILL-4387:
------------------------------
    Description: 
DRILL-4279 changes the planner side and the RecordReader in the execution side when they handles skipAll query. However, it seems there are other places in the codebase that do not handle skipAll query efficiently. In particular, in GroupScan or ScanBatchCreator, we will replace a NULL or empty column list with star column. This essentially will force the execution side (RecordReader) to fetch all the columns for data source. Such behavior will lead to big performance overhead for the SCAN operator.

To improve Drill's performance, we should change those places as well, as a follow-up work after DRILL-4279.

One simple example of this problem is:

{code}
   SELECT DISTINCT substring(dir1, 5) from  dfs.`/Path/To/ParquetTable`;  
{code}

The query does not require any regular column from the parquet file. However, ParquetRowGroupScan and ParquetScanBatchCreator will put star column as the column list. In case table has dozens or hundreds of columns, this will make SCAN operator much more expensive than necessary. 


  was:
DRILL-4279 changes the planner side and the RecordReader in the execution side when they handles skipAll query. However, it seems there are other places in the codebase that do not handle skipAll query efficiently. In particular, in GroupScan or ScanBatchCreator, we will replace a NULL or empty column list with star column. This essentially will force the execution side (RecordReader) to fetch all the columns for data source. Such behavior will lead to big performance overhead for the SCAN operator.

To improve Drill's performance, we should change those places as well, as a follow-up work after DRILL-4279.

One simple example of this problem is:

{code}
   SELECT DISTINCT substring(dir1, 5) from  dfs.`/Path/To/ParquetTable`;  
{code}

The query does not require any regular column from the parquet file. However, ParquetRowGroupScan and ParquetScanBatchCreator will put star column as the column list.


> Improve execution side when it handles skipAll query
> ----------------------------------------------------
>
>                 Key: DRILL-4387
>                 URL: https://issues.apache.org/jira/browse/DRILL-4387
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Jinfeng Ni
>             Fix For: 1.6.0
>
>
> DRILL-4279 changes the planner side and the RecordReader in the execution side when they handles skipAll query. However, it seems there are other places in the codebase that do not handle skipAll query efficiently. In particular, in GroupScan or ScanBatchCreator, we will replace a NULL or empty column list with star column. This essentially will force the execution side (RecordReader) to fetch all the columns for data source. Such behavior will lead to big performance overhead for the SCAN operator.
> To improve Drill's performance, we should change those places as well, as a follow-up work after DRILL-4279.
> One simple example of this problem is:
> {code}
>    SELECT DISTINCT substring(dir1, 5) from  dfs.`/Path/To/ParquetTable`;  
> {code}
> The query does not require any regular column from the parquet file. However, ParquetRowGroupScan and ParquetScanBatchCreator will put star column as the column list. In case table has dozens or hundreds of columns, this will make SCAN operator much more expensive than necessary. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)