drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types
Date Mon, 04 Dec 2017 02:30:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276227#comment-16276227

ASF GitHub Bot commented on DRILL-5846:

GitHub user sachouche opened a pull request:


    DRILL-5846: Improve parquet performance for Flat Data Types

    Performance improvements for the Parquet Scanner (Flat Data Types). The are two flags
to control this performance enhancement (disabled by default):
    Option I -
    Config Name: store.parquet.flat.reader.bulk
    Config Type  : boolean
    Description   : Enables bulk processing to minimize memory checks and improve JVM HotSpot
    Option II -
    Config Name: scan.optimized.implicit.columns 
    Config Type  : boolean
    Description   : Memory optimization when storing duplicate value (implicit columns have
duplicate values within a batch). Code profiling indicated this step represented one third
of Parquet processing (when implicit columns are processed).  

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sachouche/drill drill-parquet-improv

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1060
commit 0a3a8b053be85c570ff24237d4737f37668383bd
Author: Salim Achouche <sachouche2@gmail.com>
Date:   2017-12-04T01:53:08Z

    DRILL-5846: Improve parquet performance for Flat Data Types


> Improve Parquet Reader Performance for Flat Data types 
> -------------------------------------------------------
>                 Key: DRILL-5846
>                 URL: https://issues.apache.org/jira/browse/DRILL-5846
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>    Affects Versions: 1.11.0
>            Reporter: salim achouche
>            Assignee: salim achouche
>              Labels: performance
>             Fix For: 1.13.0
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to further improve
the Parquet Reader performance as several users reported that Parquet parsing represents the
lion share of the overall query execution. It tracks Flat Data types only as Nested DTs might
involve functional and processing enhancements (e.g., a nested column can be seen as a Document;
user might want to perform operations scoped at the document level that is no need to span
all rows). Another JIRA will be created to handle the nested columns use-case.

This message was sent by Atlassian JIRA

View raw message