Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 22 Dec 2016 21:01:58 +0000 (UTC)
From: "Chao Sun (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13015143.1477414991000.584742.1482440518485@Atlassian.JIRA>
In-Reply-To: <JIRA.13015143.1477414991000@Atlassian.JIRA>
References: <JIRA.13015143.1477414991000@Atlassian.JIRA> <JIRA.13015143.1477414991973@arcas>
Subject: [jira] [Updated] (HIVE-15055) Column pruning for nested fields in
 Parquet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 22 Dec 2016 21:02:00 -0000


     [ https://issues.apache.org/jira/browse/HIVE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chao Sun updated HIVE-15055:
----------------------------
    Attachment: design-doc-nested-column-pruning.pdf

> Column pruning for nested fields in Parquet
> -------------------------------------------
>
>                 Key: HIVE-15055
>                 URL: https://issues.apache.org/jira/browse/HIVE-15055
>             Project: Hive
>          Issue Type: New Feature
>          Components: Logical Optimizer, Physical Optimizer, Serializers/Deserializers
>            Reporter: Chao Sun
>            Assignee: Chao Sun
>              Labels: performance
>         Attachments: design-doc-nested-column-pruning.pdf
>
>
> Some columnar file formats such as Parquet store fields in struct type also column by column using encoding described in Google Dramel pager. It's very common in big data where data are stored in structs while queries only needs a subset of the the fields in the structs. However, presently Hive still needs to read the whole struct regardless whether all fields are selected. Therefore, pruning unwanted sub-fields in struct or nested fields at file reading time would be a big performance boost for such scenarios.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)