hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <>
Subject [jira] [Commented] (HIVE-10252) Make PPD work for Parquet in row group level
Date Thu, 09 Apr 2015 18:55:14 GMT


Szehon Ho commented on HIVE-10252:

New strategy makes sense to me.  

I have some question though, we already call ParquetInputFormat.setFilterPredicate() earlier.
 Is that the strategy that you mention doesnt work, and maybe we can get rid of it?  Or is
it doing something else?  Sorry as I am not too familiar with that code, thanks.

> Make PPD work for Parquet in row group level
> --------------------------------------------
>                 Key: HIVE-10252
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Dong Chen
>            Assignee: Dong Chen
>         Attachments: HIVE-10252.patch
> In Hive, predicate pushdown figures out the search condition in HQL, serialize it, and
push to file format. ORC could use the predicate to filter stripes. Similarly, Parquet should
use the statics saved in row group to filter not match row group. But it does not work.
> In {{ParquetRecordReaderWrapper}}, it get splits with all row groups (client side), and
push the filter to Parquet for further processing (parquet side). But in  {{ParquetRecordReader.initializeInternalReader()}},
if the splits have already been selected by client side, it will not handle filter again.
> We should make the behavior consistent in Hive. Maybe we could get splits, filter them,
and then pass to parquet. This means using client side strategy.

This message was sent by Atlassian JIRA

View raw message