hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: SARG predicate is ignored when query ORC table
Date Sat, 27 Feb 2016 19:15:09 GMT
Hi,

Can you do show create table <TABLE> on your external table and send the
sections from

STORED AS ORC
TBLPROPERTIES (

onwards please?

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 27 February 2016 at 18:59, Jie Zhang <jiezh2008@gmail.com> wrote:

> Hi,
>
> We have an external ORC table which includes ~200 relatively small orc
> files (less than 256MB). When querying the table with selective SARG
> predicate (explain shows the predicate is qualified pushdown), we expects a
> few splits generated with pruning based on predicate condition and only a
> few files will be scanned. However, somehow predicate pushdown is not in
> effect at all, all the files are scanned in MR job and SARG did not even
> show up in the MR job config.
>
> After digging more in hive code (version 0.14), looks like the split
> pruning only happens for the stripes within each file. If the file size is
> smaller than default split size, SARG is not considered. Here is the code
> we are referring:
>
> https://github.com/apache/hive/blob/release-0.14.0/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L656
>
>
> Any idea why SARG is ignored for this scenario? also can split pruning
> filter out the files with all stripes not satisfied with SARG condition?
> Thanks for any help, really appreciated.
>
> Jessica
>

Mime
View raw message