hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Zhang <>
Subject SARG predicate is ignored when query ORC table
Date Sat, 27 Feb 2016 18:59:57 GMT

We have an external ORC table which includes ~200 relatively small orc
files (less than 256MB). When querying the table with selective SARG
predicate (explain shows the predicate is qualified pushdown), we expects a
few splits generated with pruning based on predicate condition and only a
few files will be scanned. However, somehow predicate pushdown is not in
effect at all, all the files are scanned in MR job and SARG did not even
show up in the MR job config.

After digging more in hive code (version 0.14), looks like the split
pruning only happens for the stripes within each file. If the file size is
smaller than default split size, SARG is not considered. Here is the code
we are referring:

Any idea why SARG is ignored for this scenario? also can split pruning
filter out the files with all stripes not satisfied with SARG condition?
Thanks for any help, really appreciated.


View raw message