hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Bucketing is not leveraged in filter push down ?
Date Thu, 24 Sep 2015 14:55:46 GMT
Right. The big place the bucketing is leveraged is on bucket based joins.

On Thu, Sep 24, 2015 at 3:29 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I have one table which is bucketed on column name. Then I have the
> following sql:
>
> --------- select count(1) from student_bucketed_2 where name = 'calvin
> nixon';
>
> Ideally I think it should only scan one bucket. But from the log I still
> see it will scan all the bucket files. Why bucketing is not leveraged in
> filter push down ?
>
> Here's the log:
>
> 2015-09-24 14:59:22,282 INFO [TezChild] io.HiveContextAwareRecordReader:
> Processing file hdfs://
> 0.0.0.0:9000/user/hive/warehouse/student_bucketed_2/age=26/000001_0
> 2015-09-24 14:59:22,282 INFO [TezChild] exec.Utilities: PLAN PATH = hdfs://
> 0.0.0.0:9000/tmp/hive/jzhang/1ddc9a11-28f8-45e5-aa12-aa457fc9530d/hive_2015-09-24_14-59-04_435_7446484965928171550-1/-mr-10004/b2224650-d31a-4599-8606-57681287bd41/map.xml
> 2015-09-24 14:59:22,289 INFO [TezChild] io.HiveContextAwareRecordReader:
> Processing file hdfs://
> 0.0.0.0:9000/user/hive/warehouse/student_bucketed_2/age=26/000002_0
> 2015-09-24 14:59:22,289 INFO [TezChild] exec.Utilities: PLAN PATH = hdfs://
> 0.0.0.0:9000/tmp/hive/jzhang/1ddc9a11-28f8-45e5-aa12-aa457fc9530d/hive_2015-09-24_14-59-04_435_7446484965928171550-1/-mr-10004/b2224650-d31a-4599-8606-57681287bd41/map.xml
> 2015-09-24 14:59:22,296 INFO [TezChild] io.HiveContextAwareRecordReader:
> Processing file hdfs://
> 0.0.0.0:9000/user/hive/warehouse/student_bucketed_2/age=26/000003_0
> 2015-09-24 14:59:22,296 INFO [TezChild] exec.Utilities: PLAN PATH = hdfs://
> 0.0.0.0:9000/tmp/hive/jzhang/1ddc9a11-28f8-45e5-aa12-aa457fc9530d/hive_2015-09-24_14-59-04_435_7446484965928171550-1/-mr-10004/b2224650-d31a-4599-8606-57681287bd41/map.xml
> 2015-09-24 14:59:22,304 INFO [TezChild] io.HiveContextAwareRecordReader:
> Processing file hdfs://
> 0.0.0.0:9000/user/hive/warehouse/student_bucketed_2/age=26/000004_0
> 2015-09-24 14:59:22,304 INFO [TezChild] exec.Utilities: PLAN PATH = hdfs://
> 0.0.0.0:9000/tmp/hive/jzhang/1ddc9a11-28f8-45e5-aa12-aa457fc9530d/hive_2015-09-24_14-59-04_435_7446484965928171550-1/-mr-10004/b2224650-d31a-4599-8606-57681287bd41/map.xml
> 2015-09-24 14:59:22,311 INFO [TezChild] io.HiveContextAwareRecordReader:
> Processing file hdfs://
> 0.0.0.0:9000/user/hive/warehouse/student_bucketed_2/age=26/000005_0
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Mime
View raw message